Recently, Martin Kobetic and I have set up a google code project for Xtreams, so that we can share the documentation we've put together - well, mostly Martin. Xtreams is fleshing out to be a useful framework and potential replacement for classic Smalltalk streams. I thought it might be useful to blog about the basics a bit.

First of all, what is Xtreams? Xtreams is an abstract producer/consumer pipeline over arbitrary source and destination types. In otherwords you get a unified API for accessing files, sockets, pipes, strings, collections and many many other kinds of things.

Xtreams also does a lot of map/reduce like behavior. An example of the most common sort of map you might do would be encoding, such as reading a file off of disk as bytes and decoding it from UTF8 to characters in a string. So let's get started with some basics:

Creating a reading stream or a writing stream is accomplished by sending #reading or #writing to the source/destination object, eg:

reading := 'Hello World' reading.

writing := String new writing.

You can read data from a reading stream with the various read* API, eg: read:, read:into:, read:into:at:, but the simplest example would probably be:

'Hello World' reading read: 5 --> 'Hello'

Similarly, if you can write data to a writing stream with the various write* API, eg:

String new writing write: 'Hello World'

You can also work with singular objects if that is desired using the reading #get and writing #put: methods. Those aren't usually needed.

When you're done with a reading or writing stream, you can send #close to it. The #close behavior flows down to the source/destination object so if you have an open file you're streaming over, it will close that file too.

Xtreams takes a significant departure at this point from classic Smalltalk streams by adding a concept called 'stacking', where you can create a stack of streams that will transmit data between its layers as efficiently as possible. The most common version of this would be:

(ByteArray new writing encoding: #utf8) write: 'Hello World'

This stacks an encoding write stream on top of the write stream on top of the byte array. As 'Hello World' gets written, it gets encoded transforming it from characters in to bytes.

The stacking can be completely arbitrary. Here's an example that reads in integers but increments them by 1:

(#( 1 2 3 4 ) reading collecting: [:each | each + 1]) rest --> #( 2 3 4 5 )

That would be an example of a 'map' but you can also do a 'reduce':

(#( 1 2 3 4) reading selecting: #odd) rest --> #( 1 3 )

One of the key characteristics of Xtreams is that you're never going to read more than the default buffer size worth of data in to memory at a time and you'll never read ahead of the amount of data you have requested. This means you can process gigabytes+ worth of data and never strain the system. You'll become i/o bound rapidly.

In Unix there is a program called tee which splits a stream in to two. We can achieve that using a custom transformation "stackification":

reading := aSource reading.

spy := reading contentsSpecies new writing.

tee := reading transforming: [:in :out | | data | data := in get. out put: data. spy put: data].

In this scenario, any data that is read from tee will also be written in to spy. This can be a useful debugging technique when you're trying to ascertain the movement of objects and data along a stream.

The last section of this primer is about seeking. Streams conceptually have a position and while not all sources and destinations are positionable (eg: a socket) we can still create a buffer and position relative to that buffer.

positionable := reading positioning.

This is only required if your stream is not positionable. Once you have a positionable stream you can send it the seeking API of ++ to move forward from where you are, -- to move backward from where you are, += to move forward from the start of the stream and -= to move backward from the end of the stream. You can also get and set the position with #position and #position:

'Hello World' reading -= 5; rest --> 'World'

There is a lot more to Xtreams than just these core basics. Take a look at the documentation and load up the code for yourself if you're interested. The code is in the public store under the bundle XtreamsDevelopment.