I began to write up a blog post about how to do things in Xtreams compared to Streams. I was also benchmarking as I went. Things were going smoothly until I got to an example where I would read a file line by line and test if it starts with an angle bracket. In this case, Xtreams was well over 60x slower. Scanning over a large file took a long time.

I began to investigate and I discovered that there were plenty of places to improve performance, but the biggest killer of time was the testing-for and raising-of exceptions. In my particular file, 70,000 exceptions were being raised over the 22mb file. The time to test-for and raise these 70,000 exceptions was more than half the execution time of the example.

I made a branch of Xtreams that didn't use exceptions. Instead, the API methods would return the number of elements they read and you would have to check that result each time. This was not an ideal API and Martin hated it. Still, the example was capable of running only 2x slower than the regular Streams version, while doing it in a much "fancier" fashion (ie: I had more options and more power at my finger tips).

We still want the exceptions, we're not going to live without them, but we came to the conclusion that raising exceptions when you read the end of a substream (as opposed to when you read the end of a stream) was a bad thing. 70,000 substreams do not need to raise an exception internally just to move you to the next substream. If you send a method like #rest or -= 0 to a substream, you shouldn't ever have to raise an exception. It's only if you attempt to read -past- the end that you should see the exception.

Substreams are useless if you don't give them a reasonable end condition. In our current implementation you provide the end condition separate to the substream itself. Here's an example that will give you a substream that will read only 100 elements at a time:

stream slicing: [:substream | substream limiting: 100]

Because we have a separation of the slicing implementation and the ending implementation, the two must communicate between themselves and they do so by raising and listening for exceptions. This is where the expensiveness comes in.

Since you're unlikely use a substream without requiring it to end in some fashion ahead of the main stream and since you can always use a substream to do the ending even if you only ever want one substream, we've decided to merge the slicing and ending code together. This will allow us to implement substream slicing without the need for thousands of internal exception checks and raises.

We've not begun implementing this, but I suspect the implementing will probably remain fairly similar to the end user, sans the #slicing: terminology, in otherwords this would be the same example as above:

stream limiting: 100

Hopefully this new approach will mean we can have comparable speeds to Smalltalk-80 streams while still giving us all the new power of Xtreams.