StreamCruncher 2.0 Beta is ready!

This 2.0 version is the result of a major refactoring job.

1) The API has been greatly simplified. The internal architecture has changed considerably, resulting in a vast improvement in performance. The TimeWindowFPerfTest (single Query on a Stream) can do 25,500 Events per second on a 1.6 GHz Centrino! More details and log files in my next Blog

2) Plain Windows constructs that were available in previous version has been removed entirely. Partitions are the only construct now. The syntax for Simple (anonymous) Window Partitions has changed slightly, in that there is no by keyword between partition and store

3) An additional property db.schema can be specified, as the Database Schema in which the Kernel creates its internal artifacts

4) Chained Partitions from now on, must always have a Pre-Filter clause starting with $row_status is new/dead

5) The Kernel can now accommodate Events that arrive out-of-order. OutOfOrderEventTest demonstrates this new ability

6) Pre-Filter temporarily does not support the complete SQL grammar like in, exists

Phew! It took me more than a month for make these changes. The internal Event stores for Input Event Streams and Partitions have changed considerably. The idea was to keep the Events inside the Kernel's process for as long as possible and delay inserting the Events into the Database to the last stages. As a result, Events get passed around as references most of the time. The latency per Event has dropped dramatically.

There are a few things that need to be completed/cleaned up - like the Pre-Filter clause syntax, Kernel restarts do not work correctly. This version comes with the latest version of the H2 Database, which now supports Table-level concurrency - a much needed feature for StreamCruncher.

As mentioned in my previous Blog, StreamCruncher 2.0, when it will be ready (hopefully, in a week or two) will have a much simpler API, will be easier to setup Input Streams, publish Events and also to consume the results from the Output Stream. The end User will not have to deal with the Database at all.

Even inside the Kernel, the use of the underlying Database has been minimized to the extent possible. Most of the operations are done inside the Kernel! Which means that the Events are passed around as references, which reduces the overall GC pause times because of reduced garbage (Event copies). Pre-Filtering in the Partition-Where clauses are also done inside the Kernel itself, instead of relying on the Database. Chained-Partitions will be much faster because the intermediate Database Tables have been got rid of!

As a result of all these changes, the performance has gone up considerably and the average latency has gone down.

Coming soon...a leaner and meaner StreamCruncher 2.0. With rather major changes to the internal architecture and API. Some preliminary tests on the TimeWindowFPerfTest revealed a 75% increase in performance!! Previous results.