Following up on the ANTs Data Server issues, their Tech Support guys promptly sent me a email saying that the Bugs have been logged and that they are working on it:
StreamCruncher 1.06 Beta is now available!
This release contains 2 new features, both of them have to do with improvements in the Query language. The first addition is the case..when..then..else..end clause. Most Databases support this in the select.. clause. An extremely useful feature to handle null column values, to rewrite column values etc.
The second addition is the first n or top n or the limit m offset n clauses. SC does not validate which clause you should use for the Database being used underneath. If the Database you are using supports the first n clause like Oracle TimesTen, then use it. If you are using H2 Database and you want to truncate the ResultSet, then you should use the limit m offset n clause. Check your DB's manual to find out which clause to use.
Both features in this release work only if the Database being used supports those clauses. Have a look at CaseWhenClauseTest and TopOrLimitClauseTest to see how it works.
The first m or its equivalent clause will prove to very useful if you just want to sample just a few Rows/Events without having to retrieve all the Events/Rows, which is always time consuming.
StreamCruncher 1.05 Beta is out!
Ah, finally..I got the time to re-do the Partitioning and Pre-Filtering logic. Until this 1.05 release, Partitions had to pull Events from the source Stream/Table just before Query Processing - one step before the final Query could be executed. So, the overhead of fetching the Events and Pre-Filtering them ("where.. " clause in the Partition definition) would add some latency to the overall Query processing. Even though each Partition used to run in its own Thread, the whole process would have had to wait for all the Partitions to draw the new Events into their respective Partitions. From the 1.05 release, the Events are pre-fetched for each Partition by a separate group of Threads. This should improve the speed and CPU utilization on Multi-Core/Multi-Processor Hardware. Overall Latency should reduce noticeably on such Hardware.
As a consequence of this change in architecture, Partitions with the Pre-Filter clause do not trigger the Query unless the Events have passed through the Pre-Filter. In previous releases, unfiltered Events would trigger the Query (if the total Event Weight reached 1.0 or higher) an then would get filtered before reaching the Partition. This was very awkward for Queries with "New Events Windows", because Events that would trigger the Query spuriously would result in the "New Events Windows" to discard their contents.
Load distribution across multiple Queries is possible now without any untoward consequences (like the "New Events Windows" problem) because the Pre-Filtering works in a Thread pool of its own.