"Flexible" design (Willis Tower)

Inside Chicago's Willis/Sears Tower - creaking sounds (mp3) when the tall building sways in the wind. The swaying is by design - to help it cope with the structural stresses it encounters at the top, from strong winds in stormy weather.

A little short of half a kilometer in height and one of the tallest buildings in the world. Now, that is extreme engineering!

Even Star Trek technology has memory fragmentation issues

Memory fragmentation problems? Well, even Star Trek holograms have similar issues (1) (2). This is why you need a Compacting Garbage Collector :)

select 'Whoa..' || 'Cool!' from mozilla_history_storage

SQLite is a brilliant embedded SQL engine and it is used by software that you and I use everyday - Firefox, Android and many more. See earlier reference.
What you might not know is how extensively it is used in Firefox. History, bookmarks all use SQLite. The kind of queries that are used would put many Enterprise applications to shame. I was just curious to see what was under the hood and decided to hunt down the History and Bookmarks source code.

 "SELECT v.visit_date, COALESCE( "
   "(SELECT r.visit_type FROM moz_historyvisits_temp r "
     "WHERE v.visit_type IN ") +
       nsPrintfCString("(%d,%d) ", TRANSITION_REDIRECT_PERMANENT,
                                   TRANSITION_REDIRECT_TEMPORARY) +
       NS_LITERAL_CSTRING(" AND r.id = v.from_visit), "
   "(SELECT r.visit_type FROM moz_historyvisits r "
     "WHERE v.visit_type IN ") +
       nsPrintfCString("(%d,%d) ", TRANSITION_REDIRECT_PERMANENT,
                                   TRANSITION_REDIRECT_TEMPORARY) +
       NS_LITERAL_CSTRING(" AND r.id = v.from_visit), "
   "visit_type) "
 "FROM moz_historyvisits_temp v "
 "WHERE v.place_id = :page_id "
 "UNION ALL "
 "SELECT v.visit_date, COALESCE( "
   "(SELECT r.visit_type FROM moz_historyvisits_temp r "
     "WHERE v.visit_type IN ") +
       nsPrintfCString("(%d,%d) ", TRANSITION_REDIRECT_PERMANENT,
                                   TRANSITION_REDIRECT_TEMPORARY) +
       NS_LITERAL_CSTRING(" AND r.id = v.from_visit), "
   "(SELECT r.visit_type FROM moz_historyvisits r "
     "WHERE v.visit_type IN ") +
       nsPrintfCString("(%d,%d) ", TRANSITION_REDIRECT_PERMANENT,
                                   TRANSITION_REDIRECT_TEMPORARY) +
       NS_LITERAL_CSTRING(" AND r.id = v.from_visit), "
   "visit_type) "
 "FROM moz_historyvisits v "
 "WHERE v.place_id = :page_id "
   "AND v.id NOT IN (SELECT id FROM moz_historyvisits_temp) "
 "ORDER BY visit_date DESC LIMIT ") +

Now...see this. Imagine doing this without SQL? I am however, a little disappointed. I was expecting the code to be using some fancy NGram search like what Lucene does, but it doesn't look like it.

 SELECT DISTINCT null, 
        'place:type=%ld&sort=%ld&domain=&domainIsHost=true
          &beginTime='||:begin_time||'&endTime='||:end_time, 
        :localhost, :localhost, null, null, null, null, null, null, null 
 WHERE EXISTS( 
   SELECT h.id 
   FROM moz_places h 
   JOIN moz_historyvisits v ON v.place_id = h.id 
   WHERE h.hidden <> 1 AND h.rev_host = '.' 
     AND h.visit_count > 0 
     AND h.url BETWEEN 'file://' AND 'file:/~' 
     {QUERY_OPTIONS_VISITS} {QUERY_OPTIONS_PLACES} 
     {ADDITIONAL_CONDITIONS} 
   UNION 
   SELECT h.id 
   FROM moz_places_temp h 
   JOIN moz_historyvisits v ON v.place_id = h.id 
   WHERE h.hidden <> 1 AND h.rev_host = '.' 
     AND h.visit_count > 0 
     AND h.url BETWEEN 'file://' AND 'file:/~' 
     {QUERY_OPTIONS_VISITS} {QUERY_OPTIONS_PLACES} 
     {ADDITIONAL_CONDITIONS} 
   UNION 
   SELECT h.id 
   FROM moz_places h 
   JOIN moz_historyvisits_temp v ON v.place_id = h.id 
   WHERE h.hidden <> 1 AND h.rev_host = '.' 
     AND h.visit_count > 0 
     AND h.url BETWEEN 'file://' AND 'file:/~' 
     {QUERY_OPTIONS_VISITS} {QUERY_OPTIONS_PLACES} 
     {ADDITIONAL_CONDITIONS} 
   UNION 
   SELECT h.id 
   FROM moz_places_temp h 
   JOIN moz_historyvisits_temp v ON v.place_id = h.id 
   WHERE h.hidden <> 1 AND h.rev_host = '.' 
     AND h.visit_count > 0 
     AND h.url BETWEEN 'file://' AND 'file:/~' 
     {QUERY_OPTIONS_VISITS}  {QUERY_OPTIONS_PLACES} 
     {ADDITIONAL_CONDITIONS}         
 ) 
 UNION ALL 
 SELECT DISTINCT null, 
        'place:type=%ld&sort=%ld&domain='||host||'&domainIsHost=true
          &beginTime='||:begin_time||'&endTime='||:end_time, 
        host, host, null, null, null, null, null, null, null 
 FROM ( 
   SELECT DISTINCT get_unreversed_host(rev_host) AS host 
   FROM moz_places h 
   JOIN moz_historyvisits v ON v.place_id = h.id 
   WHERE h.rev_host <> '.' 
     AND h.visit_count > 0 
     {QUERY_OPTIONS_VISITS} {QUERY_OPTIONS_PLACES} 
     {ADDITIONAL_CONDITIONS} 
   UNION 
   SELECT DISTINCT get_unreversed_host(rev_host) AS host 
   FROM moz_places_temp h 
   JOIN moz_historyvisits v ON v.place_id = h.id 
   WHERE h.rev_host <> '.' 
     AND h.visit_count > 0 
     {QUERY_OPTIONS_VISITS} {QUERY_OPTIONS_PLACES} 
     {ADDITIONAL_CONDITIONS} 
   UNION 
   SELECT DISTINCT get_unreversed_host(rev_host) AS host 
   FROM moz_places h 
   JOIN moz_historyvisits_temp v ON v.place_id = h.id 
   WHERE h.rev_host <> '.' 
     AND h.visit_count > 0 
     {QUERY_OPTIONS_VISITS} {QUERY_OPTIONS_PLACES} 
     {ADDITIONAL_CONDITIONS} 
   UNION 
   SELECT DISTINCT get_unreversed_host(rev_host) AS host 
   FROM moz_places_temp h 
   JOIN moz_historyvisits_temp v ON v.place_id = h.id         
   WHERE h.rev_host <> '.' 
     AND h.visit_count > 0 
     {QUERY_OPTIONS_VISITS} {QUERY_OPTIONS_PLACES} 
     {ADDITIONAL_CONDITIONS}         
   ORDER BY 1 ASC 

There's plenty more where this came from:
  - mozilla-central/source/toolkit/components/places/src/nsNavHistory.cpp
  - https://developer.mozilla.org/en/Places/Query_System
  - https://developer.mozilla.org/en/Storage
  - https://developer.mozilla.org/en/Places/History_Service/Design

If you wish to use SQLite from your Java program, there are some drivers available. SQLite also supports a pure memory mode, much like my other favorite database H2.
  - http://www.xerial.org/trac/Xerial/wiki/SQLiteJDBC
  - http://code.google.com/p/sqlite4java/

Time series, disk swapping, shazam patents and other stories

Here's a collection of useful Cassandra and HBase articles I've come across in the past few months:

Time series storage in the big 2 NoSQL systems - Cassandra and Hbase:
  - http://outerthought.org/lily/417-ot.html
  - http://markmail.org/search/?q=cassandra+counter+increment
  - https://www.cloudkick.com/blog/2010/mar/02/4_months_with_cassandra/
  - http://github.com/zohmg/zohmg

Apache Hive vs Pig:
  - http://www.jroller.com/otis/entry/hadoop_land_hive_vs_pig
  - http://www.larsgeorge.com/2009/10/hive-vs-pig.html

Cassandra GC and swapping:
  - http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html
  - http://www.riptano.com/blog/whats-new-cassandra-065

Geohashing sounded like an ingenious concept. Here's something built on Cassandra:

  - http://www.slideshare.net/mmalone/scaling-gis-data-in-nonrelational-data-stores

Well, yeah that's a lot of NoSQL articles to read. Here's a hilarious video against NoSQL to balance it. (Warning: Watch for foul language. For a more civilized roast, see this)
  - http://www.royans.net/arch/mongodb-is-webscale-humor/

Patent trouble . Here's the story of a smart guy who wrote a music recognizer over a weekend and got into some trouble with Patent lawyers.

After a long time, I found a nice JUnit presentation that made me reconsider my decision to switch to TestNG from JUnit.

Some Linux fun - swapping OS pages and opening 500K sockets (Also see above for what Cassandara did  to prevent swapping):
  - http://blog.urbanairship.com/blog/2010/09/29/linux-kernel-tuning-for-c500k/
  - http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

Until next time!

Red Munia or Red Avadavat

Red Munia or Red Avadavat, originally uploaded by SRJP.

Pic taken by my dad - Dr. Jayaprakash.

Red Munia or Red Avadavat

Red Munia or Red Avadavat, originally uploaded by SRJP.

Pic taken by my dad - Dr. Jayaprakash.