This post originated from an RSS feed registered with .NET Buzz
by Udi Dahan.
Original Post: High volume event stream processing
Feed Title: Udi Dahan - The Software Simplist
Feed URL: http://feeds.feedburner.com/UdiDahan-TheSoftwareSimplist
Feed Description: I am a software simplist. I make this beast of architecting, analysing, designing, developing, testing, managing, deploying software systems simple.
This blog is about how I do it.
I ran into this post a couple of days ago on InfoQ which described some kind of product/technology that would enable a system to perform "high volume event correlation".
These happen to be the kind of systems that I work on.
For some reason, the company behind this product thinks that databases don't support these non-functional requirements well. I'd have to say that a naive implementation probably wouldn't - but then a naive implementation in any domain would probably perform poorly.
In cases where you have to handle high volumes of information "streaming" at the system constantly, you'll often find that it's usually just the same data being constantly updated. I've seen this happen in systems that receive updates 50 times a second per source, with dozens of sources.
Would you believe that not only are databases capable of handling these loads but object-relational mapping too?
If you are building these kinds of systems, don't buy in to the hype that some vendor's product will solve you're problems. True for any kind of system, apparently.
And a solution to getting the database to perform at that level? Simple really. Analyze the data coming in. Set up tables specifically for it. Put them on a separate volume/partition at the db level. Stick a 2 gig flash disk on the db server. Map the above partition to that disk. Move the data from those tables to archive tables on volumes on regular disks as the data gets older. Voila, a database that performs amazingly well at high loads.
Previous Topic
