Dumping Lots of Data to Disk in Realtime?

Posted by Cliff on Saturday May 14, 2005 @01:04AM from the too-much-for-an-RDBMS? dept.

AmiChris asks: "At work I need something that can dump sequential entries for several hundred thousand instruments in realtime. It also needs to be able to retrieve data for a single instrument relatively quickly. A standard relational database won't cut it. It has to keep up with 2000+ updates per second, mostly on a subset of a few hundred instruments active at a given time. I've got some ideas of how I would build such a beast, based on flat files and a system of caching entries in memory. I would like to know if: someone has already built something like this; and if not, would someone want to use it if I build it? I'm not sure what other applications there might be. I could see recording massive amounts of network traffic or scientific data with such a library. I'm guessing someone out there has done something like this before. I'm currently working with C++ on Windows. "

1 of 127 comments (clear)

Min score:

Reason:

Sort:

Cluster it by canuck57 · 2005-05-14 01:24 · Score: 3, Insightful

I know your working with windows but when I read this I said yes.

I'm guessing someone out there has done something like this before.

Google has a cluster of machines far larger than you need but their approach was a Linux cluster. Plus, for the amount of writes going on your going to want not to have any burdens on the system that are not needed.