Slashdot Mirror
Replacing Traditional Storage, Databases With In-Memory Analytics
storagedude writes "Traditional databases and storage networks, even those sporting high-speed solid state drives, don't offer enough performance for the real-time analytics craze sweeping corporations, giving rise to in-memory analytics, or data mining performed in memory without the limitations of the traditional data path. The end result could be that storage and databases get pushed to the periphery of data centers and in-memory analytics becomes the new critical IT infrastructure. From the article: 'With big vendors like Microsoft and SAP buying into in-memory analytics to solve Big Data challenges, the big question for IT is what this trend will mean for the traditional data center infrastructure. Will storage, even flash drives, be needed in the future, given the requirement for real-time data analysis and current trends in design for real-time data analytics? Or will storage move from the heart of data centers and become merely a means of backup and recovery for critical real-time apps?'"
Discarding data is something that, as a programmer, I don't often do. Too often I will need it later. Real time analytics are not going to change this. As long as hard drive storage continues to get cheaper, there's going to be more data stored. Partially because the easier it is to store large blocks the more likely I am to store bigger packets. I'd LOVE to store entire large XML blocks in databases sometimes, and we decide not to because of space issues. So, yeah, no. Datacenters aren't going anywhere. Things just get more complicated on the hosting side.
Note that the article writer is a strong stakeholder in his earthshattering predictions coming true.
My little site.
For the cutting edge in this area, see what the "high frequency traders" are doing. Computers aren't fast enough for that any more. The trend is toward writing trading algorithms in VHDL and compiling them into FPGAs, so the actual trading decisions are made in special-purpose hardware. Transaction latency (from trade data in on the wire to action out) is dropping below 10 microseconds. In the high-frequency trading world, if you're doing less than 1000 trades per second, you're not considered serious.
More generally, we have a fundamental problem in the I/O area: UNIX. UNIX I/O has a very simple model, which is now used by Linux, DOS, and Windows. Everything is a byte stream, and byte streams are accessed by making read and write calls to the operating system. That was OK when I/O was slower. But it's a terrible way to do inter-machine communication in clusters today. The OS overhead swamps the data transfer. Then there's the interaction with CPU dispatching. Each I/O operation usually ends by unblocking some thread, so there's a pass through the scheduler at the receive end. This works on "vanilla hardware" (most existing computers), which is why it dominates.
Bypassing the read/write model is sometimes done by giving one machine remote direct memory access ("RDMA") into another. This is usually too brutal, and tends to be done in ways that bypass the MMU and process security. So it's not very general. Still, that's how most Ethernet packets are delivered, and how graphics units talk to CPUs.
The supercomputer interconnect people have been struggling with this for years, but nothing general has emerged. RDMA via Infiniband is about where that group has ended up. That's not something a typical large hosting cluster could use safely.
Most inter-machine operations are of two types - a subroutine call to another machine, or a queue operation. Those give you the basic synchronous and asynchronous operations. A reasonable design goal is to design hardware which can perform those two operations with little or no operating system intervention once the connection has been set up, with MMU-level safety at both ends. When CPU designers have put in elaborate hardware of comparable complexity, though, nobody uses it. 386 and later machines have hardware for rings of protection, call gates, segmented memory, hardware context switching, and other stuff nobody uses because it doesn't map to vanilla C programming. That has discouraged innovation in this area. A few hardware innovations, like MMX, caught on, but still are used only in a few inner loops.
It's not that this can't be done. It's that unless it's supported by both Intel and Microsoft, it will only be a niche technology.
1TB is still in the realm of rather specialized; but 512GB systems(while not inexpensive) are actually pretty available. A quick glance at Dell shows that(even without the benefits of a rep, volume pricing, or any sort of negotiation), a 2U R815 with 512GB of RAM can be yours for a hair under $40,000. Kitted out with the specs you actually want, of course, it might run you another $20k above that. If AMD isn't your flavor, the intel-based but otherwise similar R810 will run five to ten thousand more than the R815 with otherwise similar options...
At those prices, I'd venture to say that Flash still has a reasonably bright future ahead of it in the high-speed/low-latency storage market(not to mention the volatility issue); but(especially if your problem can handle being broken up across multiple systems with only modestly fast interconnects) the cost of enormous amounts of RAM has dropped pretty significantly.
Now, if you can't deal with the limitations of commodity cluster interconnects, and have to have more than a half terabyte of RAM in a single memory space, I get the impression that your options get more expensive pretty fast. Phrases like "up to 16TB shared global memory" and "single system image", are generally your cue to hold on to your wallet and run... If that is what you want, though, you can buy it.
I'm getting sick and tired of hearing about yet another hype in IT-land where everything has to be done in yet another new way.
All developers understand that different problems require different solutions. Will the managers who shove this crap up our asses please stop doing so? It's not productive, you're not going to get a better solution by forcing it do be implemented in whatever buzzword falls of the last bandwagon of an ever-growing parade of buzzwords.
"In-memory analytics" is what we started out with before databases, and guess what; it's never gone away. We've never stopped using it. Now just tell us what problem you have let us developers decide how to solve it.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
You're misinterpreting the post. No one said anything about long term data storage being marginalized or eliminated. Instead, the author is talking about the difference between persistent and non-persistent storage. He's saying that existing database technologies that rely on persistent storage are being marginalized as the speed difference between spinning disks and RAM widens, and the low cost of RAM makes it practical to hold large data sets entirely in memory. According to the author, data processing and analysis will increasingly move towards in-memory systems, while traditional databases will be relegated to a "backup and restore" role for these in-memory systems.
We all know what to do, but we don't know how to get re-elected once we have done it
Decentralization is the way.
That one wasn't even intentional, unfortunately. My love of puns has, apparently, seeped directly into whatever part of my brain is responsible for day-to-day verbal and written work...
Although TFA doesn't say so explicitly, I think it's talking about the race to get the best targeted advertising analytics in place for global applications like eBay, FB etc. These applications don't have the same database requirements as traditional business apps. It makes sense to talk about new ways of doing things for them, but TFA's author and a lot of other people make the mistake of thinking or implying that these new techniques will apply directly to traditional business apps as well. Sorry, not.
----------
Happy New Year, may it suck less for ya than the last one.
Exactly. It really doesn't matter if you have the slowest (and thus longer lasting and cheaper to operate) HDD on the planet if all the important data is in RAM and kept there. RAM since DDR has gotten so ridiculously fast that NO SSD has a snowball's chance of catching up anytime soon, if at all, and the economies of scale have made RAM one of the cheapest if not the cheapest upgrades you can add to any system.
Even in the consumer market falling RAM prices and changes to OS design make the hard drive pretty much a backup and long term storage medium more than anything else. I advise my customers on new builds to go ahead and let me install 4GB, because with Superfetch after a week of Windows 7 learning their usage patterns all of their apps are preloaded into RAM making launching and using instantaneous, and with suspend to RAM booting is pretty much a thing of the past. It cost less than $100 to add 8GB to mine and now everything I use is ALWAYS preloaded, making the speed just insane. Everyone that comes by the shop is always amazed at how I can launch a half a dozen apps while another 4 or 5 are doing various jobs and it is always instantaneous. But with 6GB reserved by the OS for Superfetch all the apps I use are simply waiting for me in RAM.
So I have to agree with TFA. With the prices of RAM cheap and only getting cheaper having data you actually use often needing to swap in and out of the HDD or SSD is just nuts. And then if you have it all in RAM you can use the lower speed and less power hungry "green" drives for persistent backup instead of using SSDs which haven't come anywhere near the GB per $ ratio of spinning platters yet, although their speed is incredible. But if everything is already in RAM, do you really need to spend the crazy $$$ for the large SSD?
ACs don't waste your time replying, your posts are never seen by me.
Decentralization is the way.
If you're a consultant and find a client working in a centralized way, you sell decentralization as the way to solve all their woes. If you find them working in a decentralized way, you sell them on centralizing to solve all their woes.
There are only two constants here: 1) every business has woes, regardless of structure; 2) consultants extract lots of value by shifting those woes around
what help is diesel when the main power room with Transfer Switch is on fire and the UPS don't have the power to run the systems for a long time as they are setup just to be there for the time it's takes for the diesel to start up.
I think, perhaps, that you're missing the point, at least of the article. It has nothing to do with whether to store information in memory or in the database and everything to do with the current trend of using dedicated analytics products (i.e. OLAP) to do data analysis. Whereas we used to use the same relational databases to store, retrieve and analyze all data with SQL as the Swiss Army knife that enabled it all, we're moving towards a model where the relational database is responsible for storage and retrieval of information only and dedicated analytics products have their own cache of the information for reporting and analysis purposes.
The point is that relational databases are being marginalized and one of their major selling points (i.e. the ability to analyze data based on the relationship between different types of data) is increasingly less relevant. Once you're limiting your RDBMS usage to simple CRUD operations, the rationale for choosing an RDBMS (especially an expensive one like Oracle and its ilk) over NoSQL options or open source databases with limited support for power-user options starts to disappear. MySQL may lack a lot of the features that experienced DBAs consider mandatory, but it can do INSERTs, UPDATEs and DELETEs as well as anything and it has no problems with SELECTs based on keyed columns. Similarly, Casandra, Voldemort and such can also easily support that limited subset of functionality.
That is why RDBMSs are becoming marginalized. Applications are increasingly being designed to either avoid an RDBMS back-end or to use it as simple "dumb" storage and rely on a separate analytics product to accomplish all the complicated logic that previously would be accomplished with complicated SQL and stored procedures. Beyond that, OLAP concepts allow the data-mining interface to require less development effort. It's simple to write an interface around (an) OLAP cube(s) and allow the user to choose the dimensions and measures and allow the user to pivot, drill-down and such. In fact, most analytics products do this stuff out of the box without any development necessary. With a SQL database, an interface needs to be created that will translate the user's instructions into SQL, which can often become very complex and requires significant effort to ensure that the resulting SQL will perform well.
This isn't about RDBMSs becoming unnecessary, it's about them now being best served in a much more limited role than they've previously occupied in the application architecture.
I work for a large (global) web hosting company, and I'd just like to counter the 'low cost of RAM' idea... Yes, most RAM is cheap, but when you start looking at 'large data sets', cheap is a relative term.
For example, the HP DL580 G7 can hold a Terabyte of RAM, but to do so it uses 16GB DIMMs, at $1000 each. http://h30094.www3.hp.com/product/sku/5100299/mfg_partno/500666-B21
When you add that up, it's $64,000 just for RAM in ONE server. And we don't sell it to you, (in fact we only lease it from HP ourselves) we add a ridiculous additional monthly charge to your bill, well above what it costs us. Also keep in mind, anybody spending that kind of money has a multiple times redundant system... So, no, I would not call it 'low cost'.
Experience teaches only the teachable. -AH