Slashdot Mirror
MySQL Creator Contemplates RAM-only Databases
Aavidwriter writes "Peter Wayner asks Michael 'Monty' Widenius of MySQL, 'When will RAM prices make disk drives obsolete for database developers?' From Monty's answers, it sounds like hard drives may be nothing but backups before long." From experience, I'd wager that RAM failure rates are less than hard drive failure rates, so it might also mean more stability from that perspective.
But also a very strong Memory Manager. We've all seen a poorly written program corrupt memory.
but doesn't RAM need power running through it to hold its data? If this is true and we do switch to RAM for our SQL servers, all it would take is one fool to trip over a power chord (or just a power-outtage) to lose one heck of a lot of data.
RAM is highly susceptible to transient faults. Things like comic radiation at high altitudes make computing a real problem. ECC helps this but it won't totally eliminate it. With a hard drive, the probability of a hard fault goes up but a soft fault goes down.
Surely a well tuned database server stores uses quite a lot of RAM for buffering?
Nobody in their right mind would have a busy database server which accesses the hard disk like crazy. A few years back I saw Oracle servers running NT with 4GB of RAM, so I guess they're using even more now.
If you have a database that is stored in RAM and periodically written out to the hard disk (for backup reasons) then you get better performance than if you have a database that is reading and writing most of the time.
UPS would prevent the data loss, the database could be written to disk when the power fails.
I guess the issue with databases is not only speed and reliability, but a totally different ballgame called 'user-perception'. Even now, tape drives are used to archive databases; despite the fact that less than 1 in 1000 of the tape media get used for actually retrieving the data during a crash. NAS devices and the like have changed this, but the temptation remains to use tape.
I guess the RAM vs disk debate is on similar lines - but there are some vital differences:
1. Disks (esp. IDE) have become a commodity item and can be accessed by different system architectures easily.
2. IDE and SCSI standards have stood the test of time - 13 and 20 years respectively, unlike RAM wihch has evolved from Parity, non-EDO, EDO, DRAM, SDRAM, DDR-RAM, RAMBUS RAM etc., and suffers several patent and copyright encumberances.
3. Although RAM prices are driving down, the h/w to interface speciality RAM banks is proprietary and hence cost-prohibitive, and comes with attendant long-term supportability risks - think Palladium, or even Server mobos over the last 10 years. TCO for RAM based systems could thus be much higher than disk-based systems.
Overall, except for apps that need super-high speeds, and users that can risk proprietary stuff, disk-based databases shall remain.
My 0.02
If you keep throwing chairs, one day you'll break windows....
Many databases contemplate database sizes in the tens to tens of thousands of gigabytes. For smaller databases, RAM is an easy thing, and even for a small number of gigabytes it might be reasonable. For larger databases RAM would be unthinkable. The fact that a database developer doesn't know what databases are used for is disturbing.
Most modern databases also make very effective use of RAM as a cache in order to speed up queries. I don't know about MySQL since I don't use it. My guess, however, is that it does not, since that would eliminate the need for this stupid measure.
As far as reliability, RAM is more vulnerable to transient things like cosmic radiation. ECC memory will take care of most single-bit problems (there are lots of them...), but all it can do for multi-bit failures is determine that yes, your data is screwed.
Also, swapping out a bad hard disk in a RAID configuration is relatively simple and has a recovery process. Suppose your RAM stick fails; what is your recourse? You've permanently lost that data, and systems with hot-swappable RAM are much more costly than ones with similar capabilities for hard drives.
Finally, consider the problem of catastrophic system failure. If the power goes out, your RAM dies but your hard disk is still safe. If it is worse (say your facility burns down) then it is much easier to recover data from the charred remnants of a hard disk than from the charred remnants of a DRAM chip.
The idea of replacing disks with DRAMs has been around for quite a while now. But disks continue to get (a bit) faster and (much) larger. Every time the morons want to replace it they get shot down. More sensible people focus on using the resources available in ways such as caches that make systems faster and more reliable.
This is interesting. Lots of responses so far have said that putting a database into volatile memory is preposterous. But from reading the article I'm not certain if it's such a bad idea in some situations. There are often sites that have a lot of relatively static data in their databases. These sites often use a backend database because it's easier, programmatically and as far as maintenance is concerned, to do so rather than create lots of static pages. Writes to the database could be done as a pass-through so they do get written to the disk "backup". A good example may be Google's cache -- the pages do not need to be re-indexed all the time but speed is critical. If RAM can be faster and, possibly even use less power than a hard drive, then there is a benefit. In Google's case, there is no writing, only queries.
This means that in any situation where data is unchanging except for periodic updates this could be a good idea.
...have nothing to do with the medium the data is stored in! What you're trying to guard against is concurrent access of resources by transactions which in cases can cause incorrect or inconsistent results in a RDBMS. I think this article is a bit obvious for most people who've had any training in how databases actually work and I think Monty was actually pretty gracious for taking the time to give the interviewer a smidgeon of clue.
A gig is nothing in the enterprise space. What happens when a terabyte is the unit you allocate between applications or departments, and a petabyte is still big but no longer guaranteed to be the biggest on the block? Gonna walk down to the store and buy a terabyte of RAM, plus a CPU and chipset and OS capable of addressing it? This whole discussion is based on a faulty concept of what "big" is nowadays. For a truly big database, RAM isn't going to cut it. RAM and disk sizes have grown rapidly, but database sizes have (at least) kept pace. That will probably always be the case. If something came along that was even bigger than disks, but even more cumbersome to access, databases would interface to it anyway. General-purpose OS access won't be far behind, either. VM is far from dead; if anything, the change that's needed is to get rid of bogus 2x/4x physical-to-virtual ratios in broken operating systems like Linux and Windows so they can address even more virtual memory.
I worked at a company several years ago (Dolphin) that allowed just this sort of remote memory access at the hardware level. Even then, there were so many issues around consistency, varying latency (this is NUMA where access is *really* non-uniform), and system isolation (it sucks taking a bus fault because something outside your box hiccuped) that the market resisted. InfiniBand HCAs don't even do that; access to remote memory is explicit via a library, not just simple memory accesses. RDMA over Ethernet is even less transparent, and has a host of other problems to solve before it's even where IB is today; it's a step backwards functionally from software DSM, which has been around for at least a decade without overcoming the same sort of acceptance issues mentioned above.
You could start with IRAM at Berkeley. There are links to other projects as well, and some of the papers mention still more that don't seem to be in the links. A lot of what you talk about is possible, and being thought about, but a lot further off than you seem to think.
Slashdot - News for Herds. Stuff that Splatters.
Yeah, RAM is good for read-only databases, but since MySQL has never been writing to disk properly and been inherently unsafe I don't find his answer that surprising.
Real databases requires concistency!
Running the DB from RAM is nice, but as far as I can see this won;t require any changes to the software itself, you could just mount your DB on a RAMdisk and be done with it. Whats the big deal?
What MySQL and PostgreSQL really lack is the ability to replicate on-the-fly and to support running on clusters for *real* failover and fault tolerance.
For Postgres, this means multiple 'postmaster' processes being able to access the same database concurrently, and probably something similar for MySQL.
Being able to run a database on an OpenMOSIX cluster, for example, would make it massively scalable, and being able to run multiple independent machines with an existing HA (High Availability) monitoring system would provide a truly fault-tolerant database.
There are of course major technical difficulties involved in making databases work this way, but an Open Source DB that can compete with Oracle's 'Unbreakable' claims would be a huge shot in the arm for OSS in the business world.
I gots ta ding a ding dang my dang a long ling long
First, as other have said, a properly designed RAM subsystem can be battery backed up. In terms of getting the data out, loss of power to the RAM is no more catastrophic than loss of power to the CPU, the router, the computer running the middleware, or whatever. Because RAM is a purely semiconductor approach, any battery backup system can be simple and reliable.
In fact, it should not be too difficult to design a system which, in the event of power fail, dumps data to backup disk drives. To get to that state, the main system has already failed to do a clean shutdown, so this is a last resort issue.
The next thing is error detection and correction. It's true that single bit ECC is limited, but it also takes only limited resources (7 additional bits for a 32-bit subsystem, 8 for 64). Memory subsystems could have extra columns so that if bit errors start to multiply in one column, it can be switched out for a new one. Just as with any error detection and correction strategy, single bit detection in columns can be combined with further error correction down rows, using dedicated hardware to encode and decode on the fly. Just good old basic electronics.
In the worst case, it should be possible to build an extremely reliable memory system for a bit penalty of 50% - no worse than mirroring two hard drives. It won't be quite as fast as writing direct to motherboard RAM, but we don't want to do that anyway (we want to be able to break the link on power fail, save to disk, then later on restore from disk. And we want the subsystem in its own cabinet along with the batteries. No one in their right minds is suggesting having a couple of C cells taped to this thing and held on with croc clips.)
I'd even venture to suggest that most MySQL databases are not in the terabyte range, and that most databases aren't in the gigabyte range even if they are mission critical in SMEs.
Conclusion? As usual we have the people trying to boast "My database is far too big and complicated for MySQL! So MySQL sucks! My database is too (etc.) to run in RAM! So running DBs from RAM sucks!" and ignoring the fact that there are many web databases where transactional integrity is not an issue, and the market for a RAM store for databases in the low Gbyte range might actually be rather substantial.
Panurge has posted for the last time. Thanks for the positive moderations.
I like to call this number "software snobbery". Many people compare software applications feature for feature, paying no attention to what their requirements are. The fact is, a very large percent of the database market not need more than a single GB database for their current task.
Well, Monty started it by talking about SouthWest's booking system. Airlines are among the heaviest IT users in the world. If you want to drop names, be prepared to be called on your assertions by someone who does know. Someone asked me what a typical DB was in finance; I posted links to case studies on a couple of multi-terabyte databases. Also, SouthWest are an Oracle site, so they aren't even using MySQL for their seat reservations, so I don't know what Monty was talking about.
The fact is our MySQL deploys outnumber the Oracle deployments, and over time as MySQL and Postressql get better I'm expecting that MySQL will creep into Oracle space as well.
You assume that Oracle is a static target. It is not. PostgreSQL is a perfectly adequate database for small tasks; its SQL parser and core transaction processing is infinitely superior to MySQL. But both of them are beaten hands down by open source databases like SAP/DB and Borland Interbase (or Firebird, whatever it's called these days). I'd happily use either of those with databases of a few hundred M or even a few G.
Now, if open source advocates talked about those two when they talked database, I'd show some respect. But the constant worship of MySQL just shows that they don't know.
With everything in RAM, why not create true object servers, rather than distorting and maiming our object models by breaking encapsulation & imposing database layout restrictions?
http://www.prevayler.org
Others have rebutted your assertion on RAM availability.
:-)
Clearly you haven't used XP much. I've got an XP Video Server hooked up to a TV; it has uptime of around four months right now. Good luck getting a Win9x machine to do that -- 95 literally could never stay up more than 47 days, due to a clock related overflow. They've done ALOT to fix stability, and it's nothing but ignorance to claim otherwise.
It's nice to be able to finally change your IP address without rebooting, too.
95->98 was a huge jump. ME was an ummitigated disaster, but 2000 and XP have been herculean tasks that really have paid off well.
I'm a hardcore Linux/FreeBSD/OpenBSD user and programmer, but credibility demands honesty.
Linux would have many, many more viruses if it was even remotely as popular as a client platform. Since it's a popular server, it actually has more out-of-the-box remote roots (IIS notwithstanding).
Disks aren't necessarily simpler; the amount of work you need to do to keep really slow data on disk efficiently cached in RAM is monstrous. What you want to do is batch all sorts of operations together in RAM, then blast it onto the disks in an atomic operation, but do it such that if the disks crash during a blast, the data is still accurate...it's messy; disks introduce major latency within which failures can occur. RAM is so low latency that this is much, much less a problem -- the moment a batch is ready to be checkpointed, it's already practically there. So RAM approaches, beyond being faster, become simpler.
--Dan