Slashdot Mirror
MySQL Creator Contemplates RAM-only Databases
Aavidwriter writes "Peter Wayner asks Michael 'Monty' Widenius of MySQL, 'When will RAM prices make disk drives obsolete for database developers?' From Monty's answers, it sounds like hard drives may be nothing but backups before long." From experience, I'd wager that RAM failure rates are less than hard drive failure rates, so it might also mean more stability from that perspective.
Considering the non-existent ACID support in MySQL it sounds like a good idea, it's not like MySQL will get any more errorprone than it is now...
Surely a well tuned database server stores uses quite a lot of RAM for buffering?
Well, a professional database like Oracle manages its own cache, but MySQL really only relies only on the OS-level cache. The problem with that approach is that the database knows a lot more about what you're doing, so it can make much smarter decisions about what to cache, what to age out, what to prefetch, etc. On an Oracle server, you want to lock the buffer in memory, and leave as little as possible over for the OS filesystem cache. You see, if a block doesn't exist in the cache, it has to be fetched from disk into the database cache, and if it does, the db will go straight to its own cache. Another caching layer inbetween provided by the OS is just wasted.
I don't think Monty understands any of this; in the article he seems to say that ACID, rather than being a fundamental principle of relational databases, is just something you need to do because disks are slower than RAM. The only reason that you might not want full ACID and use semaphores instead, as he suggests, is because you are only updating one record in one table in a single transaction!
Further, if he is thinking in terms of a few Gb of data, then he is a little out of touch with modern database usage. SouthWest airlines do not have a database that stores 10 bytes of data for every seat on every flight, and I have a hard time figuring out why they would want to - the database of seats would be tied into the customer records, the freight/baggage handling database, the billing records, the accounting system. That is the point of relational database, that you can query data based on data that exists elsewhere. Monty thinks in terms of databases that have only a few tables, because that's all you can do in MySQL. He says that database programmers are forced to be very careful about the consistency of their data - well those using MySQL are, but those using Oracle (or any other database with real transactions and real integrity constraints) find it's all taken care of transparently.
From the article:
Umm, no. There is no difference in the ACID algorithm whether the database is stored in memory or on disk. The only thing that is easier to do in memory is to fake it, because for low levels of concurrency you can serialize everything without anyone noticing. But that strategy will collapse under load. Far better to do it properly the first time. Yeah, it slows you don't with a single user, but when you have tens or hundreds of users connected, it'll still work.
Actually, the algorithms are the bottleneck on disk too. Monty would know this if he had a query optimizer like the one in Oracle (or had even looked at an explain plan).
Nonsense - you need ACID if there is any conceivable case in which two users might want to access the same data at the same time, or if there is any conceivable case that a write could fail, or if you want to support commit/rollback at all. In other words, if you're running a database and not a simple SQL interface to flat files. Hell, there's an ODBC driver for CSV that does all that MySQL does.
You really cant compare things like that for databases. AISITA (as is stated in the article) the big bottlenecks for both are similar in nature but orders of magnitude in scope. I currently work with a medical database where everything has to be logged. Disk access is a big factor for us, so we use fibre channel scsi (specifically Seagate 73.4GB 10000RPM) where the cost is more like 700 dollars for 70gb) (basically $10 per GB not the $1 per GB you are showing) Also there is the issue of supporting hardware but we will ignore that for the time being.
time for some napkin math:
1 512MB ecc reg pc2100 dim -> $ 78 or $156GB
1 70GB Fibre Channel Drive -> $700 or $ 10GB
Now lets factor in raid (for access speed and redundancy)
we typically put 8 drives in a bundle which tends to give us 36% of the total drive capacity (mirrored raid 5 aka raid 6 remember teh ram is ecc reg so this factoring is already in place for it)
8 * $700 -> $5600 for
36% * 8 * 70 = 200GB
This give me approximately 1GB for $28
now thats a factor of 5.6 (call it 6) in price from ram only. AND i still get a prolly 4 fold increase in throughput. Not bad at all in my book.
Bad Panda! No Bamboo for you! In matters of importance ACs will not be responded to. Want to say something critical,OK
this is true, google does hold everything in RAM, but google does not care if one of those boxes goes down and they have to rely on a couple of hours old data for searches. However, a financial institution, or even a webstore, can not afford to just lose a couple of transactions if a machine goes down. I do not think this model would work well except for Databases that are primarily read only (IE you wont have to write to disk that often), since for this to work for most DB's you are going to need an up to the millisecond snapshot of the DB. Google is in a unique position where it is not critical for its data to be 100% up to the minute, and that is why it works. There are many applications for this, but this is not really a one size fits all solution.
Just from curiosity, how much data are we talking about for a large corporation, say SW Air or BofA?
Impossible to put a figure on the total amount of data that exists within an organization, but a typical SAN in a major financial institution has terabytes online. UBS Warburg has 2 Tb in just its general ledger database. Acxion has 25 Tb in its data warehouse, which will mainly be used for queries, whereas the GL database will be more transaction heavy. SouthWest is an Oracle customer, but it doesn't say here how much data they have.
We need archival storage devices that won't lose data unless physically destroyed. We don't have them. Tapes don't hold enough data any more. Disk drives don't have enough shelf life.
DVD-sized optical media in caddies for protection, maybe.
(It's annoying that CDs and DVDs went caddyless. Early CDs drives use caddies to protect the CDs, but for some idiotic reason, the caddies cost about $12 each. We should have had CDs and DVDs in caddies, with the caddy also being the storage box and the retail packaging for prerecorded media. There's no reason caddies have to be expensive. 3.5" floppies, after all, are in caddies.)