Slashdot Mirror
MySQL Creator Contemplates RAM-only Databases
Aavidwriter writes "Peter Wayner asks Michael 'Monty' Widenius of MySQL, 'When will RAM prices make disk drives obsolete for database developers?' From Monty's answers, it sounds like hard drives may be nothing but backups before long." From experience, I'd wager that RAM failure rates are less than hard drive failure rates, so it might also mean more stability from that perspective.
But also a very strong Memory Manager. We've all seen a poorly written program corrupt memory.
*cough* Google? Big enough for ya? *cough*
"To any truly impartial person, it would be obvious that I am right."
Better make a hard disk backup... constantly!
I remember reading somewhere that, due to things like thermal radiation and cosmic rays, every so often a bit in RAM is changed by 'accident'... isn't the ECC RAM (which, IIRC, negates the effects of such interaction) horrendously expensive though, more so than the 'normal' SDRAM variants we have these days?
but doesn't RAM need power running through it to hold its data? If this is true and we do switch to RAM for our SQL servers, all it would take is one fool to trip over a power chord (or just a power-outtage) to lose one heck of a lot of data.
With our Exchange server, we use a Platypus Qik Drive to send our retrieval times through the basement. We put the database on Qik Drives (but mirror it hourly on to HDDs)...it makes our effective Exchange bandwidth limited to the gigabit ethernet port on the server.
Q: "Why do sound techs say 'check 1, 2'?"
A: "Cause if they could count any higher they'd be lighting techs."
"will power outages or system crashes make RAM only databases obsolete for database developers?" :-P
So what to do in case of reboot (Kernel upgrade)?
;)
Saving the "ramdisk" on memory sticks?
--
One by one the penguins steal my sanity...
RAM is highly susceptible to transient faults. Things like comic radiation at high altitudes make computing a real problem. ECC helps this but it won't totally eliminate it. With a hard drive, the probability of a hard fault goes up but a soft fault goes down.
Surely a well tuned database server stores uses quite a lot of RAM for buffering?
Nobody in their right mind would have a busy database server which accesses the hard disk like crazy. A few years back I saw Oracle servers running NT with 4GB of RAM, so I guess they're using even more now.
the creator is not amused.
...goes to whoever is crazy enough to put their entire database in RAM.
Now if the RAM was non-volatile and was static with the power off that would rock, but volitile RAM - are you crazy?!!
Comment removed based on user account deletion
"RAM failure rates are less than hard drive failure rates, so it might also mean more stability from that perspective" Well that is because they havnt been subjected to that sort of load as yet. RAM could pose its set of unique problems once implemented as databases.
Siggy Say, Siggy Do
we (being in the dbase 'business' for years) were also suspect of the the lack of clirking & whirring as evidence that our inf. wouldn't poof.
truth is, hard rammed data media, is possibly more reliable.
the access time/dependability make it worth investigation.
in the 'future' the counter-person will ask: do you waNT payper liesense FUDge on your chip, or the penguin drop-ins. we know which we'll choose for ours.
If you have a database that is stored in RAM and periodically written out to the hard disk (for backup reasons) then you get better performance than if you have a database that is reading and writing most of the time.
UPS would prevent the data loss, the database could be written to disk when the power fails.
See subject.
Considering the non-existent ACID support in MySQL it sounds like a good idea, it's not like MySQL will get any more errorprone than it is now...
When our site was slashdotted last year, we were able to cope with the load after putting our database into RAM. It's probably not the best solution, since the RAM would get deleted if the system crashes (or the power goes out, etc.), but it's a good temporary measure.
OLPC Australia
I guess the issue with databases is not only speed and reliability, but a totally different ballgame called 'user-perception'. Even now, tape drives are used to archive databases; despite the fact that less than 1 in 1000 of the tape media get used for actually retrieving the data during a crash. NAS devices and the like have changed this, but the temptation remains to use tape.
I guess the RAM vs disk debate is on similar lines - but there are some vital differences:
1. Disks (esp. IDE) have become a commodity item and can be accessed by different system architectures easily.
2. IDE and SCSI standards have stood the test of time - 13 and 20 years respectively, unlike RAM wihch has evolved from Parity, non-EDO, EDO, DRAM, SDRAM, DDR-RAM, RAMBUS RAM etc., and suffers several patent and copyright encumberances.
3. Although RAM prices are driving down, the h/w to interface speciality RAM banks is proprietary and hence cost-prohibitive, and comes with attendant long-term supportability risks - think Palladium, or even Server mobos over the last 10 years. TCO for RAM based systems could thus be much higher than disk-based systems.
Overall, except for apps that need super-high speeds, and users that can risk proprietary stuff, disk-based databases shall remain.
My 0.02
If you keep throwing chairs, one day you'll break windows....
Everything in pc market is going faster month by month, everything but disks. Disks are becoming a bottleneck, sure, but we dont need to replace them with RAM, we need to replace them with a good subsitute. Ibm NanoDrives could be the answer, dunno (speed like RAM, 100gb/cm, no power needed to keep data...), but, RAM is RAM, hard disks are hard disks, we need just better ones.
RAM-resident Database? Yes, that would be Google -- a massive, massive cluster of x86 boxen with a couple gigs of RAM apiece. Each gets a portion of the hashspace, leading to near-O(1) searchability. I'm pretty sure all the big search engines work this way, at this point -- the DB is checkpointed into RAM, but is never actually run from it.
:-)
Recent discussions about disks vs. CPU's have ignored the massive decreases in the cost of RAM. For a very long time, the secret bottleneck in PC's (in that it wasn't advertised heavily) was RAM. That's starting to disappear -- there's a gig in my laptop, and there's no discernable improvement in all but the most intense applications if I were to go beyond that.
Virtual Memory is already on the chopping block; any time it's imaginable that a system might need another gig of storage, it's probably worth going to the store and spending the hundred dollars.
But what if more RAM is indeed needed? One of the most interesting developments in this department has involved RDMA: Remote DMA over Ethernet. Effectively, with RAM several orders of magnitude faster than disk, and with Ethernet achieving disk-interface speeds of 120MB/s, we can either a) use other machines as our "VM" failover, or more interestingly, b) Directly treat remote RAM as a local resource -- a whole new class of zero copy networking. This Is Cool, though there are security issues as internal system architectures get exposed to the rough and tumble world outside the box. It'll be interesting to see how they're addressed (firewalls don't count).
What next, for the RAM itself? I don't think there's much that value in further doublings...either of capacity, or soon, of speed. What I'm convinced we're going to start seeing is some capacity for distributed computation in the RAM logic itself -- load in a couple hundred meg in one bank, a couple hundred meg in another, and XOR them together _in RAM_. It'd just be another type of read -- a "computational read". Some work's been done on this, though apparently there's massive issues integrating logic into what's some very dumb, very dense circuitry. But the logic's already done to some degree; ECC verifiers need to include adders for parity checking.
My guess...we'll probably see it in a 3D Accelerator first.
*yawns* Anyway, just some thoughts to spur discussion. I go sleep now
Yours Truly,
Dan Kaminsky
DoxPara Research
http://www.doxpara.com
Many databases contemplate database sizes in the tens to tens of thousands of gigabytes. For smaller databases, RAM is an easy thing, and even for a small number of gigabytes it might be reasonable. For larger databases RAM would be unthinkable. The fact that a database developer doesn't know what databases are used for is disturbing.
Most modern databases also make very effective use of RAM as a cache in order to speed up queries. I don't know about MySQL since I don't use it. My guess, however, is that it does not, since that would eliminate the need for this stupid measure.
As far as reliability, RAM is more vulnerable to transient things like cosmic radiation. ECC memory will take care of most single-bit problems (there are lots of them...), but all it can do for multi-bit failures is determine that yes, your data is screwed.
Also, swapping out a bad hard disk in a RAID configuration is relatively simple and has a recovery process. Suppose your RAM stick fails; what is your recourse? You've permanently lost that data, and systems with hot-swappable RAM are much more costly than ones with similar capabilities for hard drives.
Finally, consider the problem of catastrophic system failure. If the power goes out, your RAM dies but your hard disk is still safe. If it is worse (say your facility burns down) then it is much easier to recover data from the charred remnants of a hard disk than from the charred remnants of a DRAM chip.
The idea of replacing disks with DRAMs has been around for quite a while now. But disks continue to get (a bit) faster and (much) larger. Every time the morons want to replace it they get shot down. More sensible people focus on using the resources available in ways such as caches that make systems faster and more reliable.
z/OS Data In Virtual.
You're still going to need to write any changes to the disk or else you could risk loseing everything if something (hardware or software) failed.
useing ram to read out from the database wouldnt be a bad idea tho.
This is interesting. Lots of responses so far have said that putting a database into volatile memory is preposterous. But from reading the article I'm not certain if it's such a bad idea in some situations. There are often sites that have a lot of relatively static data in their databases. These sites often use a backend database because it's easier, programmatically and as far as maintenance is concerned, to do so rather than create lots of static pages. Writes to the database could be done as a pass-through so they do get written to the disk "backup". A good example may be Google's cache -- the pages do not need to be re-indexed all the time but speed is critical. If RAM can be faster and, possibly even use less power than a hard drive, then there is a benefit. In Google's case, there is no writing, only queries.
This means that in any situation where data is unchanging except for periodic updates this could be a good idea.
http://www.imperialtech.com/success_ebay.htm
The basic idea: use solid state ram drives (with separate power supply) for your busy tablespaces and your redo logs.
This leverages 'cheap ram' technology with existing (and proven and scalable) db architecture.
For ebay, for example, they might store 'active items' in 'ram-drive-backed' tablespace and 'old items' in the 'hard-drive-backed tablespace'.
These solid-state drives are expensive, but additional Oracle licenses (or moving from 'standard' to 'enterprise' or to 'clustered') are very very expensive.
bill m
There is already a leading in Memory database that is extremely fast. Check out TimesTen. That is what we use. There is also another one called Polyhedra. But the redundancy on Polyhedra doesn't appear to be as good as TimesTen, and it doesn't support Unicode either.
I imagine that most databases are read far more than they are written, if all new writes or modifications of data were immediately written to hard drive, wouldn't that create a minimal performance hit while preventing any data loss due to catastrophe?
...have nothing to do with the medium the data is stored in! What you're trying to guard against is concurrent access of resources by transactions which in cases can cause incorrect or inconsistent results in a RDBMS. I think this article is a bit obvious for most people who've had any training in how databases actually work and I think Monty was actually pretty gracious for taking the time to give the interviewer a smidgeon of clue.
There are some cool ideas there. They use two copies on disk for backup in case of system failure. Because of this they don't have to do page-latching.
In some configurations, though, this is irrelevant, because write transactions lock the whole database! Because they know all transactions will be extremely short, this is faster than locking at page or row level.
You are spot on. The guys at MySQL have never had a clue about relational databases, and this probably explains why MySQL is not a relational database. :)
I told my boss that it was a very Bad Thing as the stores could lose data so easily. He told me several things:
- Running entirely in RAM, the system was very very fast. When the system could smoke a more expensive IBM PC-XT running a dBase app, he could sell more systems
- Every system would have a UPS as he refused to sell them without
- He signed my paycheck, not the other way around
As best I can figure, Darwin was more interested in awarding JATO assisted drag racers back then because we got lucky and actually had more trouble with the systems using hard drives. That was back during the heyday of the small mom-and-pop video stores. The last of those RAM disk based systems that I knew of converted to a "real" system in 1993. I believe they were assimilated by one of the national chains soon after.You either believe in rational thought or you don't
Memory sounds like a good idea in theory .. but what about power failures or momentary blips .. UPS can help but not eliminate that risk.
.. with some options for active standby and hot replacement available.
.. as the number of individual ram components increases the risk of a single bit non ecc correctable fault scales up accordingly .. such that with 8 gig + arrays the chance of uncorrectable error approches 50% per time interval
.. several of the Sun Fire 6800 series machines I work with on a regular basis develop these kinds of errors occasionally .. though with Sun the hardware is smart enough to map around the error and make relevent OBP console & syslog entries.
A recent hardware write up I read from HP / Compaq has ram partitioning / raid'ing on some of the higher end x86 servers
Another little burb was that with ram
I know memory can develop stuck bits without any warning
way to go the mod who uses mysql as the db for his homepage and thinks it can do it all, parent is +1, Informative
A quick run of CDW.COM should answer your question. The cheapest 1GB SDRAM DIMM runs a little over $220, making a terabyte a little under a quarter million dollars. On the cheap end of SCSI hard drives, about 20GB can be had for about $120, making a terabyte RAID about $6,000.
Since hard drive space is keeping pace in increasing size and decreasing price, while data storage requirements are shooting further through the stratosphere everytime a manager or executive utters the words "data warehouse," the most economical fail-safe solution will always win and a 27:1 cost ratio isn't going to convince anyone to switch. Cheap IDE & SCSI arrays will continue to dominate OLTP applications for the reasonably forseeable future until such time as that ratio is cut to 1:1, which will happen about the time probe storage atomic memory hits CompUSA.
Ive used (Praedictus' (goofy name i know..)) database. resides totally in memory, used (in my case) to do Statistical Process Analysis in a manufacturing environment.... closed source.
I'm confused. I actually haven't used MySQL much, and someone else can clarify its current ACID compliance. My application involves multiuser financial transactions. When making my DB selection a couple of years ago, at that time it was claimed that MySQL had some ACID deficiencies that made me nervous. I settled on PostgreSQL, which I'm very happy with.
But there's a lot more to ACID than just keeping RAM and disk in sync, and I don't see how RAM would make ACID that much easier, and certainly not "almost trivial". You still have all the transactional semaphores, record locking, potential deadlocks, rollbacks, etc. to worry about. In fact I don't see why you wouldn't just have the RAM pretend to be a disk and be done with it, since the disk version already has stable software. Then, if it is important to increase performance further, RAM-specific code optimization could be done over time, but slowly and carefully.
I'm sorry - I really don't want to get into a religious war here, but the interview didn't do much to bolster my confidence in MySQL for mission-critical financial stuff. Educate me.
We have a highly specialized database (custom code) of the entire human genomic sequence (DNA and translated protein) resident in RAM and it totally works. It works better than anything else out there. The nay sayers can whine all they want about power failures and cosmic radiation, but from my perspective it WORKS. It works really well. Ask any genomics researcher out there where they do their human genome searches :-)
If you've got enough ram for your database to fit in, why not mmap it and do a simple search? It tends to take up much less memory than a database and you can search a whole lot of records in the time it takes to do a context switch (which is what you get when you use a socket to talk to the database program).
Here's a tip kids, when you stop playing around with the toy databases, give us a call and we'll show you why you still need hard drives.
Back in the early 90's IBM added a machine instruction to their mainframes called DIV. It treated data in a file system as if it where in virtual mememory - ie addressRecord[12345] appeared to the program as an in memory array, but was backed by disk storage - the same format that was used for paging virtual memory - brilliant. It's a shame it never caught on - it would make advances like this transparent in implementation. Well I guess you can't really say it never caught on - it was a big reason IBMs mainframe databases outperformed everyone else for so long.
Is there a similiar kind of instruction on Intel? It's probably too late though - indexed arrays have become less useful since associative array patterns have become better defined. A hardware implementation (RAM) of JDO would be interesting.
slashdot troll = you make a compelling argument I do not like the implications of.
Why not a use a ram based hard drive? The drive would have a battery backup and the speed of ram.
Plus with new standards like fiber channel and varies SCSI you wouldn't lose much if any speed.
You say things that offend me and I can deal with it. Can you?
Hey! If it's good enough for Slashdot, I _HAS_ to be good. Right?
Yeah, RAM is good for read-only databases, but since MySQL has never been writing to disk properly and been inherently unsafe I don't find his answer that surprising.
Real databases requires concistency!
There are components of ACIDity that would be implmented very differently for RAM-persistent databases than for disk-persistent ones. Maintaining ACIDity on disk-persistent databases requires complicated algorithms to mitigate the disatrous disk seek times. These complicated algorithms would be rendered unnecessary if disks were no longer used.
For example, disks have incredibly slow seek times and much better bandwidth; therefore it's far cheaper to write things to disk in big chunks. The purpose of write-ahead logging (or "redo logging") is to mitigate the performance impact of slow seek times by blasting all the transactions to disk at once, in the redo log, thereby avoiding the slow seeks that would be required by putting each transaction in its proper place. Putting the transaction data in its proper place is deferred until after the load has died down somewhat. This could be seen as exchanging seek times for bandwidth.
This redo log mechanism would be unnecessary for ram-persistent databases. It's a significant source of complexity that would be obviated by the removal of disks. And that's just one example of complexity required to get adequate performance from disk, a medium that has disastrously slow seek times.
The new Sparc chip was supposed to have some nifty way of massively boosting RAM access times. Without that sort of advance, RAM-resident databases aren't much of a win over RAM-cached DBs (what everyone does now), sicne the win for fully resident DBs (which Oracle can do if you force it) is that you can lock huge sections of RAM for just the database, but that leads to the problem that for very large amounts of RAM the latency is starting to get too large.
Hopefully this will be a solved problem soon.
the interview didn't do much to bolster my confidence in MySQL for mission-critical financial stuff. Educate me.
Cancel that financial adjective. You're thinking too narrowly.
Infuriate left and right
The ability to replicate a database, in real time, onto a slower system, would seem to be a key item.
As long as the DB is replicated onto a slower HD based DB. This would have other advantages, IE duplicating a DB to a remote site for disaster recover purposes.
Search google for many products available. One product we looked at is from TimesTen. It was very expensive!
The primary reason people use it is because of performance. They are situation where data update and access time is very important but the data is small enough to fit in memory and persistence is less important. One example is for keeping real time system status.
At present, the principal performance bottleneck of a relational database is disk seek time. Since disks have disastrous seek times, database servers often have an incredible number of disks (hundreds or more), in order to have those disks seeking in parallel, thereby mitigating disastrous seek times of individual disks.
These hundreds of disks often have very little on them. The purpose of having lots of disks isn't for more storage, but for more drive heads, because lots of heads can be seeking in parallel.
Oftentimes, OLTP databases aren't even that large, and most of the space on the disks is unused.
RAM has a seek time (latency) that is about 1/10,000th that of disk. Since latency is the primary bottleneck, this could vastly improve the speed of databases. And database servers could become cheaper, since all those unnecessary (and unfilled) disks, used just for parallel disk seeks, could be thrown away.
The difficulty with RAM is that it loses its data after you turn off the power. This is the reason disks are still used for databases. Losing any data because of power loss would not be acceptable to any financial institution. Battery backups etc would never be deemed sufficient.
The moment we have RAM that retains data despite power loss, disks will no longer be used in database servers. This will be true even if such RAM is vastly more expensive than the disks they replace, because the RAM would be 10,000x as fast and so would render unneccessary the huge number of "redundant" disks kept around simply so data can be read in parallel from multiple disks.
"Flash RAM" and other current nonvolatile memory technologies will not suffice, for several reasons. First, they can only be written a limited number of times before failure. Second, their write speeds are very low, and OLTP databases are write-intensive.
I believe it surfaced a while back on /. - can't find any links at the moment, but AFAIK the entire Google index is stored in RAM.
grisha.org
I fondly remember learning to program in C with a recoverable RAM disk on the Amiga. I was able to copy most of my source, headers, libs and even the compiler into VD0: and do everything from RAM. Reset the computer due to a crash and most of the time the RAM disk was intact. I'm sure that quick cycle helped me to learn C faster.
How much does TimesTen cost?
The price is not listed.
Running the DB from RAM is nice, but as far as I can see this won;t require any changes to the software itself, you could just mount your DB on a RAMdisk and be done with it. Whats the big deal?
What MySQL and PostgreSQL really lack is the ability to replicate on-the-fly and to support running on clusters for *real* failover and fault tolerance.
For Postgres, this means multiple 'postmaster' processes being able to access the same database concurrently, and probably something similar for MySQL.
Being able to run a database on an OpenMOSIX cluster, for example, would make it massively scalable, and being able to run multiple independent machines with an existing HA (High Availability) monitoring system would provide a truly fault-tolerant database.
There are of course major technical difficulties involved in making databases work this way, but an Open Source DB that can compete with Oracle's 'Unbreakable' claims would be a huge shot in the arm for OSS in the business world.
I gots ta ding a ding dang my dang a long ling long
and it will become like the king of message boards.
Bwahahah Bwahahah!
So why would anyone use MySQL when there is Postgres? Postgres is such a far superior product. I am so confused by such actions.
With a dynamic RAM system (DRAM also isn't all that reliable...SRAM is better, and SRAM is very expensive) you are highly vulnerable to this.
I suppose you could implement a kind of write-back system to the disk where you pile up things in some kind of buffer, but under heavy load, you're going to overwhelm it. Or at the very least cause the thrashing that this supposedly helps avoid.
Really though, at that point, you're just using the RAM as a cache. While this sounds all nice and fanciful, it doesn't sound to me like he's thought it all the way through. Perhaps some people who know more about database design can point out any simple mistakes ive made...
-
I posted something else along the lines of this, but how would you do it under heavy load? The disk is so enormously slow compared to RAM, you'd overwhelm whatever buffer you're using to do the write-back. You'd have to throttle back requests on the RAM, thus negating the performance increases.
-
depends on the application and the definition of reasonable. if you're trying to store your 37,000 images of p0rn, well ya, RAM based won't work. But if you're dealing with primarily text information in a well designed database you would be suprised how small it can be.
If you disagree, you can suck my 14 year old MySQL powered cock!
Any Perl programmers in the audience may wish to check out DBD::RAM. From the CPAN documentation: "DBD::RAM allows you to import almost any type of Perl data structure into an in-memory table and then use DBI and SQL to access and modify it. It also allows direct access to almost any kind of file, supporting SQL manipulation of the file without converting the file out of its native format." More information here
First, as other have said, a properly designed RAM subsystem can be battery backed up. In terms of getting the data out, loss of power to the RAM is no more catastrophic than loss of power to the CPU, the router, the computer running the middleware, or whatever. Because RAM is a purely semiconductor approach, any battery backup system can be simple and reliable.
In fact, it should not be too difficult to design a system which, in the event of power fail, dumps data to backup disk drives. To get to that state, the main system has already failed to do a clean shutdown, so this is a last resort issue.
The next thing is error detection and correction. It's true that single bit ECC is limited, but it also takes only limited resources (7 additional bits for a 32-bit subsystem, 8 for 64). Memory subsystems could have extra columns so that if bit errors start to multiply in one column, it can be switched out for a new one. Just as with any error detection and correction strategy, single bit detection in columns can be combined with further error correction down rows, using dedicated hardware to encode and decode on the fly. Just good old basic electronics.
In the worst case, it should be possible to build an extremely reliable memory system for a bit penalty of 50% - no worse than mirroring two hard drives. It won't be quite as fast as writing direct to motherboard RAM, but we don't want to do that anyway (we want to be able to break the link on power fail, save to disk, then later on restore from disk. And we want the subsystem in its own cabinet along with the batteries. No one in their right minds is suggesting having a couple of C cells taped to this thing and held on with croc clips.)
I'd even venture to suggest that most MySQL databases are not in the terabyte range, and that most databases aren't in the gigabyte range even if they are mission critical in SMEs.
Conclusion? As usual we have the people trying to boast "My database is far too big and complicated for MySQL! So MySQL sucks! My database is too (etc.) to run in RAM! So running DBs from RAM sucks!" and ignoring the fact that there are many web databases where transactional integrity is not an issue, and the market for a RAM store for databases in the low Gbyte range might actually be rather substantial.
Panurge has posted for the last time. Thanks for the positive moderations.
We need archival storage devices that won't lose data unless physically destroyed. We don't have them. Tapes don't hold enough data any more. Disk drives don't have enough shelf life.
DVD-sized optical media in caddies for protection, maybe.
(It's annoying that CDs and DVDs went caddyless. Early CDs drives use caddies to protect the CDs, but for some idiotic reason, the caddies cost about $12 each. We should have had CDs and DVDs in caddies, with the caddy also being the storage box and the retail packaging for prerecorded media. There's no reason caddies have to be expensive. 3.5" floppies, after all, are in caddies.)
Valid points, of course :) But you have to admit that for simple home pages (and not corporate databases) MySQL is simple, to-the-point, and easy to use.
And free, although many other ones are free as well. (I wouldn't want to run Oracle@Home ... of course, postgreSQL is also free, and I hear it's more mainstream as far as true database functionality.)
"Careful" people can enforce their own data integrity - obviously, it gets harder as the size (number and complexity of tables, I mean) of the database expands.
Can you tell I use it myself? :) You sound like you have experience with other database systems, how difficult do you think it would be to port an existing MySQL+PHP system to PostGreSQL or something similar?
keep up the good work!
Lucent had an in-memory database that they used a couple of years ago--it was called DataBlitz.
They offered me a job (I was also talking with the folks at TimesTen).
Main memory databases are great for a couple of tasks
1) Native hardware
2) Serving up static data accessed through a relational model
3) Being a front end for a (standard) relational databse
4) Being a nice programmatic way of defining data structures in your own application. (Wouldn't it be nice to be able to have tables and joins as 'regular' data structures in a C++ program?)
The internal technology is quite different--the bottleneck usually is CPU rather than disk related, even though most of the CPU instructions that deal with minimizing disk usage have been removed.
The field is really exciting, and I can't wait until the prices of the products come down enough so that I can just go otu and buy it.
...But such a condition (DB in RAM) will make his product pretty much obsolete.
The Prevayler Project is a RAM-only Java persistence project that works and is so simple not a single bug has been found in the production release.
3000 times faster that MySQL (9000 times faster than Oracle) even with the database in caches entirely in RAM simply because of the JDBC overhead that is eliminated .
The only sticking points I've seen are:
1. Normal PC's boards generally will only take 1GB of RAM. Sure there are thos expensive Sun machines...
2. Querying objects in an efficient manner.
3. Others, but I've gotta take a dump real bad...
Prevayler already does this. It's written in Java.
CAn'T CompreHend SARcaSm?
Pardon me, but I have lived under assumption that hard drives are indeed a hack to solve the problem of high RAM memory prices and their volatility. If RAM would have been cheap and non volatile from the beginning, it's not like hard drives would have been so widely spread by default. It's the same with main memory and cache memory. Why would anyone use slow main memory if fast cache-type memory would be as cheap?
(This sig intentionally left blank)
Cosmic Radiation? Isn't that how the Fantastic Four got started? I've just named my four Dimm modules Ben, Susan, Reed and Johnny. I can't wait to see my RAM's new super powers.... What DB wouldn't benefit from setting itself on fire at will? Yeah, baby...
What word rhymes with buried alive?
High-end systems, from IBM and other vendors, are capable of transparently dealing with RAM failures. They can detect and correct errors, even if caused by the total failure of a DRAM chip. They include built-in spare DRAM chips that can be electronically swapped for bad chips. The memory boards can be swapped out when it is convenient for the customer.
Mea navis aericumbens anguillis abundat
Several companies are already doing this. As one poster stated before, Lucent is doing this, however the leader in this type of RDBMS is a company called TimesTen. http://www.timesten.com/ It was originally part of HP and it was spun off into its own company. Its a rather brilliant product. It only uses the disks for logging and disaster recovery. The unfortunate thing about the whole deal is the volatility of the ram and the fact that it is still cheaper to have 1TB of disk space for a huge database then it is to have half as much ram.
and if your power outage or other complete system failure occurs before the disk operations are complete?
-
That is plain not true. MySQL does have it's own query caching. Even the most entry level databases have that.
The problem with that approach is that the database knows a lot more about what you're doing, so it can make much smarter decisions about what to cache, what to age out, what to prefetch, etc.
You're right. Accept MySQL does have query caching.
Further, if he is thinking in terms of a few Gb of data, then he is a little out of touch with modern database usage.
I like to call this number "software snobbery". Many people compare software applications feature for feature, paying no attention to what their requirements are. The fact is, a very large percent of the database market not need more than a single GB database for their current task. Why have a bunch of databases around our organization that are all a few megabytes large. For the large databases ( ERP ) we use Oracle. The fact is our MySQL deploys outnumber the Oracle deployments, and over time as MySQL and Postressql get better I'm expecting that MySQL will creep into Oracle space as well.
Based on upvotes, Ageism is the only "-ism" Slashdotters care about and think isn't SJW
Due to contractual obligations, I can't name names. So, keep that in mind.
One client, an NYC financial house, has a distributed database with ~40TB in it at any given time, and certain activities temporarily (couple of days) occupy another 2-5TB. (Oracle)
A former full-time employer of mine, a dot.com that still exists in a much smaller (10% of peak number of employees) form, has about 1.5TB under database management. (Oracle + mysql + Redbrick + other proprietary)
A tiny, 3 person firm of consultants (me and my partners) have about 80GB of data. (Postgres)
Take that for what you will.
A place I worked at a few years (about 5-6 I guess) ago kept their databases on VMS systems completely in RAM... It was a couple terabytes at the time I think, and took up a good section of the data center floor...
And when you talk about RAM failure rates, though, you have to take into account power failure, power supply failure, and wiring glitches, which of course, they did there. Redundant UPSes of redundant UPSses, and the largest generator I'd ever seen, and ever did see afterwards, (even at much larger installations)...
With everything in RAM, why not create true object servers, rather than distorting and maiming our object models by breaking encapsulation & imposing database layout restrictions?
http://www.prevayler.org
He says that database programmers are forced to be very careful about the consistency of their data
I think you will find that by 'database programmers', he means the developers that are responsible for actually coding the database engine, not developers responsible for writing applicaitons using the engine..at least that was my take on it. I must admit I read that sentence a couple of times as well.
I read it and read it again, and I still feel you are drawing the conclusion that you want to not based on what the sentence is saying. In particular, "consistency of their data." What is "their data" in the context of someone who programs a database management system?
Prevayler is a system where your java objects persist as objects in RAM. No need for a database at all, retains the goodness of OO design.
Their bold claim: "Queries with Prevayler are more than 3000 times faster than querying MySQL through JDBC."
You manipulate them via serialised commands which are logged and you take a snapshot every day or so. If there's an accident, you reload the last snapshot and replay the command log until you catch up and then you're off again.
Cheers
Large financial houses have had these type databases for many years already. Most were developed in-house and the public never saw them. Now-a-days there are a few companies in the buissness and by far the best is TimesTen. I've had extensive experience with them and there product is fantastic for many application. These are by far the future of databases and I do belive that sybase, oracle etc will turn into backup systems for most applications that need any form of speed.
Modern storage solutions (like EMC) use redundant battery backed ram to buffer writes, greatly reducing perceived write latency. This gives you a lot of the performance gain of a ram only database, and also scales very well to large loads. (in fact, when choosing RAID stripe size you take into account whether writes are buffered; if not, keep stripes small for log files)
...most databases don't cleanup indexes after deletes, forcing periodic rebuilding. Other index schemes not generally considered because of poor locality prinicles could be considered. Note that Hash Indexes would probably still use Linear Hashing.
If you know that your data will always fit into available ram then there are a number of performance optimizations that can be done. I'm not sure about ACID becoming "trivial"; You still need most of the same db components: indexes, lock managers, operation journaling, etc. But many of these could be greatly simplified:
1. Page/Buffer Manager Eliminated. Since no disk IO will be required for the normal running of the db, there will be no need for a page manager. This eliminates complexity such as block prefetch and marking and replacement strategies. In fact, the data will probably not be stored on pages at all. Details such as block checksum, flip flop, log position, page latches etc can all be removed. The values in the rows would be sitting in memory in native language formats rather than packed making retrieval much faster. There would be no need for block chaining.
2. More flexible indexing. Since it is not necessary to store data in pages, traditional B-Trees are not absolutely required. Other index structures like AVL trees would be faster and might allow better concurrency. These trees would also be easier to keep balanced
3. Lock Manager Simplified. Row level locking (and MVC) are still desired features, but keeping the locks all in memory simplifies implementation. Oracle and InnoDB store lock information in the blocks (associated with transaction) to allow update transactions larger than memory.
4. Log manager simplified. You will still need journaling capability for rollback, replication, recovery from backup etc. But the implementation of the log need not be traditional. Any structure that maintains information about transactions and contains causal ordering will do. Techniques such as keeping old versions of rows adjacted to current versions that are unacceptable for disk based databases (ahem, Postgres) could be used.
Although these may seem like small things, they can add up: less code to run is faster code. A company called TimesTen offered a product that they claimed was 10x faster than Oracle using an all memory DB. Generally the corporate world doesn't care to split hairs. They want something that works, and they are willing to throw some money and iron at it. Thats why battery backed ram in the disk controller to buffer writes is probably going to be fine for now.
A last note: modern databases already know to not bother with indexes when a table is sufficiently small.
JJ
Speaking of Lucent/Bell Labs...
This is one of the integral design choices in Plan 9. RAM is considered nothing more than a cache for the hard drives, and the hard drives are considered nothing more than a cache for the gigantic WORM drive backups (which happen nightly, and are made available as part of the normal "filesystem".
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
Gee... Gee... Nome? Nay? Sayers? Sooth?
Not everyone needs Ora$le to do his, and especially her, database needs. I don't need it, and don't want it. I have what I need and it's a LOT faster and cleaner than any query language from the 70s designed for mid-level managers to use -- sysr that.
Blinn's Law is what those in the graphics industry know as the counterpoint to Moore's Law. It states, in its original form: The amount of time it takes to render a frame is constant over time.
The point is that computers are getting faster, but we're also asking them to do a lot more, and these effects cancel each other out. Consider, for example, how long it takes your word processor to load. It's probably the same amount of time that it took 15 years ago.
So how long until RAM-only databases are practical? For some databases, right now. The database which sits behind your typical weblog, for example, could probably work today. For your typical enterprise application, it will never happen, because what is "typical" is a constantly moving target.
sub f{($f)=@_;print"$f(q{$f});";}f(q{sub f{($f)=@_;print"$f(q{$f});";}f});
Google "in memory database toolkit"
Hmmmmm,
Ahhh,
Yes, indeed, certainly a pleasent thought.
*sigh*
Not too bad, probably.
Unfortunately, AFAIK PHP doesn't have something like Perl's DBI, so you have to re-write all your db function calls, etc, instead of just changing the connection string and type like you would in Perl/DBI.
Of course, it may be (slightly) more efficient without the DBI type interface, but it'd still be nice if PHP gave you the choice.
Nothing to see here; Move along.
Well, in theory, all of my DB function calls are passed through a single function, anyway. (I like to write easy-to-modify code ;-)
Would the SQL be identical? I don't do anything too stressful, not too many cross-table selects or anything.
all the above comments, youse guyse are arguing about whether this would be suitable for large enterprise DB systems...
wrong! there is a definite need for RAM resident DB in embedded systems. from handhelds to core routers with 200k-entry BGP tables...
berkeley DB and a few others are already there, mr MySQL is on the right track. but for the wrong reason; he doesn't understand this market yet. not surprising, no one else does either.
Should be (as long as you stick to standard SQL); it's just all the functions are quite different (e.g. pg_connect vs mysql_connect, etc.), and the parameters may be different.
Nothing to see here; Move along.
A 16GB superssd drive plus fibre channel switch and card costs a great deal of money.
This is definately NOT for a small company...on the other hand, an 8GB database would fit very nicely in RAM on a 64Bit computer, and the Opterons will be a decent price when compared to Sun and IBM.
Do you know if there is an equivalent to the mysql_fetch_object function? It rocks. :)
:)
Oh, wait, php.net rocks - pg_fetch_object. Kickass. I might have to try it one day
The very idea of click-click-doubleclick MCSEs giving every Mom and Pop shop a RAM database is scary as hell, lots of businesses will go bankrupt when slack-jawed yokel says, "Why y'all I can here ye speed up all ye databases y'all YEE HA, lookey here it's a called a RAYME-data-er-base YEEEEHA!"
A caveman dreams of being us, the incalculable power and riches. We dream of being Q, then what?
If Ram cost are going down so quickly, I think it might be a good idea to get special harddrives: half Ram, half old fashioned platters. Let the harddisk use the RAM for all the read and write actions, and implement that all write actions are replicated to the old fashion platters.
For someone using these, it works just like harddisk (no changes, settings etc. in software neccesary). But you do get the extra speed.
For very reliable disks, it might be interesting to add some build in some kind of battery to make sure that at a powerfailure the last write actions can be sent to disk as well.
---
YOU FAIL IT!
I'd wager that RAM failure rates are less than hard drive failure rates
While this may be true now, I'm concerned about the reliability of RAM in years to come. Noting how reliability is on the decline in almost every other category of hardware, what makes RAM so special that it should defy the trend? I refer for example to hard disks, now warrantied to only a year; 3.5" disks, where the term "certified 100% error free" seems only to refer to the spelling on the box; monitors that are outlived by those twice their age.
ADODB is a library which lets you avoid rewriting db function calls to switch databases.