Slashdot Mirror
"Slacker DBs" vs. Old-Guard DBs
snydeq writes "Non-relational upstarts — tools that tack the letters 'db' onto a 'pile of code that breaks with the traditional relational model' — have grabbed attention in large part because they willfully ignore many of the rules that codify the hard lessons learned by the old database masters. Doing away with JOINs and introducing phrases like 'eventual consistency,' these 'slacker DBs' offer greater simplicity and improved means of storing data for Web apps, yet remain toys in the eyes of old guard DB admins. 'This distinction between immediate and eventual consistency is deeply philosophical and depends on how important the data happens to be,' writes InfoWorld's Peter Wayner, who let down his old-guard leanings and tested slacker DBs — Amazon SimpleDB, Apache CouchDB, Google App Engine, and Persevere — to see how they are affecting the evolution of modern IT."
FTA: "The world won't end if some snarky, anonymous comment on Slashdot disappears." :/
What? Nothing more important than anonymous slashdot trolls to moderate
Is it just me or did this article go out of its way to insult people who use "traditional" RDBMSs?
I mean, I'm well versed in SQL and data consistency et al, but I'm still more than willing to consider new technologies. What the hell?
It's better to vote for what you want and not get it than to vote for what you don't want and get it.
- E. Debs
Now that disk space is so cheap and many of the data models don't benefit as much from normalization, ...
You don't want to store the same data in multiple places. Your query might run faster, but your data integrity is going to suck.
And, uh, I have the pleasure of working now with a huge data warehouse that hasn't normalized status codes, so instead of quickly searching for an integer, the queries run slow as hell scanning char fields. It's not good.
Whale
Like the article says, "The world won't end if some snarky, anonymous comment on Slashdot disappears."
Slacker DBs like CouchDB and SimpleDB, have taken off for the simple reason that most developers have absolutely mediocre database knowledge or skills, and rather than learning it's just as easy to just wave it all off as obsolete.
It's no surprise that the creator of CouchDB, for instance, hadn't a clue about databases when he began his project. All of that built up knowledge just ignored while someone invented their own, and it's as rational as rolling your own encryption from scratch without the slightest clue about encryption algorithms or theories.
Either is cool with me, as long they are cool and takes care of business, you know what I am saying?
It's all good.
Fuck systemd. Fuck Redhat. Fuck Soylent, too. Wait, scratch the last one.
"tools that tack the letters 'db' onto a 'pile of code that breaks with the traditional relational model"'
If "database" were intended to mean only "relational database", we wouldn't have had any need for the latter term...
the article is right that in some cases it doesn't matter if a transaction is lost. but in any case where money is involved it's a must. you can't just start a fund from your Oracle or SQL Server savings to pay for mistakes because it will kill your brand and you may lose a lot of future business. and any savings will be eaten up by the extra cost to hire people to solve all the data problems
i've seen this. no constraints on the data that is orginally put in, not enough referential integrity and you get customers opening up a lot of trouble tickets and you end up hiring people to clean up the data every time a mistake is found
The problem of distributed consistency has kept researchers occupied for quite a while. For example, see project Scalaris. They are using a distributed hash table to distribute data among many nodes. This should be relatively easy, at least once you have a good hashing function on your hands. But a lot of research has been done on P2P networks during the last decade, so there is quite a lot of stuff to read and take ideas from.
The interesting part is that it can maintain consistency and support ACID properties. From the site it appears that they accomplish that by using a modified Paxos Algorithm which basically is a way to maintain consensus among many different peers in a non-Byzantine system (this means that there are no malevolent peers in the system -- peers can break down and cease working but not sabotage the system). Leslie Lamport of Microsoft Research has done a lot of work on this, anyone interested may take a look at his papers, very advanced stuff there.
Relational DB? People forget Network Model Databases (http://en.wikipedia.org/wiki/Network_model) and flat databases.
Network model databases will outperform relational all the time. You just don't have the same flexibility.
Newer models are not based on the design or performance issue, but the distribution of the data. These are not invalid reasons, but the old issues still apply.
I have had arguments with people who consider PC programming different from mainframe. The same rules apply. The difference is that many PC programmers are just sloppier. When you have cheap CPU and memory, people don't analyze and optimize as much.
Fight Spammers!
I've never understood the UNIX world's fascination with relational databases.
Speaking as a programmer in mainframe online transaction environments for the past 20+ years, I've become very familiar with very fast and simple database systems like the "freespace" files we use on the Unisys mainframe platform.
We don't need relations for real-time processing. Most programs just need a place to keep data, and a simple key to retrieve that data. Some efficiency in disk usage is nice, but the primary design factor is performance.
A freespace file is a collection of pre-allocated fixed-length records of various sizes (e.g. 256 bytes, 512 bytes, 1024 bytes, 2048 bytes, 4096 bytes, and 8192 bytes). Each record size is a assigned a type number (e.g., 1 through 6 in the above case), and a given file is created and pre-allocated with a mix of various records depending on the usage pater for that particular file. If you know all you need is tiny records, create a file containing a few hundred or thousand type 1 and maybe 2 records.
Records not allocated are filled with a deallocated fill pattern.
A program uses a record by performing a Write New operation. That tells the database manager to find a record in that file closest and >= to the size required, stick the presented buffer in the record, save it, and return a key to that record to the calling program. Typical key format is where Record Number is a number from 1 ... n. If your file has 1000 Type 3 records, it'd be from 1...1000 or 0...999.
To read a record, use a key from a previous Write New (stored away somewhere), perhaps in another file) to read that record from a file. Length is not required.
Programs use a very simple read-and-lock mechanism when modifying existing records. If one program has a record locked, another program must wait. Not a problem with intelligent coding.
We've used this system in airline systems for 40+ years. It works well. Sometimes an environment has robust commit and rollback/recovery features to allow for an entire series of changes to be rolled back on error, sometimes not. It doesn't seem to matter that much, especially for transient data like weather, flight schedule data, etc.
I would LOVE to see a freespace database ported to Solaris, personally. We'd use it heavily. :-)
Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
The Theorem Theorem: If If, Then Then.
I'm a DB admin, and I use things that aren't toys; but what I've heard here is kinda harsh.
Look, it's all about "right tool for the right job." Why do you need a nuclear-powered drill that can make a tunnel from here to China, when really all you needed was a shovel?
For most daily projects that have small amounts of data, they may be using something like Crystal Reports or Excel or SPSS that just does all the number-crunching client-side anyway. You don't always need Oracle or [favorite DB flavor] for that.
"They said I probly shouldn't fly with just one eye," "I am Bender. Please insert girder."
When I saw the title I thought "I'm old-guard". Then I read the article and JOINs are a key concept to the old-guard.
My first few DB apps involved using a b-tree or ISAM library (or writing our own). Then the "new guys" started wanting to pay for a server that did JOINs. We did JOINs, just at the app layer and without the guaranteed consitency that a good relational design gives you. And getting a server that does it was expensive.
I wouldn't want to go back to pre-relational server days, but am also very thankful that I did write my own DBs from the ground up. I will probably never need to use the entire experience, but can often use bits and pieces of it, and I appreciate a good key/value store.
Can't quite fit the whole query into the title box, but if you were using one of those databases that Wayner's article talked about, you'd be able to query and find out if you were first...
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
MySQL strives to provide RDBMS and ACID semantics, though its quality of service (QoS) may fall short. By contrast, these "slacker" databases don't even try to support RDBMS or ACID; even if they operated perfectly, they won't provide RDBMS/ACID.
I work for one of the companies in question (no, I don't speak for them). We rely heavily on a combination of these "slacker" dbs, Berkeley dbs, memcached, Oracle, flat files, and tape backups. Each fills a niche. I wish these articles would quit trying to create a false dichotomy.
I can't believe there hasn't been any mention of Berkeley DB yet. Guess what, folks: sometimes you just don't need the features of a full relational database. Sometimes all you need is fast, robust, reliable storage of indexed key/value pairs.
I can attest that Berkeley DB does exactly that, and does it really, really well. We use Berkeley DB for all of the data storage in the Citadel system, including the mailboxes themselves. Some sites have tens of gigabytes or even hundreds of gigabytes of data, and Berkeley DB just keeps chugging along, happily and reliably doing its thing. Our biggest problem? People who point at it and say "storing email in a database is unreliable" because they know it constantly explodes when Exchange does it. Well guess what, folks: Berkeley DB ain't the Exchange database (actually, maybe Exchange wouldn't be so unreliable if they switched to Berkeley DB).
Eschewing the full set of RDBMS features isn't slacking. It's choosing the right tool for the job.
Tired of FB/Google censorship? Visit UNCENSORED!
Wayner's usually a good writer, and did some good theoretical-computer-science work back in the day, but this article was too short to answer the questions he asks at the beginning, and he mostly highlighted the new shiny things from big ASPs, which is generally what Infoworld wants.
I'm particularly disappointed that while he referred to the name and history of Berkeley DB, aka Sleepycat, aka Oracle Renamed-foo, he didn't actually talk about using it. (OTOH, Infoworld did review one version of it in 2005.) I no longer have my 4.1BSD manual on the shelf, but it was useful if you wanted something faster than using grep/sed/awk/look on tab-separated text files (which were the canonical Unix database format, and what I normally used for databases.)
These days if I want a lightweight database, I usually just put build tables in Excel, and then bitch about how it doesn't have a join or even decent text-editing and filtering capabilities, and occasionally have to save it as a CSV file and install vim on Yet Another Work-owned Windows box so I can get some bloody work done. I supposed if Excel did have a join function there'd be fewer people buying MS Access...
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
How does it work for searching though? If I just have my "freespace" file and my pointers to records, does a search for some piece of user requested data have to hit every record or is there a hash somewhere for the data contained in the record? You don't mention it in your description.
It seems that the biggest advantage to a relational DB is that the syntax for accessing it is well known, SQL. It has a human read-able interface and while sometimes whonky to work with for complex operations, it provides the simplest cross-platform way to access data. I don't need to know which data blocks hold the data, I just ask the database for them "SELECT slashdotid, name FROM users where slashdotid 20000"... and I get rows of data.
Could I just read it from a file? Yes. Would it be simpler? Maybe. But what if I have 200001 records, then I have to do some magic sorting in my program, and I have to manage memory for them, and disk space, etc. It is simpler to let the DB handle that mess and I just ask for the data I need.
It breaks up the process of programming into data storage and data manipulation/presentation. DB's for storage, my bad python for manipulation and presentation.
--Donald
www.rdex.net
You may have seen in the news recently how in the last decade or so Wall Street ignored some of the hard-won regulations and guidelines developed in the wake of the Great Depression.
We all know what happened as a result.
The same is true when dealing with data. You don't ignore the rules completely, or follow them only when you feel like it, or when you have time. As the old joke goes, Quality is *not* Job 1.1.
If the data isn't important enough to store correctly, then it's not important enough to be stored at all.
"My country, right or wrong; if right, to be kept right; and if wrong, to be set right." --Senator Carl Schurz (1872)
Databases at a very abstract level are just data structures. Choosing a relational database when you don't need that much functionality is just as wrong as choosing a flat file when you need a database.
Knowing the ins & outs of your data structures is still a vital skill of programming.
Question everything
so you start a small project, "we just need a few hundred/thousand records, a few key value links and the occasional transaction". so you start with a slacker DB. A slacker DB far too often implies a slacker hack software d00d.
Then it grows. Instead of educating themselves (Q: what's the difference between those who can't read and those who don't? A: nothing. ) and finding a better DB solution they thrash around trying to hack in DB functions into their code.
So they lose consistency etc. Soon they have a polluted DB that breaks all the time. Often they are proud of the heroics of the wasted effort they put into it. A good programmer know how to be correct form of lazy: do not reinvent the wheel.
putting the 'B' in LGBTQ+
Okay how do you find the data without a record number? I can see the value of the system but it also seems very inflexable.
I do agree that way to many programmer use MySQL for a file system, flat files, configs, and goodness knows what else.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
Every database covered in the article is a toy.
From TFA: "The problem is that JOINs are really, really slow when the data is spread out over several machines."
This is the result of a poor design, not a database flaw. If you are running a web application against multiple databases, either cluster them or store all the data for a user in one database. (i.e. hash the login_id and select the database based on the result). If someone is doing JOINs across multiple machines and doesn't have a very good reason for doing so, then nothing short of a lobotomy is going to help them.
From TFA: "Each query can only run 5 seconds. The answer can only hold 250 items. Each item can have only 250 pairs."
Yeah, I'd say that meets the definition of a toy database alright.
From TFA: "Many of the complaints about the other toy databases revolve around how a missing feature makes it impossible to find the right data. If you want to add a bit more functionality to the database here, you can whip up many of the features locally in Python. If you want a JOIN, you can synthesize one in Python and probably customize the memory cache at the same time. This is especially useful for Web applications that let users store their data in the service. If you need to add security to restrict each user to the right data, you can code that in Python too."
The writer must be joking. Who would do this when there are better options that don't involve implementing your own database?
From TFA: "there's no big reason to use Ruby, Python, Java, or PHP on the server when it can all be packaged in JavaScript"
Many people who write web applications actually want to do usefull things with the data they store like generate reports, keep logs, track inventory, or run queries. This doesn't work very well when the "database" is a text file sitting on the user's harddrive.
Yeah, when I first read this article I thought that was the dumbest thing I'd ever heard, but reading it made alot of sense. It's basically just using a simple schema like the "slacker" DBs for canonical storage, and then using additional tables as 'indexes.'
How FriendFeed uses MySQL to store schema-less data
Given their needs in terms of adding features, altering the schema, and building indexes, being able to make the indexes "eventually consistent" was huge. You have to remember that to keep things nice and denormalized, you need lots of tables, joins, and that MySQL (or any other FOSS RDMS) CANNOT build indexes across tables.
It turns out that there actually _are_ neurological reasons that music from your teenage years is extra-evocative, just as language-learning works better with young kids. Go read "This is Your Brain on Music" for more details.
A certain amount of music sensitivity appears to be hardwired into our brains, and the extra hormones after puberty increase music-remembering ability and the emotional aspects of it that younger kids don't have as much of. There's also a lot of intellectual development going on in those years, and it's easier to pick up more complex ideas from the music than you could when you were younger.
As you get older, that still happens a bit, and you'll still run into music that's new and cool which you'll enjoy years later, but now it's competing with lots of other cool music that's in your head which your teenage-years music wasn't.
What's much more annoying is when you find yourself tuning by a different radio station and wondering "What is all this noise those kids are listening to? They should turn that crap down and listen to good stuff" just like your parents said when you were a kid. Some of that's because 90% of everything is crap, and it's not the crap that you find evocative because it was around when you were a kid, and some of it's because 90% of everything on the radio is highly-packaged commercial crap, making it 99% crap instead of only 90%. And some of that's because kids always want to listen to new stuff and piss off their parents, and musicians always like to do new stuff, and if you want to bust into the Top 40 you've either got to do identical commercial crap better than anybody who's already there or else do something new. Rap was creative and interesting, but the whole gangstas-dissing-women motifs that dominated it were offensive. Hip-hop took that music and started doing lots of interesting things with it, though I haven't followed it. I'm finding my self playing a lot of old-timey (average hair color in our jam session == gray, leaning toward white :-), and starting to listen to jazz more (lots of deep classical stuff in there, which I haven't had the patience to listen to for a while.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I get tired of hearing the same old discussion about whether or not the relational database is going to die. They're not. But the new breed of *specialized* databases work well for their *specialized* purposes. Big surprise. But all of them inevitably make a trade-off. Anyone who works seriously with database design knows that it's all about trade-offs.
One of the main motivations for the new breed of databases is that the standard SQL database relies on things such as foreign keys and other constraints for data consistency, but that requires the data to be directly managed by that running DBMS process. When you require data to be distributed over a network (i.e. over many separate processes), then the only way a *foreign key* can work is if the DBMS process has some sort of link over the network to the separate DBMS process and then use that somewhat as if it were local. (Other strategies involve using external application code for consistency rather than foreign keys, etc.) Of course, the DBMS process can't use it's usual local low-level optimizations behind-the-scenes in order to handle that query efficiently over the network, so it doesn't scale. Specialized DBMS's for distributed data focus on optimizing being distributed, while the typical SQL DBMS optimizes storage and retrieval of data as if it were local. The bottom line is that the traditional SQL database scales well vertically, but not horizontally concerning hardware. Or rather, when you scale horizontally, you forgo a lot of its advantages. The new breed of databases trade-off consistency and other assurances for the sake of "good enough" consistency and really fast retrieval of domain-specific data.
But not everyone is trying to be Google or Amazon. Financial institutions such as banks can't tolerate "good enough" consistency. The biggest problem with relational databases I see nowadays is that people are ignorant about why "relational" is such a good idea, and how SQL only gets you part of the way to "relational" and that SQL's shortcomings are a different issue. The second biggest problem is that most people are used to only one or two data usage patterns, and if it "works for them", then they assume it should *always* be done that way. For example, the hordes of people who barely know Excel (i.e. not a relational database) or Access, and then like to give "expert" advice. Or a web programmer that believes that ORM's are the One True Way because they abstract away choices of DBMS in order to keep favorite language X, despite the needs of other people are the opposite: perhaps we want to abstract away the choice of programming language so that we can keep the same database, and so maybe it's a good idea if the database itself can ensure data consistency rather than relying on the ORM, etc.
As a DB admin myself, I find these "Us vs Them" arguments to be ultimately pointless. A company will choose a database based on the application's needs. If "immediate consistency" is needed they will choose a standard relational database. If "eventual consistency" is acceptable, the company may opt for one of the other "not-so-relational" databases. The fact that there are other options is actually a good thing. The "old guard" needs to find the positives and embrace change, or run the risk of being left behind in an evolving world of technology.