Researchers Create Database-Hadoop Hybrid

ericatcw writes "'NoSQL' alternatives such as Hadoop and MapReduce may be uber-cheap and scalable, but they remain slower and clumsier to use than relational databases, say some. Now, researchers at Yale University have created a database-Hadoop hybrid that they say offers the best of both worlds: fast performance and the ability to scale out near-indefinitely. HadoopDB was built using PostGreSQL, though MySQL has also successfully been swapped in, according to Yale computer science professor Daniel Abadi, whose students built this prototype."

Comments

Please stop

[Anonymous Coward]

Uber-cheap is not a word, and it doesn't even make sense because you're saying it's "above cheap". Stop making up stupid shit.

Re:Please stop

[CorporateSuit]

German prepositions do not have direct english equivalents. I suppose being an "Ubermensch" would be talking about the HATS that people wear, since that's what's Over the Mensch (person). Stop getting your panties in a twist over things you're wrong about.

Re:Please stop

[spatley]

how about Ultra-Cheap
hmmmm...

Re:Please stop

[CorporateSuit]

Considering that "Ubermensch" was translatable to "Superman" then "Ubercheap" would be "Supercheap"

It's called a prefix. We use them in the English language. This one has recently been adopted into our language. Pick up the pace or shut up about things you don't know.

Re:Please stop

[Anonymous Coward]

Uber-cheap is not a word, and it doesn't even make sense because you're saying it's "above cheap".

You remind me of my English teachers. Every year they kept saying that "ain't" isn't a word because it's not in the dictionary. Then one day I looked in the dictionary and it was there. The lesson I learned was that humans create words and "rules" of language aren't really rules at all. They are merely traditions. I suppose you think the French are just speaking bad Latin? No, languages change. From Old English to Middle English to Modern English it changed. I bet all along the way there was some know-it-all jackass pedant saying "thither isn't a word!"

Re:Please stop

[jd]

Since a high price is above a low price, "above cheap" means "expensive".

Re:Please stop

[gandhi_2]

I am a metaphor for war.

You are for metaphor war.

Re:Please stop

[Freetardo Jones]

Considering that "Ubermensch" was translatable to "Superman" then "Ubercheap" would be "Supercheap"

No, it wouldn't. It would be word soup that any German would find to be awkward. To say something is "super cheap" they would say something like "superpreiswertes" which would literally translate as "super inexpensive". They wouldn't use über in such a situation.

Re:Please stop

[camperdave]

Yes, I guess it should be Infra-Cheap.

Re:Please stop

[nog_lorp]

Word soup? Have you *HEARD* German lately? Most of their speech is made up of huge conglomerations of words and prefixes and suffixes.

For example, the word for CPR in german? Herzkreislaufwiederbelebung (heart-circle-run-again-enlivenment).

Re:Please stop

[Freetardo Jones]

Of sure, German is an extremely over verbose language at times, but the fact of the matter is that CorporateSuit, despite all his blusterings, is about as clueless in German as he tries to claim others are.

Re:Please stop

[Anonymous Psychopath]

We've been co-opting other language's words into English for a long, long time now. To a growing number US citizens prefixing anything with "uber" is the same as saying "ultra" or "super". You know the saying "it's all over except for the shouting"? Yeah, that's pretty much where this is.

Feel free to mod this entire thread, including the parent, uber off-topic.

Re:Please stop

[msimm]

Pushing a translation into colloquial English does not make it a model for translation. When I'd first come across ubermensch reading Nietzsche it was described to mean 'overman'.

Re:Please stop

[misexistentialist]

"Overcheap" would work, although it would have to be read somewhat ironically since "over" usually has a negative connotation. Anyway the Germans call cell phones "Handys", and really shouldn't complain about what we borrow from their language.

Re:Please stop

[CorporateSuit]

the fact of the matter is that CorporateSuit, despite all his blusterings, is about as clueless in German as he tries to claim others are.

I suppose that all comes from living in Germany for several years, speaking nothing but German with around 1,000 people a week, face to face. I suppose anyone who had gone through such rigors would end up being "clueless" in German as well. All sarcasm aside, perhaps you are more right than you think. Some Germans don't consider Koelsch or Hessisch (the dialects I ended up speaking) to be real German at all (Although they are more understandable than Bayerisch or Frankfurterisch - which is like Hessisch on crack). "Ich bin am lesen", while correct in some dialects, is unacceptible grammar in others.

And adding to the other ways to say "cheap" would be "Ueberaus Billig", "T.I.P." or "sehr guenstig"

Re:Please stop

[Hoi Polloi]

This thread is uber-dumb.

Cartman would say it was "hella-stupid".

Re:Please stop

[GameMaster]

The way I see it, the real question should be "does it increase the ambiguity of the language or decrease it's expressive power?". As long as someone understands what is being said (with slang like "ain't" that has been in use long enough so it is widely known) then I don't see a problem with it. We may become, somewhat Balkanized in the short-term, but, hopefully, this will serve to get those conservatives used to living in a pluralistic society and will wear down some of their xenophobia. I see the real problem as being the mindset of fearing anyone that talks differently from you. Sure, a little of it is natural but we have been way past that point for a long time and it's something that the US, as a culture, should train itself away from.

Re:Please stop

[MikeBabcock]

Uber doesn't mean 'above' in popular parlance, its an absolute measure of greatness. Therefore 'uber' cheap would refer to more cheapness.

Re:Please stop

[msimm]

If I could tag comments I'd tag 'lol'. We should be able to tag comments, and those comments could be called /twits, in memory of some other service.

Re:Please stop

[moonbender]

Compounds parse easier with correct parantheses: (herz)(kreislauf)(wiederbelebung) or (heart)(circle-run)(re-activation), where each of the bracketed words is itself a common compound. FWIW, Cardiopulmonary resuscitation has more characters than the German term. German and English aren't very different, in fact, in terms of compounds; English also has a huge number of compound words, even though they are often not spelled as a single word: circuit breaker, for instance. As English compounds get increasingly entrenched, the compounds tend to get hyphenated, and eventually they are written as a single word.

Re:Please stop

[moonbender]

You're right that ueber would not conventionally be used as a prefix in this situation, but we weren't talking about the German prefix ueber, but about the English prefix uber, which was adopted from German. The fact that you wouldn't say ueberbillig in German doesn't mean that it's improper to use ubercheap. It makes you sound a bit like an ass, but I would argue that it's in line with other conventional uses of the "uber" in English. To put this in a different perspective, English probably uses the Latin prefix super in situations in which it would not have been used in Latin; or at least if it did, nobody would care.

Re:Please stop

[beerbear]

Oh yeah cardiopulmonary resuscitation is a lot better... that's why everyone uses the acronym.

PostGreSQL

[tcopeland]

It's PostgreSQL... but I sympathize with the mixed case confusion and refer you to this Postgres vs PostgreSQL permathread.

Re:PostGreSQL

[WuphonsReach]

And... a lot of times in written communication, I simply say "pgsql".

I do try to capitalize PostgreSQL in more formal communication / documentation, however.

If it works as described it will be VERY important

[Etylowy]

If both the performance and scalability is as good as described I can safely say that this is the most important thing of the decade and not only for DBMS.
Handling large portions of data would get cheaper by an order of magnitude at least and scaling out would be way cheaper than now as well. I do hope it's true.

What about Essbase?

[r_jensen11]

I thought Essbase was supposed to be one of the best databases for managing too much information. Is this supposed to be an alternative, or act as something in-between using Essbase and a mysql server?

Re:What about Essbase?

[hemp]

Transaction speed has never been high point of Essbase, nor storing anything but numerical data.

Changes to the data are not reflected immediately except in the lowest level members until it is re-calculated. It is not unusual to find calc scripts that run for 8+ hours.

Re:If it works as described it will be VERY import

[Anonymous Coward]

It won't deliver. In the mean time for those of us living and working in the real world, hard-drives will be bigger and faster, file systems will get better, and SSDs will start to shit all over spinning platters.

Typical

[MarkPNeyer]

The grad students do all the work, and the professor takes all the credit. Anyone can come up with ideas, the real work is in actually getting things done. This is the reason I stopped grad school with my MS even though I LOVE computer science, more than anyone i've ever met.

Re:Typical

[Archangel Michael]

I hate to disagree with you ... but ....

Anyone can come up with ideas is true, HOWEVER not all ideas are GOOD ones. The problem with coming up with GOOD ideas is often people don't have a basic understanding of the problem or the implications of various ways of implementing an idea.

Getting people to do the work is often not quite as easy as it seems. First you have to have qualified people. They have to be motivated to actually complete the work given.

As for degree programs at schools and such, a MS is nothing more than a piece of paper ascribing practical and theoretical knowledge of a certain level. MS (MA) degree shows an extra two years of mastery that a BS does not. Employers like MS degrees over BS degrees because they also take extra initiative (goal setting) that a regular BS degree doesn't.

I'm not sure what good a MS degree is after say 5 or 10 years in a career, other than being a feather in a cap.

I take a look at what I know, verses what I knew graduating college, and I know substantially more, and more practical knowledge, things that no MS piece of paper can show. However, I cannot quantify or qualify my knowledge other than my work experience.

I recently had a week of training that was completely useless for me. I already either knew, or didn't need to know to do my job, and the material I didn't know I could have learned on my own, should the need arise.

That being said, I am going to go get that MS paper now (25 years after college), so that if and when I need to look for job, I have that feather in my cap. It seems fairly important to those people doing hiring.

 

Re:Typical

[grepya]

I take a look at what I know, verses what I knew graduating college, and I know substantially more, and more practical knowledge, things that no MS piece of paper can show.

Does your extensive post-collegiage learning include constructing a multi-clause sentence in the English language (..and I wouldn't even mention the spelling error) ? Ordinarily I wouldn't be an asshole about this except you screwed up the exact sentence where you're bragging about your amazing skills, acquired over a long professional career. And whatever that career might be, writing a readable sentence in your main language is a basic skill (and I know you're an American from your other posts).

Re:Typical

[Archangel Michael]

Can't spell worth shit, doesn't negate my intelligence. I know idiots who spell perfectly. I know very intelligent people who spell well, and idiots who can't spell like me. Spelling is NOT a sign of intelligence nor education level.

And I didn't know I was going to be "graded" on a spur of the moment post to a web log. Had I known you were a lurking grammar nazi, I would have proofread my post more carefully. Perhaps even hiring someone to draft(write) it for me to post as my own.

Knowing my weaknesses (spelling) is a strength. That, and I don't become a grammar nazi, who misses the point while nitpicking.

I seem to have made it through college without much of a hassle with either issue you seem to point out. I wonder what that says about the college system or even the "American" Education system?

Re:Typical

[grepya]

It's not just your writing that sucks. Going by your response, your reading comprehension is not so hot either. I like your rant (which, your probably did get proofread ironically) but it was about *THE WRONG THING*. I'd leave it at that. Maybe you should spend some time on reflection rather than defending the indefensible.

The SQL language is also an issue

[chrysalis]

Scalability is one thing, but what we appreciate in SQL-free databases is also that they don't require SQL.

When what we want is just to retrieve a record, calling get(id) is way easier and more secure than building an SQL statement, and way cheaper than using an ORM.

The Tokyo Cabinet API is absolutely excellent in this regard. And there's no need to learn yet another domain-specific language like SQL, just use the language you use for the rest of the app.

Now, SQL-zealots would troll "but how would you do with ?".
And yes, for complex requests as in data mining, SQL and XPath make sense. For people who aren't developpers, SQL makes sense as well. For interoperability with 3rd-party apps, SQL is also useful, just as FAT is still useful today in order to share filesystems between operating systems.

But for the rest of us, SQL is cumbersome. Databases like MongoDB make you achieve similar results in a more natural way instead of forcing you to learn SQL and to rethink everything in a tabular way.

Re:The SQL language is also an issue

[maxume]

Is there something terribly dangerous about parameterized queries?

Re:The SQL language is also an issue

[jd]

I would argue that all solutions that currently exist for databases are ideal for some specific set of problems AND some specific set of users for each problem within that initial set.

There is no "perfect" solution that will work for all types of data, be it a flatfile structure, a hierarchical structure, a relational structure, object-oriented or some combination of those. (The star-structure of OLAP databases is a hybrid, for example.)

What would be good is if there was a suitable metalanguage in which you could define an abstract idea of the search and have a source-to-source compiler turn that into a suitable specific solution for the type of database in use, or even the specific database engine itself.

(There are some nice tools for producing specific database SQL queries out of an abstract definition, but I know of nothing that can cross database design methodologies.)

Re:The SQL language is also an issue

[Underfoot]

I would just like to state for the record, that IMHO SQL is a beautiful thing. Its ease of interoperability (both between languages and backends) has saved my butt on numerous occasions (not to mention the ease with which you can go from very simple to very complex depending on the need of your application) ...
 
...and you can get rid of it and replace it with OOP when you pry it from my cold dead hands.

Re:The SQL language is also an issue

[BitZtream]

When what we want is just to retrieve a record, calling get(id) is way easier and more secure than building an SQL statement, and way cheaper than using an ORM.

Yes, I'm an SQL troll, but ... if using SQL to get a row by a unique ID is too hard for you and too insecure, there is no amount of code which is going to fix your problem, which is that you are a shitty developer who is far too lazy to make a function or macro to wrap around the simple sql request.

There are PLENTY of reasons to not like SQL, but your example is so simplistic that it just makes me think the problem you have is you, not what the database backend is or how its used.

As far as being forced to think of your data in a tabulare way due to using SQL, you think this is a bad thing. It isn't. Its retarded that you're going to use an SQL backend anyway, and then try to think of the data in some other form. Adding some silly layer on top to make it easier for you to understand just indicates you're too lazy to understand your own data and how to use it efficiently.

I hate when people make stupid posts like yours. I end up sounding like an SQL fanboy and I don't even fucking like it.

Re:The SQL language is also an issue

[Tablizer]

When what we want is just to retrieve a record, calling get(id) is way easier and more secure than building an SQL statement

What's so hard about building a wrapper function for the times you want simple:

function getRecord(int id) {
  return(query("select * from foo where id=%", id));
}

And you'd still have SQL for the times you want fancier stuff.

Re:The SQL language is also an issue

[ChaosDiscord]

"a record, calling get(id)"

So you're relating "id" to "a record." I assume that the record in question is a blob of potentially binary data that your program parses however it wants. So you want to relate unique identifiers to blobs. You can do that quite easily with SQL. Looking up a given unique identifier quickly is something your average relational database is very good at. And writing the wrapper function to implement your hypothetical get() function is trivial in most languages. I'm completely at a loss for what your SQL-free database is offering me in this case. It's saving you from the horror of writing 10 lines of code, once, to implement get(in)? 60 minutes with a good SQL tutorial will teach you everything you need to know. Sure, there is a lot more you can learn, but for the simple case you're describing you can understand SQL at only the most simple level.

Or are you handwaving the "a record" is actually automatically squeezed into one or more variables or objects in your code? You say get("ChaosDiscord") and out pops the UserObject populated with the relevant information. Of course, at this point you need to start teaching you database, or at least your database wrapper, how your objects are structured, and how to serialize them. This is admittedly a bit of a nuisance, but an SQL-free database doesn't magically make the problem go away. Sure, an SQL-free database can provide a layer to simplify or automate it, but so can a layer on an SQL database (Ruby on Rails is perhaps the best known). Sure, you'll need to tell it that username is a string, userid is an integer, and so on, but you only have to say it once in SQL instead of in your program. The total work hasn't gone up.

Ultimately, you appear to be complaining that SQL is too powerful (and thus complex) for your needs. But you can easily learn and use a subset of SQL that corresponds to what you claim you're looking for in an SQL-free database! You might as well complain that Java is too powerful it has thousands of classes you don't need. The time to learn the relatively minor amount of SQL you need is insignificant compared to the time to develop any non-trivial application. If even that hour is too much, you can outsource the work to a geeky college student for some pizza and soda.

There are some compelling reasons to look at SQL-free databases, but "SQL is too powerful" isn't one.

Re:The SQL language is also an issue

[MikeBabcock]

I understand putting another API above SQL to make it simpler to use, but avoiding using SQL because its powerful makes no sense.

Re:The SQL language is also an issue

[chez69]

Not to be a troll, but this sure sounds a lot like IMS. Write a program to analyze the data.

some mainframers would be laughing their asses off.

Re:The SQL language is also an issue

[Alpha830RulZ]

And there's no need to learn yet another domain-specific language like SQL,

SQL, "domain specific"? Wow. I am taken aback. Over 30 years of coding, I think SQL is singlehandedly the most productive addition to the development environment I can think of since the compiler. There are a lot of reasons that using a SQL database might not make sense (small platform, single user, low cost, small required footprint, etc) but domain specificity isn't on my list. I can't think of a less domain specific development technology out there.

If you are working with piles of data, SQL is much easier, and generally much faster in both development time and execution time than what you'd cobble up in code. It hides the details of storage from your app, which is usually a Pretty Good Thing. Apps that use SQL can be readily written to scale from small to quite large environments, without significant code changes. I don't know why you'd want to give these benefits up.

Re:The SQL language is also an issue

[ArsonSmith]

Some people are able to grasp new concepts and others cant???

I'll get off your lawn now.

Re:The SQL language is also an issue

[chrysalis]

Talked about "records" and "id" because people familiar with SQL might not be familiar with other kind of databases, but you took it the wrong way.

Now, ask Google. How many critical vulnerabilities were due to SQL injections? How many similar vulnerabilities were found in SQL-free databases?

I agree with you that workarounds do exist, and that developpers are to blame instead of SQL, but in the real world, SQL is how a lot of services are compromised by kiddies.

Why do we need to invent wrappers?
Why do tools like GreenSQL exist?
Why do we need ORMs?
Why does a stock PHP have 5 different APIs just to issue basic MySQL queries?
Why are there hundreds of third-party PHP abstraction layers and ORMs?
How come every PHP project reinvents yet another SQL abstraction layer?
How come, even though both are using the same SQL database server, it's a real hell to merge code of a PHP application with another PHP application, because they invented their own abstraction layers?
Why do people want to hide SQL at the first place (even Apple in newer iPhone SDKs)?

Maybe just because there's something wrong with the SQL language itself.

Re:The SQL language is also an issue

[chrysalis]

Actually it's quite the opposite. For any complex task, writing a script for MongoDB, CouchDB or TC/TT is way easier and faster than an unbearable 100-lines SQL statement, that even you are unable to understand the day after. Plus it's able to get things that just can't be written as an SQL query.
And your "we'll have to run it over the weekend because it'll kill the server" is also why when you need to extract stuff out of a large dataset, you write a script to process data in chunks, not a single SQL statement. If SQL is so wonderful and the answer to everything, why do stored procedures exist?

Re:The SQL language is also an issue

[chrysalis]

Not exactly since Excel macros can perform loops at least.

Re:The SQL language is also an issue

[chrysalis]

Marschaling is still required, but you don't need to think about being restricted to a schema, columns, types, to define identifiers for everything, to do explicit joins, etc. Just store your objects as they are in memory.
Look at MongoDB: http://www.mongodb.org/

Re:The SQL language is also an issue

[chrysalis]

"id" is MP3 data. How do I find the title of the song? Will your 3 liners do the trick?
In a document-oriented database id wouldn't be an issue. No need for any wrapper.

Re:The SQL language is also an issue

[chrysalis]

> WTF! I think that ranks as one of the stupidest statements I have ever read on slashdot!

Tons of people aren't exactly writing PHP websites, but are still able to install vbulletin, phpbb, phpnuke, joomla, wordpress on mutualised hosting. And then they fire phpmyadmin in order to remove bogus users, to count the number of posts or visits, etc. SQL perfectly makes sense for this.

Re:The SQL language is also an issue

[growse]

SQL injection vulnerabilities don't exist because of the database, they exist because of the crappy programmer who doesn't know how to use the database being let loose writing production code. It's a bit like saying "Lets blame the internet for Cross-site scripting vulnerabilities!".

And there's nothing wrong with SQL. There's a lot wrong with people who think SQL will solve every single problem under the sun. Unfortunately, those people seem to be employed writing 3rd-party abstraction layers and ORMs.

Re:The SQL language is also an issue

[mabinogi]

The answers to those questions will say a whole lot about why PHP sucks, but very little about SQL.

in particular:

Why does a stock PHP have 5 different APIs just to issue basic MySQL queries?

Because the PHP developers have re-invented the wheel five times and still haven't figured out it's not supposed to have sharp corners. Nothing to do with SQL. Perl's DBI is a good example of a database abstraction layer done right.

Re:The SQL language is also an issue

[elnyka]

Some people are able to grasp new concepts and others cant???

I'll get off your lawn now.

And the later types are the ones that write shitty code on a regular basis? /sarcasm-gone-amock :-/

Re:The SQL language is also an issue

[elnyka]

Looks like you never dealt with denormalized and sharded data.

Denormalized data (except in certain cases) is usually a sign of bad design, not an intrinsic RDBMs attribute.

As for sharded data (assuming that it's properly normalized, otherwise see previous paragraph), and assuming that it's properly sharded among functionally-sound partitions, what's the trouble in implementing the hypothetical request? Badly partitioned data is just as denormalized data; signs of someone who didn't know what he/she was doing.

One could also imagine a hypothetical scenario where data has been entered in a non-sql, key-value store in such a way that makes it near impossible to make any sense out of it. But just as in your hypothetical counter-example, this counter-example would be an attribute of bad design and not necessarily a sign on the limitations of non-sql key-value stores.

Re:The SQL language is also an issue

[elnyka]

Actually it's quite the opposite. For any complex task, writing a script for MongoDB, CouchDB or TC/TT is way easier and faster than an unbearable 100-lines SQL statement, that even you are unable to understand the day after. Plus it's able to get things that just can't be written as an SQL query.

Can you provide concrete proof of this? "For any complex task" pretty much uses universal quantification over all problems dealing with data representation. Considering that some (not all) data (with its associated meaning and function) is best represented using relations, while others are better represented using network models (think cyclical graphs), and others are best represented as simple key-value mappings, I would find it hard to believe that truly and verily any complex data manipulation task is best represented writing a MongoDB/CouchDB/TC/TT script.

Now, if you decide to reply by saying "well, not all, but most", please provide proof, or at least some logical demonstration that this is indeed true. For, if it is, holy crap, you need to write a dissertation on this.

And your "we'll have to run it over the weekend because it'll kill the server" is also why when you need to extract stuff out of a large dataset, you write a script to process data in chunks, not a single SQL statement. If SQL is so wonderful and the answer to everything, why do stored procedures exist?

Re:The SQL language is also an issue

[elnyka]

Shoot, I f* up the quotes, so let me try again.

Actually it's quite the opposite. For any complex task, writing a script for MongoDB, CouchDB or TC/TT is way easier and faster than an unbearable 100-lines SQL statement, that even you are unable to understand the day after. Plus it's able to get things that just can't be written as an SQL query.

Can you provide concrete proof of this? "For any complex task" pretty much uses universal quantification over all problems dealing with data representation. Considering that some (not all) data (with its associated meaning and function) is best represented using relations, while others are better represented using network models (think cyclical graphs), and others are best represented as simple key-value mappings, I would find it hard to believe that truly and verily any complex data manipulation task is best represented writing a MongoDB/CouchDB/TC/TT script.

Now, if you decide to reply by saying "well, not all, but most", please provide proof, or at least some logical demonstration that this is indeed true. For, if it is, holy crap, you need to write a dissertation on this.

And your "we'll have to run it over the weekend because it'll kill the server" is also why when you need to extract stuff out of a large dataset, you write a script to process data in chunks, not a single SQL statement.

Who told you that for that type of task you would use a single SQL statement?

If SQL is so wonderful and the answer to everything, why do stored procedures exist?

Encapsulation. Reuse. You know, all the nice things associated to modules and procedures. Stored procedures are the same. In any non-trivial system, data might change its structure over time. Say, I have a SQL statement A that goes over one or more inter-related but heterogeneous database structures X, Y, Z, producing a set of values with structure or interface B. And imagine that SQL statement A is used at different locations in my code.

Due to business changes, to obtain set of structure B, now I have to operate with structures X, Y, Z". So the newly adapted SQL is A'. Say, I'm intelligent. So I put my SQL in a stored procedure P which is called throughout my code. Changes from A to A' are encapsulated inside the store procedure. I can unit test that store procedure and have a level of assurance (but not a certainty) that it will work throughout my code.

Or maybe I'm not intelligent; I'm an idiot that doesn't know SQL and much less stored procedures (what are stored procedures for anyways?). Then I have to go through the code replacing every instance of A into A', and unit test every changed piece of code. Man, I can understand someone not knowing SQL and stored procedures, but for f* sake, these are just CompSci 101 concepts man. Please get your basic principles right before trying to lecture on what's good or not in software.

Another good reason for using stored procedures if for fine-grained execution control. You don't want just any schmuck that is clueless about SQL to execute random SQL producing cartesian joints or a long-ass query that doesn't use an index. You test and fine tune not only your SQL and server-calling SQL code, but also the underlying data structures, partitions and indexes in your integration/UAT environment prior to deployment on production. This review and Q&A is assured by forcing every SQL statement into its own procedure.

Alas then, you simply enforce the same principles of modularity and encapsulation on SQL that you also find universally in any other modern programming language. That you cannot even come to that conclusion on your own (even if you don't know SQL) is completely unacceptable for anyone that does coding for a living.

You don't need to use a SQL database when you don't functionally need that. I'd be the first to say a lot of

Re:The SQL language is also an issue

[chrysalis]

It's a bit like saying "Lets blame the internet for Cross-site scripting vulnerabilities!".

No, it's about having less issues by using modern tools, rather than trying to find who's to blame.

If HTML/JS/CSS/HTTP could be redesigned today, do you think that the way a browser manage cookies, XHR requests and sandboxing in general would be the same as it is today? Do you think that the SMTP protocol that was good enough 30 years ago is not a big pile of crap nowadays, even, just like ORMs, their content is now shown in webmails? SQL is just like SMTP. Or the FAT filesystem. An old thing. There are worthy proposals and even working products that could superscede them, but because of legacy applications and people who want to stick with the same technologies till the end of the universe, these old things remain. They just get bloated with new extensions instead, in order to keep up with mandatory requirements.

they exist because of the crappy programmer who doesn't know how to use the database

"Our GUI _is_ user-friendly! Customers just don't understand how to use it because _they_ are clueless!"

"hey, and there's a workaround to make them use it properly. Just tell them to top an GUIORM (pick one out of hundreds), an extra layer that hides our wonderful GUI and shows another GUI that's closer to reality, and transparently translates events both way, while introducing some new limitations (thinking about that other ORM that has no way to issue OR-like statements) and some grain of extra sluginess!"

Re:The SQL language is also an issue

[mbourgon]

Didn't realize Joe Celko was a Slashdotter.

Re:The SQL language is also an issue

[growse]

It's a bit like saying "Lets blame the internet for Cross-site scripting vulnerabilities!".

No, it's about having less issues by using modern tools, rather than trying to find who's to blame.

If HTML/JS/CSS/HTTP could be redesigned today, do you think that the way a browser manage cookies, XHR requests and sandboxing in general would be the same as it is today? Do you think that the SMTP protocol that was good enough 30 years ago is not a big pile of crap nowadays, even, just like ORMs, their content is now shown in webmails? SQL is just like SMTP. Or the FAT filesystem. An old thing. There are worthy proposals and even working products that could superscede them, but because of legacy applications and people who want to stick with the same technologies till the end of the universe, these old things remain. They just get bloated with new extensions instead, in order to keep up with mandatory requirements.

Of course, if you were designing something today to do the job of 'relational database', you'd probably get something different from SQL. That doesn't change the fact that today, the SQL / RDBMS combo is the best tool for solving a lot of problems. That doesn't mean that people won't try and use it improperly, but those people are idiots. People don't stick with SQL because it's old, they stick with SQL because it gets the job done, and in the hands of someone who knows what they're doing, it gets the job done well as long as the job is a sensible one for which a relational database makes sense. Like storing millions of customer records.

they exist because of the crappy programmer who doesn't know how to use the database

"Our GUI _is_ user-friendly! Customers just don't understand how to use it because _they_ are clueless!"

"hey, and there's a workaround to make them use it properly. Just tell them to top an GUIORM (pick one out of hundreds), an extra layer that hides our wonderful GUI and shows another GUI that's closer to reality, and transparently translates events both way, while introducing some new limitations (thinking about that other ORM that has no way to issue OR-like statements) and some grain of extra sluginess!"

Typically, you don't have any control over how smart your users are, so you make GUIs nice and simple to use. However, the analogy doesn't hold for developer tools and technologies. You do have control over how smart your developers are, and if they're stupid to use SQL properly, they're idiots. It's not the fault of the technology that muppets use it improperly.

Re:The SQL language is also an issue

[pamar]

I can not agree more. I started working with RDBMS on dBaseII in the early 80's. ...

Hate to nitpick, but if you are talking of the IBM product, I think you better say "DB2".

dBaseII was an Ashton-Tate product, originally conceived for CP/M. The dBase line introduced RELATIONAL functionalities only with DBase IV (and they allegedly never really worked).

Re:The SQL language is also an issue

[Tablizer]

My example is based on the original example given. But this illustrates part of my point: when you need features you didn't originally anticipate (or ask for), you have the power of SQL to help out.

Re:The SQL language is also an issue

[TheLink]

> you don't need to think about being restricted to a schema, columns, types, to define identifiers for everything, to do explicit joins, etc. Just store your objects as they are in memory.

That's not a disadvantage in many cases, especially for the long term. There's a benefit of forcing people to use SQL to talk to the DB. It becomes a layer of abstraction, somewhat like a standard protocol or interface.

When you use SQL, your database can be used by 100 different people and programs, and when you add columns/fields, if the coders have been doing things properly (not assuming column order and number of columns), everything still works. When lots of things can use the same DB, that database often becomes more valuable and useful.

In contrast what does "Just store your objects as they are in memory" mean, when you have 100 different programs? What happens if a program needs a new data field attached to the returned object? Can the other 99 programs still work without changes? Or will some require a recompile since the DB is now returning a bigger/different object?

If you need recompile or modifications for stuff like that, that means only a few programs can share the same database (otherwise it becomes a maintenance nightmare), and that could limit the extent of how useful that database could be. You might then have to add layers of abstraction so that it becomes more accessible (and stays accessible) to many other programs. And one day one of those layers might end up looking like SQL ;).

To quote myself: One man's impedance mismatch is another man's layer of abstraction.

Re:The SQL language is also an issue

[ahabswhale]

It's because SQL isn't just SQL. It's all the cruft that goes with it. Accessing a DB from an OO language is simply a major fucking pain in the ass and much harder than it should be even when using the ORM du jour. A lot of this complexity comes from the fact that OO and RDBS just don't play well together no matter how you slice it. Instead of focusing on the business domain you end up spending far too much time dicking with the data layer. A lot of this would go away by using an OO database but then you lose the scalability of these other technologies.

Re:The SQL language is also an issue

[Abcd1234]

Marschaling is still required, but you don't need to think about being restricted to a schema, columns, types, to define identifiers for everything, to do explicit joins, etc. Just store your objects as they are in memory.

Well, that's all very interesting, but it doesn't sound any different than your average OODBMS, something which has been around for a *long* time (I worked with Versant nearly a decade ago doing exactly what you describe). Heck, the Smalltalk world has had intelligent object databases for years (Gemstone comes to mind, among others). So, again, what's the big deal?

As an aside, I'm also not convinced that being forced to a define a proper schema for one's data, that's separate from the overlaying object model, is necessarily as bad a thing as you seem to think. Nor is it that difficult, given how intelligent ORMs are these days. But, when it comes down to it, that's as much a philosophical argument as anything else.

Re:The SQL language is also an issue

[ChaosDiscord]

I believe the great-grandparent poster was talking about simple key-value stores, similar to the Tokyo Cabinet system he mentioned. When people talk about Anti-SQL or SQL-Free, that seems to be what they're always talking about, although usually on the larger end with things like BigTable and HBase. My criticism was directed in that direction. Compared a key-value store to a subset of SQL, or even a key-value store implemented in SQL, the complexity difference is negligible for any but the most simplistic of projects. As such, the great-grandparent poster's objection to SQL and relational databases compared to key-value stores was wrong. But I must yield that object storage is an area where object oriented DB have a clear advantage over relational DBs.

Re:The SQL language is also an issue

[cratermoon]

> That's not a disadvantage in many cases, especially for the long term. You Aren't Going to Need It.

Re:The SQL language is also an issue

[LionMage]

Judging from your response to ahabswhale, it seems that you've pretty much made up your mind that everyone else is wrong and you're right, but I'll take a stab anyway at why I think you're missing the point:

Looking up a given unique identifier quickly is something your average relational database is very good at. And writing the wrapper function to implement your hypothetical get() function is trivial in most languages. I'm completely at a loss for what your SQL-free database is offering me in this case. It's saving you from the horror of writing 10 lines of code, once, to implement get(in)? 60 minutes with a good SQL tutorial will teach you everything you need to know. Sure, there is a lot more you can learn, but for the simple case you're describing you can understand SQL at only the most simple level.

Hoo boy... so much here, where to begin? First off, the condescension is uncalled-for. It's not that people who hate SQL are somehow incapable of writing SQL queries, or learning how to. I think the objections are twofold:

• Everything in SQL assumes that data is formatted in a tabular fashion. For data that's truly tabular, it makes sense. But trying to represent certain other data in a SQL database is just painful. For instance, it's overkill for an associative array such as a large hashtable. I've had DBAs tell me tree structures are "trivial" to represent in SQL, but my general experience has been that the standard ways to implement this (and the hacks that some commercial DBMSes employ to optimize this) are tortured and tortuous.
• The extra layers of indirection and ceremony you have to go through to marshal and unmarshal data to/from a relational database. If you have many disparate fields and you want to do complex selection based on multiple fields or multiple conditions, a relational database makes sense, but again if what you have is essentially a giant hashtable, the extra layers of indirection for what is essentially a bunch of key-value pairs is way overkill.

To use the example you latched upon in the GP post, you talk about how trivial it is to write a wrapper function to give us a get() method equivalent. Sure, it might only take 10 lines of user-generated code, but let's consider what is going on behind the scenes:

• That wrapper function will need to take that SQL and turn it into some form that can be processed more efficiently by the DB. If you take advantage of prepared statements, this might be a one-time operation. However, consider that some databases such as MySQL don't have a good or even a working implementation of prepared statements -- in the case of MySQL, I have it on good authority from one of the devs that prepared statements never worked right, so the code for that was yanked when they did the Drizzle fork; in fact, the JDBC drivers for MySQL kind of "roll their own" in their implementation of PreparedStatement, instead of letting the database do the work.
• The statement to execute then gets shipped through the network connection to the database for execution. In all but the most trivial applications, the database won't reside on the same machine, so there's network overhead to be concerned with -- that'll be true whether you're talking about a SQL database or a no-SQL alternative. It does seem to me that in the case of a giant hashtable in the sky, the request is going to be a lot smaller over the wire than even invoking a prepared statement with a single parameter.
• Once the request hits the DB engine, there's going to be some non-zero execution time. On the SQL side, even with a prepared statement, the parameter has to be plugged in and the compiled expression executed. On the no-SQL, giant hashtable side, we have little in the way of processing to do other than doing the actual lookup, which is hard-coded logic in the engine.
• Retrieval should be a trivial matter, right? Well, in a traditional relational database, it's true that the column used as the primar

Re:The SQL language is also an issue

[badkarmadayaccount]

PostgreSQL allow turning any programming language into a query language, AFAIK. BTW, this may be off-topic, but I'm pretty sure I'll get the most DBMS geek eyes on this article, so here it goes - would it be possible/feasible to integrate the compiler system and cache with the VCS within the database? My idea is about getting the flexibility of Portage/pkgsrc systems without the hassle of compiling the whole thing, start to finish. I'm pretty sure most compile time options can be recalculated quickly, and reflected in the database as binary diffs, or maybe just a link time option.

Re:The SQL language is also an issue

[ChaosDiscord]

The grandparent ultimately asserted that "for the rest of us, SQL is cumbersome," calling out that an "SQL-free" database is "easier", "more secure," and "cheaper."

If we're talking about essentially key-value stores, SQL can do it well.

It's harder, but we're talking about an hour or so of work, an hour's worth of value you can reuse for future projects. For all of the complaints about needing to worry about tables with lots of fields and serialization, it's moot if you just want a key store. All that worrying about getting your data into tabular form is irrelevant. (I'm not sure what your point about trees is; I'm not aware of an especially more elegant way to store trees in a simple key-value store that I couldn't implement with similar ease in a relational database.)

Writing secure SQL is not hard; every modern language supports placeholders that will handle your escaping.

Now a simple key-value store is "cheaper," there is less going on under the hood, less built-in functionality ("power") available and thus less complexity. But, are you really writing the next Google? Sure, you want to believe your application will have millions of users, but... really? "The rest of us" are writing much, much smaller scale applications. Little things like the phone company's billing system, Wikipedia, and EVE Online. And we're not feeling terribly encumbered by our databases.

I find your comparison to LISP/Java and C relevant, but for the opposite reason. Key-value stores and C are the simple, obvious solution. They're really fast because they don't do much, and you can accurately predict what's going on under the hood. LISP, Java, and a relational database are powerful, resultingly complex, and it's hard to predict exact run-time performance. But many real world programs frequently don't need to be all that fast, the benefits of all that extra functionality speed development, if you add the functionality yourself you end up paying the cost or more anyway, and years of research have shown that you can make them quite fast. C programmers for years wailed that all those extra layers of indirection was way overkill, that Java "is often more complex than necessary and has too much overhead" and "that much of the overhead is hidden from the average developer, which for today's lazy programmers makes for some very inefficient code." At least today, C programmer standing by this argument are wrong for most real-world problems.

And, of course, part of the price of a key-store system is that many seemingly simple tasks become hard. Sure, you can map from product ID to a product's information, but how do you query on price, name, or manufacturer? As best I can tell the answers are to throw more computers at the problem, or start maintaining your own indexes and hoping you don't screw them up. Related but separate pieces of information (products, customers, orders) don't have an elegant representation; you end up maintaining the relationships in your code; the key-store system can't catch errors. You can add layers to help you with all of these things, but as you go, you'll find you're reimplementing a relational database. On the up side, you only pay for the features you need. But the down sides are many: an existing relational database has dedicated many years of work into optimizations and research, time you can't afford; your system is non-standard, increasing the time it takes a new team member to get up to speed; if your needs are relatively modest, the cost of development will exceed the cost of using off-the-shelf software.

Systems like Hadoop or Google's MapReduce have a lot of advantages, as you note. They do scale, especially across multiple machines, in a way that relational databases suck hard at. They can provide high reliability in the face of unreliable machines. (And on a large enough system, the mean time to failure is now.) Maybe you're doing something where those or other advantages are important. Great! But "the rest of us" aren't especial

Re:If it works as described it will be VERY import

[Etylowy]

It it will deliver it will change much. Not for your average blogger with a $10 hosting, wordpress and all his 100 readers but for all the folks that have sites successful enough to go beyond that a single DB server can deliver. Now you have to work really, really hard to make it all work with replication as pretty much no free CMS offers data sharding. Now you won't have to. Just get a DB cluster (as a service) that works out of the box with none/very little modification to the software you are using. The wall that they currently hit at the point they have to invest loads of money to continue growth will be gone.

MySQL?

[trisweb]

No offense to the creators (well, maybe some offense) but why the heck would you want to put MySQL in where PostgreSQL already was? That's like taking out your star quarterback and putting in, well, me!

Re:MySQL?

[scorp1us]

MySQL has its fan boys from circa 1994-2001. During this period, the MySQL license was much more permissive, and gained a certain momentum from PHP that carries it through to this day. At the same time, PostgreSQL was still using Cygwin on Windows, the INSTALL had a table of contents, and was lacking performance enhancements (particularly on Windows). Eventually Cygwin was dropped and the threading was happy on windows, and the performance enhancements were good. Along with this came a much shorter INSTALL file and all reason to use MySQL had disappeared. But once you know something, people like to keep on using it. Then MySQL got things like triggers, foreign key constraints and full ACID compliance. So in the end it ended being a wash. However, and not to start a flame war, it seems that PostgreSQL, having been feature-complete (ACID, foreign keys, etc) maintained a performance edge. But also to this day MySQL has a very fast table implementation, provided you don't need things like ACID compliance. For a variety of applications, this is "good enough" and the trade-offs of feature completeness vs performance are worth it. Disclaimer: I have used both extensively in the past. I prefer PostgreSQL, but now use neither. Now I only do SQLite (embedded tables) or Oracle (for hot replication).

Re:MySQL?

[DrEldarion]

Well, the name is certainly a lot catchier...

Re:If it works as described it will be VERY import

[gandhi_2]

I can't say I'm looking forward to bigger, faster, shit-covered platters...but hey. Who am I to stand in the way of progress?

Yaaaaay

[caywen]

(In my best Special Ed impersonation)

Yaaaaaay, now we can scale out Hadoop! Yaaaaay! Yaaaay Hadoop! Yaaaaay!

Bits of software are tools..

[msimm]

We might create the software intending it to do and be used in one way, but how it will actually be used is determined by the users. Postgre and MySQL don't carry any intrinsic values, only the values which their users discover and, well, use. Without users they have no good or bad features.

So why is it that people feel the need to rally around or defend them? After all, only the developers who have done the work are capable of understanding the snips and criticism leveled against them, and these are the people who have given their work away, to you and me.

MySQL excels at some things. Postgre also excels at some things. If users feel there is too much overlap then they can work to reproduce these features in a single tool, such as Postgre should they feel it has more utility. But to discount a tool many people find useful shows a core misunderstanding of what it is that determines the software's value.

Postgre can not be better then MySQL, it can only provide varying degrees of value. And that value is determined by the user.

Re:Bits of software are tools..

[photon317]

I've used both pretty extensively in a wide variety of environments, and I don't take such a balanced view at all. IMHO, the best answer to most database-related problems is to use PostgreSQL or SQLite. MySQL sits somewhere between them in terms of reliability, scalability, ACIDity, etc, and kinda fails at being good at anything in particular. For that matter, even if you *like* where MySQL lies on those tradeoffs, compared to either of the other two mentioned products (especially Pg), the quality of the code and the development process both flat-out suck, and are very amateurish in places by comparison. The only reason I use it anymore is for applications that require it and haven't been ported to PostgreSQL or SQLite (as appropriate) yet.

And that was before all the fiasco in recent years with Oracle buying Innobase, then Oracle buying Sun which had bought MySQL, and the several emerging MySQL forks, etc... which doesn't leave me feeling great about the future of MySQL if I were starting a new project today.

Other approaches to scalable SQL

[owenomalley]

There are also two Hadoop subprojects that either support SQL or will shortly. They both translate SQL queries into map/reduce programs. They are:

http://hadoop.apache.org/pig/
http://hadoop.apache.org/hive/

Re:Comic books won't substitute for reality

[CorporateSuit]

Uber and Super both mean "above", knucklehead. Same proto-indo-european root, in fact.

Today may just be the day that you learn that a word may have more than one definition. In fact, the word you use "root" refers not just to a word's origin, but it can also refer to a very important part of plants. Do not squander this opportunity. It will open an entire new world of linguistics. I have nothing but hope for the grand future that awaits you and your once-tunneled view of the English language.

Re:This technology already exists

[Bill, Shooter of Bul]

And how much more than "Free" does it cost?

Re:This technology already exists

[Bill, Shooter of Bul]

Not sure what you were talking about, but hadoop and postgres are open source. Unless they're stupid, they wouldn't make the resulting product closed source.

I'm not going to make the whole free software pitch here, but lets just say I believe in the superiority of the development process and the end product through my experiences developing and using software.

I have no confidence in Intersystems Cache's long term survival.

Don't you read Information Week?

[daschlag]

"Cheap 2.0".

Unobligatory Monty Python Reference

[cowboy76Spain]

And yes, for complex requests as in data mining, SQL and XPath make sense. For people who aren't developpers, SQL makes sense as well. For interoperability with 3rd-party apps, SQL is also useful, just as FAT is still useful today in order to share filesystems between operating systems.

But for the rest of us....

Sorry, but could not help thinking but to this line from "Life of Brian":

But apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, the fresh-water system, and public health, what have the Romans ever done for us?

More seriously, if the main example of the trouble with SQL is that you want to be able to find a record by id with less keystrokes, I do not see how this can be so much of a problem.