If I were a store owner, I'd prefer a system that requires opting in to risk, rather than opting out regularly.
As a software engineer, there is indeed always a need for me to think "What if the user makes a mistake?", and it's my responsibility to understand the risks involved with all possible mistakes that my software allows. If I'm programming an alarm clock, I need to know that a mistake could cost someone a job. If I'm programming a door lock system for a store, I need to know that a mistake could cost the store thousands of dollars' worth of inventory. If I'm programming a radiation therapy machine, I need to know that a mistake could kill someone.
To clarify, I meant that between zero and five minutes after the scheduled opening, the human-operated button unlocks the doors.
More specifically, any human employee (perhaps verified by an ID card) can tell the computer "It's okay, there's a person here" at a time specified by a sufficiently-privileged manager, and the computer can then take care of unlocking all the doors, turning on signage, ambient music, and anything else necessary. It takes an employee only a few seconds to open the whole store, but still has the security of requiring a human presence. The point is that an employee isn't spending a half an hour running through an error-prone process to start everything up, and that the doors are still managed by the computer's scheduling, which can be configured by a manager at any time.
Also note that for safety, commercial buildings usually have a clearly-marked exit that's always unlocked, so being trapped in a store isn't a realistic risk.
For one week every winter, I visit family in Florida. It'd be really nice if my electronics could recognize that I'm going off to Florida again, and prepare all the WPA keys, clocks, and weather applets to reflect my new location for the week. If the cost for such service is that I see unneeded ads for cheap hotels in Orlando and Miami, then so be it.
So let me get this straight... Somebody designed and built a computer-controlled lock system (that apparently also turns on the self-checkouts), and didn't think something like this would happen?
Would it be that hard to have an "unlock" button to pair with the computer's instructions? When the store's supposed to be locked, the button would do nothing. Between zero and five minutes after the scheduled opening, it unlocks the doors. Five minutes after opening time, a nice reminder sounds. After ten minutes, the computer could assume human error, and stop trying to unlock the doors.
Developing and installing the system would likely cost a trivial amount compared to the risk of leaving a store unlocked and unattended all day.
If anyone were simply restating research for a patent, that would indeed be fraud, but that's not what's happening at Google (or anywhere else reputable, for that matter). Instead, old techniques are being reexamined and improved, and those improvements are being patented. To quote Newton, "If I have seen further it is only by standing on the shoulders of giants." Nostalgia aside, research done in the 60s and 70s did not reach the absolute end of progress in its areas.
The patent in this article, for example, claims both a linked list and a chaining hash table, where the lists used remove expired items whenever they're traversed. The novel and non-obvious improvement is the removal of expiring items upon access. This means that plain old hash tables and linked lists are not prior art in any way that invalidates the patent. If you could find an example of such a self-cleaning list prior to 1999, I'm sure Google would love to hear about it.
As another example, consider Google's patent on MapReduce. The concept of "translate sets, then combine sets" is certainly quite old, but that's not what's being patented. What's actually covered is the specific mechanism used to manage the processing tasks on a cluster, including allocation of workload, tables used, and the addition of intermediate steps to aid processing.
If you'd like to prove me wrong, please feel free to point out any patent from Google (or other reputable company) that is wholly implemented in something from the 70's.
But did your hash-pipe-induced idea involve exactly what's claimed in the patent, and no less? Is every single claim of the patent covered somewhere in your dissertation? Not in four separate projects over the course of 20 years, but all the ideas in a single location? No? Then please don't waste the court's time.
Nobody's trying to diminish the contributions of researchers in the 60s and 70s. Computer science has moved on, and further progress is being made by refining and extending earlier techniques. Do you really mean to imply that all computing progress stopped after 1979? What is happening now is that techniques from the 70s which were impractical at the time (though better in theory) can now be utilized. More use means more bright minds thinking about them, and more improvements. Those improvements can (and often should be) be patented, as they're the result of an investment in research. I'm terribly sorry if some researcher in the 70s dreamed up a great data structure and never got anything for it, but that's no reason to deny others the opportunity to profit from today's theoretical work.
...Or maybe the patent lawyers themselves already know of Knuth, since that's first on the list of prior art.
So the "patent"... would be considered a mere refinement of standard approaches.
That's exactly what innovation generally is. Personally, I doubt mankind has had an original thought in the past few thousand years, and even that's giving a lot more credit to the Greeks than I think they deserve. Everything is based on or inspired by something else.
Now the system in question here sounds like it is screwed up in some manner.
When I saw this story, I thought of a fun time back in college, where every year the campus cops would pull out their radar-and-sign machine (that thing that says "the limit is X, but you're going Y" and stick it on one of the main roads near campus. The limit was 25 MPH. Go above 30, and the display would flash red.
One year, they decided to put the machine on a different road that they thought would have more traffic. Somebody somewhere wasn't thinking, and for two weeks they warned drivers for going 30 MPH in a 35 MPH zone.
Having done no research on the matter apart from my own experience, I'd say it's because people who "abuse" sugar are more likely to also abuse more harmful things. I, for instance, will eat a candy bar when I'm staying up late. I'll also drink excessive caffeine and get less sleep than I should. Correlation, but not causation.
Years ago, they certainly would have been "making waves", but they aren't as impressive now. CVS is surpassed by Subversion and Git, with the latter mentioned in TFA. Firefox is rapidly becoming a bloated but unremarkable product. Linux isn't really a "programmer's tool", so doesn't blong on the list in the first place. GNU Make has certainly earned its place among the annals of history, but it's only had four minor releases in the past decade. These projects are important, but hardly eye-catching for an article written today.
In related news, criminals (and everybody else, for that matter) are increasingly using email and IMs to plan or discuss crimes (or family reunions).
What's happened to the rate of landline wiretaps, or good old-fashioned undercover following? How many telegrams per day are intercepted now?
Everything is done by email now, and I find it perfectly reasonable that the police are increasingly turning to email for evidence. That's where the evidence is.
Really, I would do that one by hand, given only enough time. A problem of "find X to N precision" can be thought of as "guess a N-digit answer that's closer to X than any other N-digit guess". Extensive guesswork works just fine, though it takes about 20 tries...
Yes, I was using bisection to find solutions back in high school, when I felt particularly disinclined towards thought, and showing my work wasn't necessary. Yes, I often will solve such problems by hand. It's especially useful when you want to compute something that you know has only one optimum value, but it's painful (if even possible) to calculate directly. Of course, it's certainly not the answer Harvard's expecting, but perhaps it'd amuse some reviewer enough for credit.
Bisection also makes a fun hobby: Try to come up with problems that can only be solved quickly by bisection.
After utterly failing the Latin and Greek sections, I think I'd get a pretty bad reputation with any reviewer, even though I could do the rest just fine with a slide rule. Of course, I could follow up the geometry section with a lovely essay relating the theories of computability, genetics, and medicine, and the reviewer would be equally confused.
The parts that are important in modern innovation are still certainly appropriate for an entrance exam. The only difference I see between this and a modern exam is that the Latin and Greek sections have been replaced by English tests and some basic science questions. After all, the purpose for knowing Latin was that is was supposed to be the universal language of scholars, and during the burst of scientific progress following WWII, English took a firm grasp of that role.
There are so many who benefit from the community, and so relatively few who give back. So many people claim some excuse to not contribute anything to anybody without getting paid.
Given that I was processing text for a large part of the work (if not the majority), I'll have to pass. For what it's worth, I did look into Monet. I don't recall offhand why it wasn't selected, but just that it was one of the options.
I can certainly see history repeating itself, after 100 years...
Ugh!
The problem is, this "improvement" has be tried and failed repeatedly over the last fifteen decades. Here's the cycle. Some douche says, "horse-drawn carriages suck", and offers the world a "vastly superior solution." Ignorant masses flock to it because of the buzz; and frequently because they lack stablehands. A half or full decade later, the flocks wake up and say, "Man, were we douches. What were we thinking..." Followed by the legacy vehicles they created being retired, sold off to buy horses, or some poor chump is left holding the bag, maintaining a slowly dying machine without support.
And many benchmarks have shown, almost none of the horseless carriage solutions provide performance advantages over horses. Furthermore, those which do provide performance advantages do so by making fragility of workmanship its most significant feature; typically exceeding any performance benefits. Made worse, going the horseless route means leaving being a massive pool of proven, fast, and reliable vehicles, not to mention a large stablehand pool.
So basically, to break down the two camps, you do so with fundamental labels; ignorance vs wisdom. Or, those who know history and those doomed to repeat it. Thus far, there is every indication those jumping on the ignorant horseless carriage bandwagon are bending over to repeat history.
Looking back over history, I see many improvements that were often dismissed out of hand before they became commercially viable. It was only through the perseverance of their supporters and the eventual unbiased judgment of their adopters that such progress was eventually received. Now, I doubt NoSQL is going to dramatically change the data-storage world, but I do think its impact is worth some attention. The business push toward Big Data needs faster analysis than current techniques, and various MapReduce solutions, running on NoSQL platforms, show promising results that RDBMS solutions haven't been able to match. If nothing else, NoSQL provides pressure for RDBMS progress. Time will tell, but for now I'll devote a few hours a week to learning new technology, so I can make better-informed decisions when I need to.
Thank you so much for informing me of what I think. Until now, I was under the impression that I thought RDBMSs and NoSQL databases were actually different tools for doing different (but similar) jobs. This changes my whole perception of the world! Should I also think that a hammer and a nailgun are equivalent? Should I use sulfuric acid to wash my car? What about TNT to dig a posthole?
I have never argued that NoSQL is a replacement for RDBMSs. Not once did I suggested eliminating an RDBMS that does its job. I didn't claim that NoSQL is inherently "better" than an RDBMS. I do apologize for my mistakes, and now that I have been told what to think, I'll try harder to fit your preconceived notions of what "the NoSQL crowd" should do.
My project was done in my spare time, with no funding from anyone but myself, for the purpose of my own entertainment and education. I believe that's a reasonable candidate for a "toy". It was a web crawler that scanned a few thousand websites (top hits on Google for various terms) and analysed their connections to each other. My goal was to see what sort of relationship "reputable" sites (like CNN, for example) had to "disreputable" sites (4chan), in the hope that there'd be a way to gauge a site's reputation by how it interacted with others. A few thousand sites, a few hundred links scraped from each... and that quickly makes a billion values to connect.
Sure, it could have been a million-dollar project from a big corporation, but it wasn't. It was just my hobby, run under a grad student's minimal budget.
taxi drivers should be familiar with passenger cars and the... Space Shuttle Crawler
While I do love a good car analogy, the hyperbole ruins this one. A better example would be to suggest that taxi drivers (and especially those who may be getting a new car soon) understand what a "hybrid" car is, their advantages and disadvantages, and when they're a practical car to purchase. As emerging technologies, both hybrid cars and NoSQL databases provide additional options to be considered, and a competent driver or developer should have a basic understanding of both. To dismiss a new (or old) technology merely because of its age is a dangerous mistake.
Yes, no, and yes, in that order. I'm basing my answers on HBase, with which I have the most experience. My answers are also practically guaranteed to be wrong in somebody's eyes, because HBase is so much more flexible than an RDBMS. If I describe one way of doing something, another layout may work just as well, and somebody's going to favor that way.
How does indexing work in NoSQL? Are there EXPLAIN-type tools available?
EXPLAIN tools aren't really necessary in HBase, because almost all nontrivial queries are a scan over a small chunk of the alphanumerically-sorted rows. It will take a while, but please allow me to explain. Each row is a multi-value key-value store, with each value having a column name. If you really want to stick to the RDBMS style, you could have your key be a numeric row ID, and scan everything for every query. It would suck, because you're not using any indexes.
Indexes are more or less left up to the programmer. Creating an index is effectively just adding more rows to the table. For example, that RDBMS-style layout in the last paragraph could be a table of ID numbers, usernames, passwords, and permissions (for 50 billion people, I guess...). For whatever business reason, the main key will be the ID number. Those rows are easy. They have the expected value columns: username, password, permissions. To index by username, we add new rows, with just a column for the ID number. We could just duplicate the data, but let's not. Now, our table is going to be huge, but sparse. Half of the rows have three of four columns filled, and the other half has only one. Searching by name, it'll take two requests to get to the actual row we want, but that's okay. Doubling the amount of work lets us run faster.
The reason for that is HBase's split design. HBase's table is split into column families and regions. Column families are a means to group columns, so that even on data with overlapping key space, separate data could remain separate. Column families are stored as separate files in Hadoop. In our example, the username "index" could be a separate column family. That could speed up scanning, because the rows keyed by numeric usernames won't be interspersed with the rows keyed by user id. More importantly, the table is split into regions, each containing a number of rows. Those regions are also stored as separate files, and distributed across the entire Hadoop cluster.
The cluster is really where Hadoop gets its speed. If we were to run all of our processing from one central location, it would be horribly slow and require a ridiculous number of requests. Instead, we'll distribute everything, including the query, similar to how some RDBMS sharding schemes work. We send a request to all nodes, asking for "the row with the key that matches the value of the 'userid' column of the row with a given key". Each node will report back its results. Unlike RDBMS sharding, the partitioning is handled automatically by HBase into regions that are optimal. It's these regions that are scanned for every request.
After all of that, it should be quite clear: With HBase, the programmer is expected to know the layout of the data, and write requests based on the key. There is no EXPLAIN tool, because everything is just a key-value lookup.
Whew. Next question...
Can you do just about any query you could with SQL?
Yes, but it's different. Every lookup is handled by scanning a region (in parallel on nodes that have that region's data files), and checking each column of each row to see if:
The row key matches what was requested, or falls within a given range.
The row contains a column that was requested.
A given filter approves each column.
Note that last item. The filter is simply a program that tells Hadoop whether the row (or some part of it) should be included in the returned results. That program can include other HBase requests, using other filters. If you're really stuck on using RDBMS
NoSQL certainly isn't magical, nor is it the solution for every problem. My point is that rejecting any job candidate because they're familiar with a new technology is asinine, and it's a sure way to be left behind by more open-minded companies.
Slashdot Mirror
If I were a store owner, I'd prefer a system that requires opting in to risk, rather than opting out regularly.
As a software engineer, there is indeed always a need for me to think "What if the user makes a mistake?", and it's my responsibility to understand the risks involved with all possible mistakes that my software allows. If I'm programming an alarm clock, I need to know that a mistake could cost someone a job. If I'm programming a door lock system for a store, I need to know that a mistake could cost the store thousands of dollars' worth of inventory. If I'm programming a radiation therapy machine, I need to know that a mistake could kill someone.
To clarify, I meant that between zero and five minutes after the scheduled opening, the human-operated button unlocks the doors.
More specifically, any human employee (perhaps verified by an ID card) can tell the computer "It's okay, there's a person here" at a time specified by a sufficiently-privileged manager, and the computer can then take care of unlocking all the doors, turning on signage, ambient music, and anything else necessary. It takes an employee only a few seconds to open the whole store, but still has the security of requiring a human presence. The point is that an employee isn't spending a half an hour running through an error-prone process to start everything up, and that the doors are still managed by the computer's scheduling, which can be configured by a manager at any time.
Also note that for safety, commercial buildings usually have a clearly-marked exit that's always unlocked, so being trapped in a store isn't a realistic risk.
8.8.8.8 is an address for Google's public DNS server.
For one week every winter, I visit family in Florida. It'd be really nice if my electronics could recognize that I'm going off to Florida again, and prepare all the WPA keys, clocks, and weather applets to reflect my new location for the week. If the cost for such service is that I see unneeded ads for cheap hotels in Orlando and Miami, then so be it.
So let me get this straight... Somebody designed and built a computer-controlled lock system (that apparently also turns on the self-checkouts), and didn't think something like this would happen?
Would it be that hard to have an "unlock" button to pair with the computer's instructions? When the store's supposed to be locked, the button would do nothing. Between zero and five minutes after the scheduled opening, it unlocks the doors. Five minutes after opening time, a nice reminder sounds. After ten minutes, the computer could assume human error, and stop trying to unlock the doors.
Developing and installing the system would likely cost a trivial amount compared to the risk of leaving a store unlocked and unattended all day.
If anyone were simply restating research for a patent, that would indeed be fraud, but that's not what's happening at Google (or anywhere else reputable, for that matter). Instead, old techniques are being reexamined and improved, and those improvements are being patented. To quote Newton, "If I have seen further it is only by standing on the shoulders of giants." Nostalgia aside, research done in the 60s and 70s did not reach the absolute end of progress in its areas.
The patent in this article, for example, claims both a linked list and a chaining hash table, where the lists used remove expired items whenever they're traversed. The novel and non-obvious improvement is the removal of expiring items upon access. This means that plain old hash tables and linked lists are not prior art in any way that invalidates the patent. If you could find an example of such a self-cleaning list prior to 1999, I'm sure Google would love to hear about it.
As another example, consider Google's patent on MapReduce. The concept of "translate sets, then combine sets" is certainly quite old, but that's not what's being patented. What's actually covered is the specific mechanism used to manage the processing tasks on a cluster, including allocation of workload, tables used, and the addition of intermediate steps to aid processing.
If you'd like to prove me wrong, please feel free to point out any patent from Google (or other reputable company) that is wholly implemented in something from the 70's.
But did your hash-pipe-induced idea involve exactly what's claimed in the patent, and no less? Is every single claim of the patent covered somewhere in your dissertation? Not in four separate projects over the course of 20 years, but all the ideas in a single location? No? Then please don't waste the court's time.
Nobody's trying to diminish the contributions of researchers in the 60s and 70s. Computer science has moved on, and further progress is being made by refining and extending earlier techniques. Do you really mean to imply that all computing progress stopped after 1979? What is happening now is that techniques from the 70s which were impractical at the time (though better in theory) can now be utilized. More use means more bright minds thinking about them, and more improvements. Those improvements can (and often should be) be patented, as they're the result of an investment in research. I'm terribly sorry if some researcher in the 70s dreamed up a great data structure and never got anything for it, but that's no reason to deny others the opportunity to profit from today's theoretical work.
...Or maybe the patent lawyers themselves already know of Knuth, since that's first on the list of prior art.
So the "patent"... would be considered a mere refinement of standard approaches.
That's exactly what innovation generally is. Personally, I doubt mankind has had an original thought in the past few thousand years, and even that's giving a lot more credit to the Greeks than I think they deserve. Everything is based on or inspired by something else.
Now the system in question here sounds like it is screwed up in some manner.
When I saw this story, I thought of a fun time back in college, where every year the campus cops would pull out their radar-and-sign machine (that thing that says "the limit is X, but you're going Y" and stick it on one of the main roads near campus. The limit was 25 MPH. Go above 30, and the display would flash red.
One year, they decided to put the machine on a different road that they thought would have more traffic. Somebody somewhere wasn't thinking, and for two weeks they warned drivers for going 30 MPH in a 35 MPH zone.
Having done no research on the matter apart from my own experience, I'd say it's because people who "abuse" sugar are more likely to also abuse more harmful things. I, for instance, will eat a candy bar when I'm staying up late. I'll also drink excessive caffeine and get less sleep than I should. Correlation, but not causation.
I worked at a company who marketed one of their offerings as an "isolated private cloud". Internally, everyone just called it a cluster.
Perl
Years ago, they certainly would have been "making waves", but they aren't as impressive now. CVS is surpassed by Subversion and Git, with the latter mentioned in TFA. Firefox is rapidly becoming a bloated but unremarkable product. Linux isn't really a "programmer's tool", so doesn't blong on the list in the first place. GNU Make has certainly earned its place among the annals of history, but it's only had four minor releases in the past decade. These projects are important, but hardly eye-catching for an article written today.
I'll get off your lawn now.
As a software developer, I like your first idea...
I volunteered in sub-Saharan Africa, and I saw parents fighting for the right to have their children caned in school. Is that what you mean?
In related news, criminals (and everybody else, for that matter) are increasingly using email and IMs to plan or discuss crimes (or family reunions).
What's happened to the rate of landline wiretaps, or good old-fashioned undercover following? How many telegrams per day are intercepted now?
Everything is done by email now, and I find it perfectly reasonable that the police are increasingly turning to email for evidence. That's where the evidence is.
On a very, very, very big slide rule, I could...
Really, I would do that one by hand, given only enough time. A problem of "find X to N precision" can be thought of as "guess a N-digit answer that's closer to X than any other N-digit guess". Extensive guesswork works just fine, though it takes about 20 tries...
Yes, I was using bisection to find solutions back in high school, when I felt particularly disinclined towards thought, and showing my work wasn't necessary. Yes, I often will solve such problems by hand. It's especially useful when you want to compute something that you know has only one optimum value, but it's painful (if even possible) to calculate directly. Of course, it's certainly not the answer Harvard's expecting, but perhaps it'd amuse some reviewer enough for credit.
Bisection also makes a fun hobby: Try to come up with problems that can only be solved quickly by bisection.
After utterly failing the Latin and Greek sections, I think I'd get a pretty bad reputation with any reviewer, even though I could do the rest just fine with a slide rule. Of course, I could follow up the geometry section with a lovely essay relating the theories of computability, genetics, and medicine, and the reviewer would be equally confused.
The parts that are important in modern innovation are still certainly appropriate for an entrance exam. The only difference I see between this and a modern exam is that the Latin and Greek sections have been replaced by English tests and some basic science questions. After all, the purpose for knowing Latin was that is was supposed to be the universal language of scholars, and during the burst of scientific progress following WWII, English took a firm grasp of that role.
There are so many who benefit from the community, and so relatively few who give back. So many people claim some excuse to not contribute anything to anybody without getting paid.
Then there's this guy.
I am honored to have shared a planet with him.
Given that I was processing text for a large part of the work (if not the majority), I'll have to pass. For what it's worth, I did look into Monet. I don't recall offhand why it wasn't selected, but just that it was one of the options.
I can certainly see history repeating itself, after 100 years...
Ugh!
The problem is, this "improvement" has be tried and failed repeatedly over the last fifteen decades. Here's the cycle. Some douche says, "horse-drawn carriages suck", and offers the world a "vastly superior solution." Ignorant masses flock to it because of the buzz; and frequently because they lack stablehands. A half or full decade later, the flocks wake up and say, "Man, were we douches. What were we thinking..." Followed by the legacy vehicles they created being retired, sold off to buy horses, or some poor chump is left holding the bag, maintaining a slowly dying machine without support.
And many benchmarks have shown, almost none of the horseless carriage solutions provide performance advantages over horses. Furthermore, those which do provide performance advantages do so by making fragility of workmanship its most significant feature; typically exceeding any performance benefits. Made worse, going the horseless route means leaving being a massive pool of proven, fast, and reliable vehicles, not to mention a large stablehand pool.
So basically, to break down the two camps, you do so with fundamental labels; ignorance vs wisdom. Or, those who know history and those doomed to repeat it. Thus far, there is every indication those jumping on the ignorant horseless carriage bandwagon are bending over to repeat history.
Looking back over history, I see many improvements that were often dismissed out of hand before they became commercially viable. It was only through the perseverance of their supporters and the eventual unbiased judgment of their adopters that such progress was eventually received. Now, I doubt NoSQL is going to dramatically change the data-storage world, but I do think its impact is worth some attention. The business push toward Big Data needs faster analysis than current techniques, and various MapReduce solutions, running on NoSQL platforms, show promising results that RDBMS solutions haven't been able to match. If nothing else, NoSQL provides pressure for RDBMS progress. Time will tell, but for now I'll devote a few hours a week to learning new technology, so I can make better-informed decisions when I need to.
Thank you so much for informing me of what I think. Until now, I was under the impression that I thought RDBMSs and NoSQL databases were actually different tools for doing different (but similar) jobs. This changes my whole perception of the world! Should I also think that a hammer and a nailgun are equivalent? Should I use sulfuric acid to wash my car? What about TNT to dig a posthole?
I have never argued that NoSQL is a replacement for RDBMSs. Not once did I suggested eliminating an RDBMS that does its job. I didn't claim that NoSQL is inherently "better" than an RDBMS. I do apologize for my mistakes, and now that I have been told what to think, I'll try harder to fit your preconceived notions of what "the NoSQL crowd" should do.
under no definition of 'toy' is included a billion data points.
From WordNet:
(n) toy (a device regarded as providing amusement)
My project was done in my spare time, with no funding from anyone but myself, for the purpose of my own entertainment and education. I believe that's a reasonable candidate for a "toy". It was a web crawler that scanned a few thousand websites (top hits on Google for various terms) and analysed their connections to each other. My goal was to see what sort of relationship "reputable" sites (like CNN, for example) had to "disreputable" sites (4chan), in the hope that there'd be a way to gauge a site's reputation by how it interacted with others. A few thousand sites, a few hundred links scraped from each... and that quickly makes a billion values to connect.
Sure, it could have been a million-dollar project from a big corporation, but it wasn't. It was just my hobby, run under a grad student's minimal budget.
taxi drivers should be familiar with passenger cars and the ... Space Shuttle Crawler
While I do love a good car analogy, the hyperbole ruins this one. A better example would be to suggest that taxi drivers (and especially those who may be getting a new car soon) understand what a "hybrid" car is, their advantages and disadvantages, and when they're a practical car to purchase. As emerging technologies, both hybrid cars and NoSQL databases provide additional options to be considered, and a competent driver or developer should have a basic understanding of both. To dismiss a new (or old) technology merely because of its age is a dangerous mistake.
Yes, no, and yes, in that order. I'm basing my answers on HBase, with which I have the most experience. My answers are also practically guaranteed to be wrong in somebody's eyes, because HBase is so much more flexible than an RDBMS. If I describe one way of doing something, another layout may work just as well, and somebody's going to favor that way.
How does indexing work in NoSQL? Are there EXPLAIN-type tools available?
EXPLAIN tools aren't really necessary in HBase, because almost all nontrivial queries are a scan over a small chunk of the alphanumerically-sorted rows. It will take a while, but please allow me to explain. Each row is a multi-value key-value store, with each value having a column name. If you really want to stick to the RDBMS style, you could have your key be a numeric row ID, and scan everything for every query. It would suck, because you're not using any indexes.
Indexes are more or less left up to the programmer. Creating an index is effectively just adding more rows to the table. For example, that RDBMS-style layout in the last paragraph could be a table of ID numbers, usernames, passwords, and permissions (for 50 billion people, I guess...). For whatever business reason, the main key will be the ID number. Those rows are easy. They have the expected value columns: username, password, permissions. To index by username, we add new rows, with just a column for the ID number. We could just duplicate the data, but let's not. Now, our table is going to be huge, but sparse. Half of the rows have three of four columns filled, and the other half has only one. Searching by name, it'll take two requests to get to the actual row we want, but that's okay. Doubling the amount of work lets us run faster.
The reason for that is HBase's split design. HBase's table is split into column families and regions. Column families are a means to group columns, so that even on data with overlapping key space, separate data could remain separate. Column families are stored as separate files in Hadoop. In our example, the username "index" could be a separate column family. That could speed up scanning, because the rows keyed by numeric usernames won't be interspersed with the rows keyed by user id. More importantly, the table is split into regions, each containing a number of rows. Those regions are also stored as separate files, and distributed across the entire Hadoop cluster.
The cluster is really where Hadoop gets its speed. If we were to run all of our processing from one central location, it would be horribly slow and require a ridiculous number of requests. Instead, we'll distribute everything, including the query, similar to how some RDBMS sharding schemes work. We send a request to all nodes, asking for "the row with the key that matches the value of the 'userid' column of the row with a given key". Each node will report back its results. Unlike RDBMS sharding, the partitioning is handled automatically by HBase into regions that are optimal. It's these regions that are scanned for every request.
After all of that, it should be quite clear: With HBase, the programmer is expected to know the layout of the data, and write requests based on the key. There is no EXPLAIN tool, because everything is just a key-value lookup.
Whew. Next question...
Can you do just about any query you could with SQL?
Yes, but it's different. Every lookup is handled by scanning a region (in parallel on nodes that have that region's data files), and checking each column of each row to see if:
Note that last item. The filter is simply a program that tells Hadoop whether the row (or some part of it) should be included in the returned results. That program can include other HBase requests, using other filters. If you're really stuck on using RDBMS
NoSQL certainly isn't magical, nor is it the solution for every problem. My point is that rejecting any job candidate because they're familiar with a new technology is asinine, and it's a sure way to be left behind by more open-minded companies.