Slashdot Mirror
The Big Promise of 'Big Data'
snydeq writes "InfoWorld's Frank Ohlhorst discusses how virtualization, commodity hardware, and 'Big Data' tools like Hadoop are enabling IT organizations to mine vast volumes of corporate and external data — a trend fueled increasingly by companies' desire to finally unlock critical insights from thus far largely untapped data stores. 'As costs fall and companies think of new ways to correlate data, Big Data analytics will become more commonplace, perhaps providing the growth mechanism for a small company to become a large one. Consider that Google, Yahoo, and Facebook were all once small companies that leveraged their data and understanding of the relationships in that data to grow significantly. It's no accident that many of the underpinnings of Big Data came from the methods these very businesses developed. But today, these methods are widely available through Hadoop and other tools for enterprises such as yours.'"
Couldn't this be done using regular sql and an indexed timestamp column?
There are 2 types of people in this world. Those who understand ternary and those who don't.
It isn't about Facebook so much as it's a shift in what problems are practically solvable.
First, realize that traditional approaches like SQL are limited mostly by the single box (or the few mirrors) the platform runs on. Querying a large (a billion rows) table can take minutes on a very fast machine, hours if there's significant disk access needed, and months if the query's complex enough. Clusters can process those same billion records far faster, bringing that time down from months to hours, or even seconds for a simple scan. Advances in cluster computing over the last few years have made this parallel processing much easier.
The promise is that problems that were previously too big to even think about are now easy. If your solved problem is something people want, like showing what their friends are up to, your product will do well.
You do not have a moral or legal right to do absolutely anything you want.
Assuming the maximum configuration is thousands of cores, how does it compare in other aspects to Facebook's 23,000 cores and 36 petabytes of data, with unlimited scalability to come?
For all intents and purposes, mainframes are still mainframes. They're parallel, and they grow, but they still have those limits that clusters just don't have.
(I consider price to be a limit as well)
You do not have a moral or legal right to do absolutely anything you want.