Slashdot Mirror
Too Much Data? Then 'Good Enough' Is Good Enough
ChelleChelle writes "While classic systems could offer crisp answers due to the relatively small amount of data they contained, today's systems hold humongous amounts of data content — thus, the data quality and meaning is often fuzzy. In this article, Microsoft's Pat Helland examines the ways in which today's answers differ from what we used to expect, before moving on to state the criteria for a new theory and taxonomy of data."
SQL DBs are not appropriate for storing, processing, querying, and browsing unstructured documents.
Conclusion
NoSQL systems are emerging because the world of data is changing. The size and heterogeneity of data means that the old guarantees simply cannot be met. Fortunately, we are learning how to meet the needs of business in ways outside of the old and classic database.
Which was apparent to everyone, and missed the real point: We have lots of data, and we're too impatient to wait for it to be aggregated, synchronized and processed. There goes 10 minutes of my life I'll never get back.
Here's a hint: People working on the solutions to this problem work in the financial sector and in quantum physics.
The researcher is just throwing together a bunch of problems that have existed, in some fashion, for a very long time, and concludes with open questions rather than even vague proposals for solutions. So I would say this article is both too detailed, and not detailed enough.
It's not that there is too much data. That's not a problem at all. From my own experience (I work as a senior analyst for a multinational retailer employing around 200,000 people) it is rather that there isn't a single plan to utilize all the data we have available. Every time we introduce a new system or change the way we do something, the project inevitably drops a new table into our data warehouse. Now, this may seem like an acceptable way to do things, but after this has happened twenty times, it is nigh impossible to run a query that will return data from all these tables in any sort of reasonable time.
Would it cost more time, effort and money to properly introduce the new data to proper fact tables each time? Of course. However, the benefits would be that we could stop pretending that "we have too much data these days..." - because we don't. We just have too much mess with our data and it becomes unusable.
In the example above (different descriptions for green) the base system may need these particular terms, but if the data needs to be aggregated or used in another system, then the jobs that pass this to your data repository need to make those changes to adapt the data to work with the rest of your data warehouse. Having said that, if the new system is being developed inhouse, then during development the question should be asked "Can we store the color information in RGB right off the bat and adapt our own system to mask these values behind pretty descriptions?" rather than having to later do it via an ETL.
Moved to http://soylentnews.org/. You are invited to join us too!
It's not that there is too much data. That's not a problem at all.
Often, (more often then not, I contend), there is indeed just too much data.
Because we have all these marvelous computerized data capture system doesn't mean the data is necessary, useful, or worth keeping. However, someone always comes along in the project design stage and insists the millisecond by millisecond weight of a bag of popcorn weighed in real time as it is being filled is going to provide a wealth of data for the design of future bagging systems and materials handling in general.
The scale was only there to assure that 10 pounds were in the sack and to shut the hopper. Then some fool found out it measured ever few milliseconds and recorded the data.
So the project manager gets brow beaten into recording this trash which invariably never gets used for anyone for any purpose at any time, as those who lobbied for it wander off to sabotage other projects and never revisit the cesspool they created.
This happens way way more than you might imagine in the real world these days.
It used to be projects had to fight for every byte of data collected, there were useful sinks identified for every field. But with falling storage costs the tendency is to simply keep shoveling it in because its easier than dealing with the demands by those "researchers" looking for another horse to ride.
Sig Battery depleted. Reverting to safe mode.
This article is confusing because most of the verbiage is made up by the author (such as "inside" or "locked" data). It is also misleading because it seems to indicate that structured and unstructured data usage is the same. Well it's not - a very large proportion of unstructured data is blog posts and emails but the amount of search and aggregation that is performed on this type of information outside of a few major companies (such as Google) is very low, which makes this usage a niche and not a trend maker.
The reality is that there are three categories of data that are relevant for databases: numbers, text and spatial. Everything else, which falls under the umbrella of "binary", is very unlikely to benefit from a database engine; only the metada can be manipulated and this metadata falls under one of the other categories and is a very good target for ETL. And so far nobody came up with a reliable way to search binary, such as video or audio, without relying on heavy indexing, metadata or any kind of transformation that takes binary and make it text data.
If a piece of data cannot be searched or aggregated, it does not belong in a database, it belongs on a filesystem. Anything can be done with blob columns but performance is usually not very good because the database engine cache is not designed for large objects. NoSql or not.
Also there is so much happening with storage infrastructure, such as sub-volume tiering or block-level replication, any analysis of data that does not take a look at storage is flawed.
lucm, indeed.
+1 Insightful. I would argue that -- just like you have a lifecycle for software development -- you have a lifecycle for nontrivial amounts of data. Some data is useful in detail for a short term, but wherever possible it should be more coarsely aggregated as time progresses, and you should get sign-in from executives that it can be dumped after a period of time.
Where I work, I estimated the cost to upgrade our SAN to continue to store a set of large tables which helped everyone understand the cost in real terms. People tend to think once the data is imported or created that it's a small incremental cost to house it from that point forward, but backup times and storage along with execution plan costs increase with size. There is a performance benefit to this trimming; partitioning and check constraints will only get you so far.
What is difficult to gauge in advance sometimes is how the data will be used -- some things are obvious in the short-term, but as the company looks to different metrics or to shine some light on an aberration, you really need to be able to determine how quickly you can dump the detail. Get signoff then add some padding so you are conservative when you destroy. Make a backup "just in case" and delete it after a few months. The good news in my work is that changing your mind later to adapt to the new requirements means expectations are already set to change the way it works "from this point forward". There are many fields of work that do not have that luxury, because of the time or cost to gather detail again.
We don't read articles, just skim the headline, maybe the submittal, and then a few top ranked posts.
That's Good Enough! (tm)
If the people that write these stories would familiarize themselves with Information Theory (Claude Shannon, in the 1940's) then they'd understand that you still can't make silk purses from sow's ears.
Yes, it's a lot of records. Yes, the data entry people made mistakes. All this really means is that there's more noise in the data. As the signal to noise ratio declines, the value of the results also declines. Making decisions based on noisy data isn't science, it's only guesswork. That's fine for weather forecasting (a similar problem) but expecting the results from the described data to be more accurate than weather forecasts is foolish. Remember: garbage in, garbage out.