Slashdot Mirror
How Big Data Creates False Confidence (nautil.us)
Mr D from 63 shares an article from Nautilus urging skepticism of big data: "The general idea is to find datasets so enormous that they can reveal patterns invisible to conventional inquiry... But there's a problem: It's tempting to think that with such an incredible volume of data behind them, studies relying on big data couldn't be wrong. But the bigness of the data can imbue the results with a false sense of certainty. Many of them are probably bogus -- and the reasons why should give us pause about any research that blindly trusts big data."
For example, Google's database of scanned books represents 4% of all books ever published, but in this data set, "The Lord of the Rings gets no more influence than, say, Witchcraft Persecutions in Bavaria." And the name Lanny appears to be one of the most common in early-20th century fiction -- solely because Upton Sinclair published 11 different novels about a character named Lanny Budd.
The problem seems to be skewed data and misinterpretation. (The article points to the failure of Google Flu Trends, which it turns out "was largely predicting winter".) The article's conclusion? "Rather than succumb to 'big data hubris,' the rest of us would do well to keep our skeptic hats on -- even when someone points to billions of words."
For example, Google's database of scanned books represents 4% of all books ever published, but in this data set, "The Lord of the Rings gets no more influence than, say, Witchcraft Persecutions in Bavaria." And the name Lanny appears to be one of the most common in early-20th century fiction -- solely because Upton Sinclair published 11 different novels about a character named Lanny Budd.
The problem seems to be skewed data and misinterpretation. (The article points to the failure of Google Flu Trends, which it turns out "was largely predicting winter".) The article's conclusion? "Rather than succumb to 'big data hubris,' the rest of us would do well to keep our skeptic hats on -- even when someone points to billions of words."
A key that opens many locks is a great key.
A lock that is opened by many keys is a shitty lock.
When all that data that Google, Microsoft, etc gets mined for potential criminal activity, many peoples lives will be ruined by false positives. This is why I try to avoid being monitored in the first place.
It's part of the reason why I'm always skeptical of one-off statistics being used to try to draw conclusions on things that aren't there.
I'm reminded of this one article I read about a week ago, where it tried to draw some sort of conclusion as to how sexist the movie industry is by looking at the age ranges for male and female characters in movies, and then seeing that it was mostly women in their early 20's while men had a broader age range. And sure, it might look that way if you only go with that single statistic.
Problem is, there's other statistical data that indicates that men of all ages find women in their early 20's more attractive, whereas women are attracted to men who are closer to their own age. Last I checked, the whole point of most movies is to make money and thus appeal to the largest number of people possible, and wouldn't you know it, it happens that leading roles tend to go to actors and actresses that others find attractive. Imagine that.
Film at 11.
We suffer more in our imagination than in reality. - Seneca
In every discussion of Big Data that I've ever read it seems that the "goodness" of big and especially unstructured data sets is basically taken for granted. First off, why shouldn't unstructured data itself be considered something of a failure? If the data collection was more organized and thoughtful from the start, we wouldn't be looking for all sorts of esoteric algorithms and methods for cleaning up messes that seem mostly to be the result of laziness in the initial data collection and not any inherent complexity in the data being collected. Second, why is more information necessarily a good thing? Hasn't anyone ever heard of information overload or signal to noise? People, and especially young people, trash the old ways but I wouldn't be surprised if SQL and table oriented storage is still alive and kicking long after we're all gone.
Lies. Damn Lies. Statistics.
Big data doesn't have to be about an analysis of that data, it can also be about finding a needle in a haystack.
Googles search engine is searching an enormous set of data, bigger than any big data set and people find it useful every day. Even normal people not scientists.
I can go to the library and use the index to find books on a subject then flip through pages for weeks to find a how to procedure for making coal tar, or I can just search Google for it and have it in a moment.
For me that's what book searching is about, unfortunately Google does not offer this and that is why I wrote my own engine for searching the Gutenberg database. That's what big data can be about, it doesn't have to be about predicting or finding trends just because some people use large data sets for that and a bunch of people decided to term the idea. The term doesn't lock it in.
Are you suggesting that the "skeptic hats" be worn even by those who possess minimal domain specific and analytic knowledge?
a lot of the big prediction sites have been predicting the kansas city royals to be average to poor the last few years. so far they have been to the world series twice and are in first place in their division this year. all with average to slightly above average mainstream stats but if you look at them they built a team using a team strategy instead of simply signing guys and looking at individual stats
Getting folks in the Bay Area to realize that is still an unsolved problem. Maybe they have an AI team working on it.
In all seriousness, I saw this a lot when working within a monitoring team, and in consulting I've done for other orgs. Big Data is great for vast, multi-dimensional analysis of massive amounts of data, but it's not a substitute for domain knowledge about *WHAT* you're monitoring, critically thinking about what you're looking for and what types of failure modes might occur, and simple(r) heuristics for triggers.
Trend analysis is very useful as an adjunct, for example, but within a server monitoring context it's not a *substitute* for having hard limits on, say, CPU load, or HTTP response time, or memory usage.
Somehow, people managed to come to conclusions and make good decisions even before we had terabytes of raw data being sifted through by statistical algorithms to come up with a result.
To place it into a broader cultural context, I see this in parallel with "data fetishisation" where nothing at all can be possibly true unless Science. And Data. Hipster praying at the altar of data.gov as some sort of left-wing (or Millennial) shibboleth for smug certainty when the basics -- the entry-level, basic 101 class of domain knowledge for the field -- is being forgotten.
I'm all for bringing in new tech and new analytic techniques, but you can't look at it as a panacea for failing to understand what's going on in your domain on a philosophical level.
Hire a Linux system administrator, systems engineer,
"Well, if I generate (by simulation) a set of 200 variables — completely random and totally unrelated to each other — with about 1,000 data points for each, then it would be near impossible not to find in it a certain number of “significant” correlations of sorts. But these correlations would be entirely spurious. And while there are techniques to control the cherry-picking (such as the Bonferroni adjustment), they don’t catch the culprits — much as regulation didn’t stop insiders from gaming the system. You can’t really police researchers, particularly when they are free agents toying with the large data available on the web.
I am not saying here that there is no information in big data. There is plenty of information. The problem — the central issue — is that the needle comes in an increasingly larger haystack."
Just because you have amassed the largest pile of shit ever doesn't stop it from being shit....
The problem is not some quasi-religious cult but rather the bandwagon approach toward a rising skill set in the labor market. There is a demand for skilled analysts, and there is a need for hundreds of thousands of people to find new work at the same time. Those conflicting forces reduce skill applied in jobs filled by the mostly unqualified. There is no substitute for a quality university education with actual mathematical background if not statistics concentration. These errors encountered now are actually not all related to statistics, some are blind faith in algorithms that are functional only by heuristic not by well developed theory, or at least not be accessible theory outside of a specialized graduate education in both computer science, business skills, and math. Those skills are in fact what is required instead of resume filler used by people who create the problem you've identified by also falsely attributed cause.
... ignorant comments. Good to know the competition pool is not as large as I expected.
I agree this article has no significant substance. Of course having more data to run statistical models against gives more confidence. So does hiring more Ivy League grads to work as your analysts or paying pricey firms like McKinsey to help make strategic decisions.
Everything that can help boost confidence has the potential to boost it too far. And everything that will help you make better decisions is confidence boosting. So unless you intentionally want to limit your ability to make intelligent decisions, you will have to carefully monitor whether you have too much confidence.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
Getting data is dead easy. I can get you gobs of it. I can store it fairly quickly.
Now for the hard part. What are you trying to find? Have we been collecting the right data? Is it in the right form? Do we actually have enough? Is it at the proper interval? These are where most people fail and they just keep collecting more of the same data. Even though it has 0 use for them somehow magically expecting the data to self organize itself.
A lot of people put in crap to get past prompts, or answer ideally instead of truthfully, etc. You have to imagine a lot of this data is biased and doesn't reflect reality anyway.
Twinstiq, game news
The key advantage of big data is the ability to show us where to look. But after that we need to dig further with much smaller data and science to see what the cause is.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
There was a very interesting bit of sleuthing done to track down biases in the popularity of certain dates in scanned books. like the prevalence of Sept 11 *before* 2001.
"Those conflicting forces reduce skill applied in jobs filled by the mostly unqualified." Did you draw this conclusion by collection a lot a data and running the data through your own analysis? Or do you have some other way to prove such a broad accusation? The biggest problem we face to day is from those who use statistics to support their cause and opinions. We are constantly bombarded with poll results and nobody every questions how these results are derived. What statistical methods are being used that allows the pollsters to take a very small sample size and project those results on very large datasets? How do you ask 500 people their opinion on something and then apply those results against 400 million people?
https://xkcd.com/1138/
Exactly! Big Data is about finding patterns, not conclusions. The whole point is that humans are capable of searching through only so much information, and at some point you need a computer to do it for you.
Of course, once a pattern is found, it's up to humans to determine if it makes sense -- and you'd do the same for any pattern found by a human.
dom
"data fetishisation" where nothing at all can be possibly true unless Science. And Data.
The problem is that it's all data, very little science.
Real scientists know how to scrutinize their data, and how to rule out false positives. Actual science will not only give you a statistical level of confidence, but use domain expertise to the uttermost to rule out systematic errors. A nice case study in that regards is the recent LIGO gravitational wave results.
Most of the people who like to call themselves "data scientists" these days know as much about science as "computer engineers" know about proper engineering.
Try reading that again, without the political BS that you added.
I call this approach "adding more haystacks".
Ask me about repetitive DNA
...false data create big confidence.
You measure the wrong thing ten times, and get the wrong answer. You measure the wrong thing ten billion times, and you still get a wrong answer.
It's almost like quantity and quality are different things!
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
That's "BANANAL"
I'm reminded somewhat of "The Bible Code" - the theory/idea that there is a bunch of stuff hidden in the bible, visible when viewed different ways (like when skipping characters, etc - Google it) The reality is - the bigger the dataset - the more patterns - even false patterns may be present in it. If I had a billion money's, what would they type...
Getting data is dead easy. I can get you gobs of it. I can store it fairly quickly.
Now for the hard part. What are you trying to find? Have we been collecting the right data? Is it in the right form? Do we actually have enough? Is it at the proper interval? These are where most people fail and they just keep collecting more of the same data. Even though it has 0 use for them somehow magically expecting the data to self organize itself.
Good point, also one failing I see with big data analyses is when the researcher is looking for or referencing a finding within their data that is at the very edge of the resolution of what they are able to measure, and they think they can make up for it by pouring more data in without examining whether what they are seeing is a repeatable measurement or if it is a fluke that is thrown one way or the other due to variations that are below the resolution of what they are able to measure.
Prime example:
The people who point to cell phone radiation causing cancer. You could appeal to ignorance and say that there is no way that we know for sure that a cellular phone can't cause cancer, however most people who use cellular phones also use microwave ovens and the radiation from a microwave oven is on the order of a million times stronger. That observation aside, you can see a correlation between people who use cellular phones and people who get cancer, however there are numerous confounding factors that make showing a causal relationship problematic in the extreme.
Extra skepticism is required to explain the data and to make lucid assessments of the data and conclusions you draw from it, otherwise it is not science it is more along the lines of politics. Often times I see researchers trying to prove a conclusion that they have already come up with, rather than testing a hypothesis and there is a world of difference between the two concepts.
"Those conflicting forces reduce skill applied in jobs filled by the mostly unqualified." Did you draw this conclusion by collection a lot a data and running the data through your own analysis? Or do you have some other way to prove such a broad accusation? The biggest problem we face to day is from those who use statistics to support their cause and opinions. We are constantly bombarded with poll results and nobody every questions how these results are derived. What statistical methods are being used that allows the pollsters to take a very small sample size and project those results on very large datasets? How do you ask 500 people their opinion on something and then apply those results against 400 million people?
By assuming random sampling, that's how. Whether that assumption is correct is a critical issue, but that is the case for any universe population significantly larger than your sample set. 500,000 or 400 million really does not matter - if you are ignorant of the demographics and how those interact with your sampling strategy, you're not gonna get a correct result.
If, on the other hand, you somehow manage to do random sampling of the true population, 400 people would be enough to nail preferences down to a few percent, (almost) no matter the total population size. And I guess this is the danger of statistics and big data. Intuition says one thing, simple statistical assumptions say another, and a more thorough treatment is rare.
The lack of thinking is somewhat appalling. I am a "data scientist". I came from one of the science fields that understands data, high energy particle physics. People are often surprised when I tell them that their fancy map-reduce tools are not particularly interesting when it comes to actually understanding your data. The tools are not interesting. Do you hear that "big data" conference organizers. Too little time is spent understanding what the data is telling and how do you know that it is telling you that and not something else.
Making sense of data takes knowledge and common sense. Qualifications I often find lacking in many of the job candidates I've interviewed. They know how to run the latest tool, but can't explain what the results mean when they get them.
Well, all human decision-making is appalling, really. What one should be wary about is how others will misunderstand and abuse knowledge.
Captcha: revoke
I know a lot of humans don't like the way the big websites save their information and sell them. But there is a reason for this, if we say Google they sell their information to all who want to promote their products through Google. That means Google creates more jobs and wroth in the world. Companies can promote their products and their customers can see them in Googles search engine. Alot of SEO companies are dependent on Googles way to sell more information.
PanadasDigital is a SEO bureau
Garbage In, Gospel Out
Many use big data systems and techniques to:
- Identify potential new customers for products and services. Mistakes here result in poor choices and losses.
- Identify and prevent fraud of all types. Mistakes here result in losses.
- Identify existing customers that could be successfully marketed new or additional products and services. Mistakes here result in disgruntled customers and losses.
Sometimes it's hard to determine if a technology is useful or even functional. Money often is a good indicator.
deleting the extra space after periods so i can stay relevant, yeah.
Oh man, I ran square into this just last week. This guy was claiming to work in big data as an economist. Said any sort of inefficiency should ultimately impact the GDP. I countered that, there are lies, damned lies, and statistic and that it might not make him comfortable, but the metrics he's using could be lying to him.
And get this: "I work with data. Statistics is for losers". ... Can you believe that guy? Even after I point out that while every call to Map() might sort data very nicely, every Reduce() call AGGRIGATES data into a statistic, he still stuck to his guns and claimed to be somehow holier then icky statistics.
The summary example (Google scanning 4% of books), while it may be "a lot" of data, isn't really big data, is it? I understand the whole point about more data not necessarily being better, but here I don't even think the example shows proves the point?
Who put this thing together? Me, that's who.
We used to teach stats with the principle "data size cannot overcome data basis" back in my day ..
Big Data and Statistics were the problems Hari Seldon ran into, didn't he? Only worked with supra-large populations.
Tracy Johnson
Old fashioned text games hosted below:
http://empire.openmpe.com/
BT
Gary Taubes (author of "Good calories - bad calories" and "Why we get fat and what to do about it") is my favourite scientist because he just exhibit such a healthy, integrated "given that what we believe today is correct" attitude, e.g. being totally open to be proven incorrect. There is a saying "follow those that seek the truth, run from those that have claimed to found it", and Gary is most certainly a truth seeker in that respect.
For instance in the interview https://www.youtube.com/watch?..., he says during the first minutes "That's what we should believe until we have remarcable evidence to reject it" and "Don't take my word for it, anyone can try it out for themselves", without this being specifically emphasised or made a big point of, it is just his natural way of reasoning which I love so much.
And now to what triggered me to answer your post: I think it is later in that interview that he points out that observation studies can only be used to form hypothesis, dawing conclutions from them is wrong, you actually need to perform controlled experiments to do that.
When you are sure of something, you probably are wrong (search for "Unskilled and Unaware of It").