Whistleblower: NSA Is So Overwhelmed With Data, It's No Longer Effective (zdnet.com)

An anonymous reader cites ZDNet's Zack Whittaker report: William Binney, a former NSA official who spent more than three decades at the agency, said the US government's mass surveillance programs have become so engorged with data that they are no longer effective, losing vital intelligence in the fray. That, he said, can -- and has -- led to terrorist attacks succeeding. Binney said that an analyst today can run one simple query across the NSA's various databases, only to become immediately overloaded with information. With about four billion people -- around two-thirds of the world's population -- under the NSA and partner agencies' watchful eyes, according to his estimates, there is too much data being collected. Perhaps that's one of the reasons why NSA wants to dump the phone records it gathered over the past 14 years.

Comments

T.S.Eliot called it!

[Tim the Gecko]

"Where is the Life we have lost in living?

Where is the wisdom we have lost in knowledge?

Where is the knowledge we have lost in information?"

Search Tools

[Anonymous Coward]

Sounds to me like their search and filtering capabilities are the problem, not the amount of data available.

Re:Search Tools

[Anonymous Coward]

False positives, false negatives.

If you have a correlation that gives an impossibly good 1% false positive rate and 1% false negative rate, you can expect that 1% of the subjects you are looking for will be overlooked and 1% of those who you are not concerned with will match. So, let's apply that to the current nuisance.
1% false negative: for every 100 people with hostile intent, 1 will slip through the net and either bomb something or be stopped by civilians.
1% false positive: for every 100 people without hostile intent, 1 will hit the flags anyway.
With a data set of roughly 4 billion people, and an expected upper bound on hostile actors measured in hundreds of thousands, there will be roughly 100 false positives for every true positive. (*/4, so about 25 to 400 false matches for every valid one)

If the real threat is smaller (numerically, not necessarily in regard to potential destructive ability), then the math just gets worse.

Re:Search Tools

[Aighearach]

Even with good search tools, signal to noise ratio is still important.

Excess data with no correlation to the problems NSA is trying to solve (without getting into a debate over what those are) is simply noise.

Re:Search Tools

[Aighearach]

The signal to noise ratio doesn't change when you merely use less data.

False. Your statement is not true by default; it requires all the data to be known to be of equal quality.

Any time that some data is more strongly correlated than others, your noise is going to go down when you throw out the lower quality data.

Don't wave your hands, think it through and make a logical, reasoned argument.

Wait a minute...

[fuzzyfuzzyfungus]

Is this guy saying that the NSA used to be effective? I do remember them doing good work back when they emphasized playing defense; and they have probably assisted with some really juicy targeted attacks on specific people of interest(whether criminals or well-placed figures in governments we are interested in getting to know better); but has the Total Information Awareness/dragnet-all-the-data stuff ever shown the slightest evidence of providing useful data?

Re:Don't conflate those things

[kheldan]

How about YOU shut the fuck up? You're no better than a tinfoil hat-wearing conspiracy theorist; you represent the other end of the same gods-be-damned scale, and as such is equally elligible to be ignored.

The real problem? An ages-old human tendency: Power corrupts, and absolute power corrupts absolutely. Knowledge is power. Also, power seeks more power. These are no-brainers; no tinfoil hat required; everyone knows this. The NSA (and pretty much every other 'intellgence' organization) always wants more, more, more information, even if they can't use it -- but still they want more. They're like a little kid who discovered sugary candy; it's up to the parents to tell them no. Here in the U.S., citizens must play the role of Parents -- but we haven't been doing our job. The NSA/CIA/FBI/{insert government agency here} has been holding their breath until they turn blue, pitching fits, and throwing their dinner on the floor (read as: doing everything and anything they can to keep us in a constant state of terror) so we'll just give them the candy they want (read as: ability to surveil anyone and everyone) to keep them quiet. What they need at this point is a good spanking on their spoiled little bottoms (read as: U.S. citizens speak the hell up to their representatives and tell them in no uncertain terms that mass surveillance has to stop!) and send them to their room for a good long spell without dessert (no more data for you!). It needed to be done years ago but we've been neglectful, overly-permissive parents. Time to fix that!

Re:Don't conflate those things

[Anonymous Coward]

The reason the U.S. government can't get anything done these days is this Republican idea that Government isn't good for anything. This leads to putting people in charge of government agencies that don't believe that government can do anything. Think back to Heckofajob Brownie during hurricane Katrina. The guy had no emergency management experience and ended up running the worlds largest emergency management organization. This is pervasive through many agencies though and leads to a self fulfilling prophecy.

We were able to effectively end childhood hunger in the 70s, we were able to create the national highway system, put a man on the moon, lots and lots of large projects that were ultimately very successful but now a large chunk of the country thinks we don't even need the IRS anymore. We have a Presidential candidate campaigning on that very idea. Cruz is probably equally as crazy as Trump but they represent a good solid chunk of the population.

I hate how the conversation has been turned into Government should do everything versus nothing. That's why I like Bernie, he thinks Government could do more but recognizes that some things are better in private hands, he probably goes too far but when you think of it more as a direction instead of an overnight mandate things look a whole lot more sane. My health or my parent's health should not come down ensuring someone makes a profit for a hospital stay. Injecting money into health is counter productive, much the same way insurance is. Insurance companies should take our money and use it to invest, but they have people who's job it is to deny you your claim rather than figuring out how to fairly deal with a situation. So we pay care insurance for years and don't use our benefits, the moment we have a traffic accident we have to start paying more, or we have to hire a lawyer to make sure the insurance company actually provides the coverage they promised.