Edward Snowden Kills Team Trump's Conspiracy Theory By Explaining How The FBI Can Quickly Comb Through Email (geekwire.com)

FBI director James Comey told Congress Sunday that the further investigation of emails related to Hillary Clinton didn't turn up anything that would cause the bureau to recommend charges against her. The FBI had reviewed over 650,000 emails under nine days. Upon hearing this, GOP presidential candidate Donald Trump and his supported started to question whether the FBI could go through all those emails in such a short period of time. We will never know for sure until the FBI explains its process to us all (which is unlikely to happen), so people turned to Edward Snowden over the weekend for answers. And Mr. Snowden didn't disappoint. From a report on GeekWire: How easy would it be to cull out the duplicate emails? Outspoken journalist Jeff Jarvis posed that question to Snowden in a tweet, and got a quick response: "Drop non-responsive To:/CC:/BCC:, hash both sets, then subtract those that match. Old laptops could do it in minutes-to-hours."

Comments

Unless we know the number of non-dupes.

[Anonymous Coward]

... then we still don't know how plausible it is that they reviewed XXXX number of emails in 11 days, after taking months to review 80,000 emails before.

Re:Unless we know the number of non-dupes.

[Kichigai Mentat]

Because each email had to go through several Federal agencies to have any retroactively classified information redacted before they could be publicly released.

In this case we have a trove of emails . Also note what Comey said: he said that this doesn't change their decision with regards to recommending to indict Clinton or not, so that means once they hit this point all they have to do is figure out if Clinton had sent any of the remainder of the emails, which is easily accomplished with a simple search.

Badda bing, easy work.

Re:Unless we know the number of non-dupes.

[beelsebob]

The point is that Trump's supporters have no solid evidence that there was not enough time to review the emails.

Their extreme view of it is that all 650,000 emails were relevant, and that therefore it should have taken 18 months * 650,000 emails / 80,000 emails = 146.5 months to review them.

The other extreme of possibilities is that the FBI filtered the emails by "To/From 'Hillary Clinton', date within period of being secretary of state, not a duplicate of any of the already reviewed emails" and the output of the filter was 0 emails.

The truth is likely to be somewhere between the two, it's also likely to be towards the very low end of the range.

Re:Unless we know the number of non-dupes.

[PvtVoid]

Their extreme view of it is that all 650,000 emails were relevant, and that therefore it should have taken 18 months * 650,000 emails / 80,000 emails = 146.5 months to review them.

Had it actually taken that long, they would have claimed that there was a conspiracy to delay an indictment.

Re:Unless we know the number of non-dupes.

[alvinrod]

Presumably part of the difficulty with solving any problem the first time is figuring out a good method to use, implementing it, and testing to make sure it works properly. Once you've solved that problem, it becomes much easier to do it a second time because you know what needs to be done and people have experience doing it so it goes considerably faster. That isn't proof that they were actually capable of going through everything, but if we want to think about it logically, the outcome was always going to be the same.

First of all, if someone actually had something really damning they would have released it months ago if they had any intention of going public at all. Anything that's immediately and undeniably legally actionable gives you perfect blackmail material that can be used to control the president of the United States. No one in a position to collect that kind of information (blackmail) is going to waste that kind of opportunity. If you want to argue that someone who might have said information wants to release it to cause disarray, there's a more compelling argument that disarray is maximized if you only release the data some time after a Clinton victory.

It's therefore safe to assume that there's no silver bullet in the new data dump to start with and that it only contains more of the same, which the FBI have already said isn't going to get anyone to indict Clinton, even though they've essentially stated she's been pretty duplicitous about the whole thing. She's hardly the only corrupt person in D.C. and it's more likely than not if she were to go down, she could take a lot of other people with her on both sides of the isle. As much as the Republicans love talking about how corrupt she is, exposing it probably slits many of their own throats in the process. Elections are basically a trial in the court of public opinion anyways, so making swing voters think Hillary is guilty is effectively just as good as legally proving it.

Re:Unless we know the number of non-dupes.

[Anonymous Coward]

Which once again begs the question why Comey broke the FBI guidelines to not insert himself in the middle of the political process, especially so close to the election. His ass should be canned for throwing all that red meat to the Trump campaign 11 days before the election as Clinton was pulling away in the polls. How do you defend yourself against innuendo from the FBI?

FOX news seemed to be getting daily updates on how Clinton was going to jail immediately after the election from an unnamed FBI source. They would report them in primetime with great fanfare and then retract them Friday morning where the viewership is much smaller with no fanfare.

The FBI is a mess right now.

Re:Unless we know the number of non-dupes.

[Anonymous Coward]

Trump nuked his campaign when the "grab them by the pussy" thing hit the news cycle. Clinton, seeing that there was no time for the Republicans to scrounge a new candidate, ran with it and hit Trump hard. So Trump responded with the whole "rigged election" schtick. Hillary responded by having a close aide with a sketchy husband release a bunch more inert stuff, then called in a favor with Comey to publicize it. Now, Trump can't call the election "rigged" without looking like a complete twit. And a week later, the all-clear signal comes from Comey, just in time for the election. Hillary 2016.

Is it fact? I have no idea. Is it farfetched? Not terribly. Hillary is a politician with a lot of allies. She could probably pull off something of this caliber. Maybe she did.

(BTW, I'm not voting because I have no confidence in the US government or any other government. I really could not give less of a crap who wins tomorrow.)

Re:Unless we know the number of non-dupes.

[J053]

Comey probably knew about the (apparent) cell in the NY FBI office who have a hard-on about HRC, and knew they were going to leak something, so he decided to get out in front of it with his letter to Congress. I do give him some credit for his subsequent letter yesterday, but it would have been better IMO if he had shut up and let the leaks happen, then come down like a ton of bricks on the leakers.

Re:Unless we know the number of non-dupes.

[hajile]

Democrats said he was an outstanding, honest man when he dropped the case (while Republicans decried him as dishonest). When the case came back up, the Democrats and Republicans both completely flipped positions. I don't know if he's playing politics or not, but it seems obvious that everyone's hatred/love is tied to their party rather than the truth.

In any case, what could he have done differently? He announced the case closed going into election season. If he didn't mention the new evidence at all, then congress would have him for perjury sooner or later. If he released after and Hillary won, everyone would say he killed the investigation so Hillary could win. If he released before and Trump won, he would be accused of bringing up the investigation again so Hillary would lose.

Given that Hillary looks to win the election, he can claim that his release didn't adversely affect the election. That's about the best outcome he could hope for.

Bingo

[PvtVoid]

It would be very easy via automation to tag the emails which are dupes of ones already in the data set.

Which, apparently, was all of them. No shit, Sherlock.

He didn't do shit

[darkain]

Snowden didn't do shit. As much as we all "love" him for his previous leaks, he didn't shoot ANYTHING down. He only answered how to dedup a list to make it smaller, not answer how large the list would be after the fact or how long it would take to comb through said remaining list.

Re:He didn't do shit

[110010001000]

Um, if you "deduped" the list using hashing the resulting list was zero because there were no fucking new emails. It would literally take less than 5 minutes to run the algorithm.

RegEx, it's a hell of a drug.

[Kichigai Mentat]

Seriously, anyone who's ever had to do de-duplication or pattern matching or anything like that could have told you how easy this is to do. It's almost like computers are good for this kind of stuff!

Re:RegEx, it's a hell of a drug.

[MightyMartian]

The problem is that if de-duping is easy, that means that it could be quickly ascertained if the new mail dump had anything significant in it, which means there was only a brief period of time in which some fantastical new load of Clinton-destroying emails would be found, and if that were the case, then the Trump camp was literally hanging on to a false hope.

So now we have some of the most tech savvy people on the Internet pretending they're simpering halfwits with know technical know-how at all, just so they can keep a faint hope alive. I guess they can keep imagining Clinton impeachment, though they won't have the votes in the Senate, and it may turn out they don't even have the votes in the Senate to do much else but filibuster Clinton nominees.

Re:RegEx, it's a hell of a drug.

[thegarbz]

Seriously, anyone who's ever had to do de-duplication or pattern matching or anything like that could have told you how easy this is to do.

So not a Trump supporter then. RegEx? That's just a Clinton propaganda piece. Computers too good? They are made in China. Nothing good comes from China. You're just a paid shill standing between us an a great America.

Re:RegEx, it's a hell of a drug.

[Xest]

As much as you may wish to hope you can stop it the trajectory is towards liberalism, this is why even with the odd hiccup such as Brexit and possibly a Trump victory they're still ultimately only blips on the overall timeline of history. Trump and Farage alike are entirely dependent on people who will be dead in 10 - 20 years to even remotely achieve the numbers they need to reach the goals they want. Beyond that they and their mindsets are well and truly done.

Liberalism goes hand in hand with intellectualism, as people become better educated on average, more knowledgeable on average, they want more freedom, more rights. They're never going to vote for someone who wants to create interment camps, who calls for political opponents to be assassinated, who hates people over arbitrary and meaningless traits such as sex, sexuality, skin colour and so on. The only way you can stop this tide of change is by making people more stupid, and guess what happens when you do that? you lose the global geopolitical race to someone who hasn't made their population more stupid, and who is progressive, does respect intellect, and in turn pushes human advancement forward with or without you, at which point you adapt and follow or face poverty and irrelevance.

Human advancement is a basic instinct that no amount of conservatism can put a stop to. Japan and Germany didn't lose World War II because of any particular military strategy, because of bad luck, and so forth, but because when you don't respect intellectuals, those that do get things like the atom bomb instead, and then they win.

When you understand this, you'll understand why liberalism is such a powerful and effective force that you should probably embrace, rather than continue to fight a war you will never win, as much as a handful of ultimately irrelevant short term victories many excite you.

This is why liberals have nothing to worry about. They're not losing, and human progress ensures that will always be the case - it's been the overarching trajectory throughout the entirety of human history.

Email Threading and DeDupe

[dave562]

I work for an organization that is heavily involved in electronic discovery processing for large corporations, law firms and the United States government.

Email threading, and duplication detection / dedupe are standard tasks that are performed on a daily basis on huge datasets. (As part of the Processing phase of the EDRM model.)

It is not at all unfeasible that the FBI could have used standard, off the shelf software to identify duplicates and generate an exception report for all 'new' emails that were not in the previously collected datasets.

https://en.wikipedia.org/wiki/...

This just in

[Verdatum]

Guy answers high school freshman-level tech support question. We'll have details on this exciting story as they develop.

Re:This just in

[Kichigai Mentat]

This seems to be the case. Demographic polling shows him losing the vote with anyone who has more than a High School Diploma.

Re:This just in

[smooth wombat]

Well apparently Trump and his supporters aren't high school level then.

Trump does poll well with the uneducated.

Re:This just in

[MightyMartian]

Well, considering that the Trump crowd, including a few posters here who should have the ability to actually write the code to de-dupe a bunch of fucking text files, claiming this was some impossible task that could not be completed in a few days, I think it was useful to have story reminding those poor suffering Trump-support /.ers who seemed to have a major brain fart about some pretty trivial algorithms.

Re:removing dupes is easy...

[110010001000]

There were no "non-dups". There were no new emails. Not too hard to grasp.

Re:Stupid

[beelsebob]

the point is, if they can do it this quickly, why did take some many months the first time they investigated this?

Because the first time, they had tens of thousands of emails, none of which were duplicates of ones they already had, and all of which were sent to or from Hillary Clinton, and all of which were sent during her time in office as Secretary of State. Further, they had to investigate several different avenues for finding more emails.

This time, they have hundreds of thousands of emails, only a small percentage of which were sent to or from hillary clinton, only a small percentage of the remaining were sent while she was in office as secretary of state, only a small percentage of the remaining were not duplicates of existing emails that they had already reviewed. The result is that even though the original number was larger than the original cache they had to search, it's likely that they only had to look through a couple of hundred in the end this time.

I don't get why people are having such a hard time grasping this.

Re:Ignorance is bold

[bobbied]

You know, it would be refreshing if folks could just understand all of what Trump is saying for what it is.. Politics... You may consider yourself smart because you can see how technically stupid what he's saying is, but let's not forget the other major candidate's stupidity either "Wipe it? You mean with a cloth?" when she clearly knew better. Surely there is plenty of duplicity to go around here. Trump is just saying stuff that he thinks appeals to folks who might be willing to vote for him, and you have to admit that for the FBI this was pretty fast. At this point, with the election nearly here and the obviously narrow margins by which this race will be decided, you say what you need to, and Clinton (and her campaigners) are doing this too.

Trump may not be all that clued in on technology or how a review of 650K E-mail's may have actually taken place in such a short time, but for not being a career politician he's obviously a very quick study. Just think, it was less than a month ago he was doing 3AM twitter wars about stupid stuff and now he's going to win or loose by the skin of his teeth. Not bad...Well Better than I expected anyway...

Re:Drone Snowden's ass already

[MightyMartian]

Not really. I thought Occupy was an absurd waste of time, and I thought the whole "1%er" nonsense was simply contrived. Not that I don't want to see the wealthy made more accountable, and large corporations brought more firmly under the rule of law, but to imagine a guy like Trump, whose business history has been one of screwing over investors, using every trick in the book to evade taxes, and who is, by definition, one of the Elite, was going to bring the "1%ers" to bear was so ludicrous and laughable that I just have to imagine that most of his supporters are either complete morons or were more likely hoping he'd be so fucking awful that he'd bring the system down (which is absurd, the Founding Fathers built the system to deal with even the most terrible Presidents).

Re:But

[Galaga88]

Because it took that long to go through 33,000 emails.

They didn't have to go through 650,000 this time. They could just analyze the data to see if any were new from the prior set.

Which is exactly what you know, the original fucking article you decided not to read says.

Re:Drone Snowden's ass already

[__aaclcg7560]

That you elected a lying bitch instead of a lying asshole?

As one Republican consultant said in a Politico article, "Given a choice between crooked and crazy, the American people will always vote for crooked."

Re:Drone Snowden's ass already

[AutodidactLabrat]

and we will pity sheeple like yourself for thinking it's ok to have a corrupt felon running your country.

A felon is a person convicted of a felony
Thus, OBJECTION YOUR HONOR!! Reference to facts not in evidence anywhere!

Re:Ignorance is bold

[MightyMartian]

I don't think anyone can actually understand what he's saying. His supporters pick out the good bits of his word salads and declare him a genius, his opponents pick out the bad bits (not exactly a hard job), and declare him a dangerous idiot. There never really was a serious effort to put out a message or a coherent set of policies. It was just sound bites wrapped up in some sort of bizarre alpha male charisma schtick. Trump was the product of a whole lot of peoples' imaginations. Honestly, up until the last week or so, he hasn't even acted like someone who had the vaguest hope that he'd ever be president, and to wait until the last week of an election before you decide you're going to behave with some self control and dignity indicates to me that you're either a complete idiot or you never seriously wanted the job to begin with.

Trump has wasted a vast number of the GOP's resources, probably harmed a number of downticket races, enough that it's likely the Senate will either be deadlocked or at least marginally in the Democrats' hands, not to mention the damage done to the GOP's efforts in states like Florida and Arizona to reach out to minority voters. And for what? To be a hit with a demographic that the GOP has recognized for eight years now will fade in importance?

Any Republican angry at what will transpire tomorrow shouldn't blame Clinton, they should look at the fools in their own party that put one of the most unsuitable presidential candidates in modern US history in the place he's in right now.

Re:But

[quantaman]

That still doesn't explain why the FBI boasted that they had 400+ agents working for many months for 33,000 emails and yet magically can go through 650k in just a few days. Either they were lying before, or they are lying now.

There's a third explanation. You don't understand how computers and basic problem solving works.

The 400+ agent review involved someone personally reading and evaluating every email.

The 650k email review involved extracting the small subset of email to/from Clinton, extracting the even smaller subset of emails not in their previous already-reviewed sample, and then reviewing those.

That may have been as simple as going through a few hundred personal emails that weren't part of the initial dump.

Re:Drone Snowden's ass already

[fahrbot-bot]

That you elected a lying bitch instead of a lying asshole?
That you elected someone who has no real accomplishments because the other guy has no real accomplishments?

Well. Some accomplishments. Electing someone smart enough to (apparently) get away with killing bunches of people and lots of other illegal stuff over 30 years vs. someone who managed to somehow bankrupt a casino -- even after getting loans from his father and not paying people for all the work they did. Someone smart vs. someone stupid. Someone inclusive vs. someone divisive. Hmm... :-)

Re:removing dupes is easy...

[ScentCone]

because the entire investigation was to check whether Clinton's use of a private server had in some way broken the law

No, the FBI was looking into whether SHE broke the law(s). And as Comey pointed out in July (and hasn't changed since), she demonstrably did things that would result in any other government employee facing punishment. This isn't about "the server," it's about the double standards. That she mishandled classified information is established. That she lied about it, repeatedly, is established. That she's being held to a different standard is established. Anyone else applying for a high-level, sensitive job in the government with her track record would never, ever be hired (presuming they were out of jail and able to apply in the first place).

Re:Stupid

[nine-times]

I don't get why people are having such a hard time grasping this.

They're not. They're having a hard time accepting it because they don't want it to be true.

Re:Corrupt vs wild tweets

[Frigga's Ring]

That you don't find the things Trump has done unethical is a bit disturbing. I hope, for both our sake, that whoever is elected president doesn't screw things up so much that they can't be repaired in four years when, hopefully, a more sane candidate will be on the opposing side.

Re:Drone Snowden's ass already

[NatasRevol]

Trump is way worse than a lying asshole.

1. He is a facist - populist us vs them mentality (muslims), denounces anything said about him as lies & blames others, whips up the armed/racist minority.
2. He hates the constitution. Wants to change 1A, 4A, and sure as hell the 19A.
3. He's a racist - see twitter war with Jon Stewart, see endoresments by David Duke & KKK, see his talk in MN yesterday
4. He's a power abuser - see every business dealing ever, see how he grabs women by the pussy because he can get away with it
5. He wants to take us straight to war - see his comments on why we can't 'bomb the shit out of the Middle East'.
6. He's an exploiter - uses & abuses every religious, ethnic, regional divide he can
7. He hates America - he detests freedom of speech - in others, he hates freedom of religion - in others, he tries to get people to stop Americans from voting - 'help watch out for a rigged election'

He is as bad as any other dictator in the world today. He has single handedly torn America apart to stroke his own ego.

You can hate Hillary all you want, I won't disagree with you, hell I'll probably mostly agree with you. But she is the lesser of two evils, and it's not re-fucking-motely close.

http://www.nydailynews.com/opi...

Re:Drone Snowden's ass already

[NatasRevol]

If congress stays GOP (likely), guess who will be 'holding him in check'?

The same idiots that let him take control of their party, and have a psychopath running for president.

So, why would that happen AFTER he gets into office, when it didn't happen before?

Re:Drone Snowden's ass already

[fahrbot-bot]

Wait. Who did she kill?

Seriously? If you believe the Republicans and those fostering conspiracy theories: Vince Foster, Seth Rich, everyone in the Benghazi consulate, etc... Just Google: who did hillary clinton kill Of course, there's *no* proof of anything - and if she *did* do all that *and* got away with it, then she's a serious bad-ass and wouldn't we actually want her as President to go up against Putin, etc... :-)

Conspiracy Theorists Won't Stop Accusing The Clintons Of Murder

Re:We know because we're DOING it!

[Xenographic]

So tell me, how are there 650 dupes of 33k emails? I've yet to see proof that these are duplicates.

We've found more than just "suspicious" stuff if you read /r/wikileaks.