I order quite a bit from Amazon, including things that split shipments (ship different days or are a mix of Prime and non-Prime). The "Your order [...] has shipped!" e-mails list an amount charged for the items that actually shipped, and these are the same values that appear on my credit card. While the default "Your Orders" view on the website groups things by order (which is not the same as shipment or credit card charge), the "Invoice" link on each order breaks down the order correctly (by shipment, with separated charges). These also match up with credit card charges.
I know the Slashdot trope is that n is always too small in any study, regardless of the actual size of n.
The sample size you need to demonstrate statistical significance (or, conversely, the level of statistical significance achieved for a given sample size) depends on the behavior you're measuring. If you're measuring a small change in a rare occurrence, you need a very large sample population. If, on the other hand, your hypothesis is "black sheep exist" or "this vaccine reduces the mortality rate of a disease that has an untreated survival rate of 1 in 100,000", then a single occurrence (black sheep, surviving subject) is significant at n=1, and two occurrences out of even a tiny n is excellent.
No. "Specific" and "catch-all" are pretty different.
Dark energy, for example, is essentially the discrepancy between the observed expansion rate of the universe and the quantity of detectable matter in the universe.
Dark matter is not some astrophysics catch-all explanation. Dark matter and dark energy separately refer to a specific observed discrepancy for which we don't have an answer yet.
Shade, local terrain, building codes, subsidies, power company buyback policies and rates. There are also a lot of odd business arrangements that are localized that can dramatically reduce the solar capital cost.
I have no idea if they account for any of these factors, but there are certainly a lot more factors than weather-adjusted annual insolation.
The US has officially been proven to be an oligarchy as described here
You know you're on the Internet when a single study counts as "official proof".
Now you just need someone to reply asking for confirmation, then a person to reply that it is confirmed, since they saw that the same study does in fact exist. (Needless to say, no involved parties have read the study.)
Yes. That is made clear. Almost all of the article is about the NSA's capabilities. Then, at the end, some text, including the quoted part, about how this is important even if you don't mind the actions of the NSA.
"Even if you're not inclined to view the NSA as an adversary... America is hardly the only intelligence agency capable of subverting the global communications network.... While it's cheap to hold China out as some sort of boogeyman, it's significant that someday a large portion of the world's traffic will flow through networks controlled by governments that are, at least to some extent, hostile to the core values of Western democracies."
No they don't. The likelihood of collisions depends only on the effective number of bits in the hash. If you have a 32 bit hash, and more than 2^32 images, you are guaranteed to have collisions.
Yes, they do, because the shortest available cryptographic hash has a 128-bit output.
Let's say that the chance of two unrelated images matching is, say, one in million. Great. That sounds amazing - and it is, that's ridiculously optimistic for phash alone, but we can assume they have something better involving composite hashes.
For the reasons you outline, a hash with a one-in-a-million collision rate (one in 2^20) is worthless for this purpose and for many purposes. Maybe this is an accurate rate for phash. That's because it's a fuzzy hashing algorithm. Typically, all of these law enforcement applications use MD5 or SHA-1, which have collision rates around 2^128 to 2^160 (not including manufactured hash collisions).
Now feed into that a sizable database of child abuse imagery - say, ten thousand images. And a copy of the facebook photo library for one day, which is 350 million photos. Yes, that's facebooks claim, do not underestimate the number of compulsive photographers. That's 3,500,000,000,000 comparisons, and at your optimistic one-in-a-million error rate, 3,500,000 false positives to investigate every day.
It can be done, but it's going to need a bit more than just perceptual hash comparisons.
The numbers are, of course, much different when your hash has collision rate that's many, many orders of magnitude lower.
The interesting thing is that your hash mechanism really is untrustworthy if any two images in your total pool (Facebook photos + child abuse photos) have a collision. Since the Facebook photos dominate that set, you really can just look at the probability of a collision within the Facebook set. MD5 hashes are good up to about a trillion different items, which is much larger than even years of Facebook photo data.
IWF, incidentally, claims they will be publishing MD5, SHA-1, and PhotoDNA hash lists. I can't comment on the collision rate of PhotoDNA hashes.
As paranoid as is sounds, these days I think it is entirely plausible that a national security letter or somesuch was used to say "if you tell anybody about this, we will put you in a deep dark hole... whether it's for the rest of your life or marking the end of it is your choice".
NSLs are not magic. They are not for making arbitrary legal requests. Even the EFF will tell you that--as well as telling you that NSLs cannot possibly have anything to do with ProxyHam.
I've only been in Berlin (a big city, obviously), but the public transit there was incredibly convenient. Comparable to NYC--the parts serviced by subway. But the S-Bahn (surface small train) covers most of the major suburbs, which is as good as the outer subway coverage of NYC and better than, say, commuting by LIRR.
I currently live in a small city that has pretty good public buses--for a city of its size in America. I don't use it, because, as most Americans here will complain, it's painfully slow and inefficient given the realities of where one lives, works, and shops. It's unfortunate but it's true. In a big city that's designed appropriately, public transit is great: it's faster than getting through traffic and it's cheaper than parking. Outside of that, in the US, it's often just not economically possible to have good enough service to all the places where people need to go.
"And you're transmitting your key to your car" "Yeah!" "Aaaaand... constantly while you're walking around." "Uh.... well,... yeah..." "Whew. Glad mine doesn't inform anyone and everyone what key I use wherever I go. Someone bad might listen..."
Active keys transmit only when you press the button. Passive keys transmit only when a challenge is transmitted to them. That's why the latter only functions if you're fairly close to the vehicle.
So it is not constantly transmitting the key while you're walking around. It's transmitting the key to anything that can sufficiently imitate the key-request transmission of a car.
Most of these systems implement appropriate rolling-key or challenge-response protocols so that the transmissions are not easily replayable. There are certainly dysfunctional implementations, but most hacks against remote-entry systems attack weaker parts of the overall system than key transmission.
Of course, if you want to duplicate a physical key, all you need to do is get a high-resolution picture when the victim takes their keys out of their pocket.
Yes, that's what I was saying. Salting uses a non-secret nonce. You could set up some system with a secret nonce, but then it would be a different construction than "salt" (and hard to argue that it's better).
Having access to the salt does make it much easier to crack the password. In fact, it's basically necessary to crack the password. It is still considered non-secret, though.
That situation is partially the result of more widespread use of salt. It doesn't magically make bad password hard to crack (as you point out). But it used to be the case that, with rainbow tables, you could crack even moderately difficult passwords very quickly. It also adds a pretty substantial slowdown for large password breaches -- even though all the easy passwords will be cracked anyway, a factor of hundreds of thousands slowdown starts changing the "easily crackable" threshold.
Salting is nice, but when the attacker gets both the hash and the salt, they can attack specific users.
Of course they can. The entire purpose of salting is to make it so that the same password, hashed two different times, produces completely different hashes. This has two important consequences. First, it makes it basically impossible to precompute password hashes. That's a big deal compared to the "without salt" case, where rainbow tables make checking against precomputed hashes very easy. Second, if two users on a system have the same password, you can't tell without computation. Said another way, it means you need to crack passwords individually rather than in bulk. This isn't game-breaking, but it's significant when you have million-user breaches.
All of the typical ways of storing password hashes store the salt alongside it. It's expected that an attacker that obtains the hash will obtain the salt. It's within the design.
If you want the password hash separate from a piece of key password-validation data, at that point the extra piece of data is a secret and what you're basically making is a message authentication code. But, it's very difficult to argue that this is ever really more secure.
100k rounds of SHA256 is decent. The longer SHA2 variants are better, sure. More rounds is always better, of course. 100k is better than what most people use. But, if the decryption is always happening client-side (which it should), then ideally you can afford and should use many more rounds of SHA1. Maybe if they're using JavaScript, that limits how high they can jack the number of rounds up and still get reasonable performance on low-end devices.
I don't know that bcrypt is necessarily much better than what they're doing. It may be, but at a "details" level, not a "major benefit" level. Both bcrypt and PBKDF2 support many rounds and prevent precomputation, which are major features.
What would be better, if the devices they want to support can run it, is something like scrypt, which is resistant to hardware acceleration and thus much harder to crack in practice.
Slashdot Mirror
I order quite a bit from Amazon, including things that split shipments (ship different days or are a mix of Prime and non-Prime). The "Your order [...] has shipped!" e-mails list an amount charged for the items that actually shipped, and these are the same values that appear on my credit card. While the default "Your Orders" view on the website groups things by order (which is not the same as shipment or credit card charge), the "Invoice" link on each order breaks down the order correctly (by shipment, with separated charges). These also match up with credit card charges.
That why I always work with "N" in my studies - it's much bigger than "n".
Good call.
I know the Slashdot trope is that n is always too small in any study, regardless of the actual size of n.
The sample size you need to demonstrate statistical significance (or, conversely, the level of statistical significance achieved for a given sample size) depends on the behavior you're measuring. If you're measuring a small change in a rare occurrence, you need a very large sample population. If, on the other hand, your hypothesis is "black sheep exist" or "this vaccine reduces the mortality rate of a disease that has an untreated survival rate of 1 in 100,000", then a single occurrence (black sheep, surviving subject) is significant at n=1, and two occurrences out of even a tiny n is excellent.
No. "Specific" and "catch-all" are pretty different.
Dark energy, for example, is essentially the discrepancy between the observed expansion rate of the universe and the quantity of detectable matter in the universe.
astronomy it is dark matter or something similar
Dark matter is not some astrophysics catch-all explanation. Dark matter and dark energy separately refer to a specific observed discrepancy for which we don't have an answer yet.
Shade, local terrain, building codes, subsidies, power company buyback policies and rates. There are also a lot of odd business arrangements that are localized that can dramatically reduce the solar capital cost.
I have no idea if they account for any of these factors, but there are certainly a lot more factors than weather-adjusted annual insolation.
The US has officially been proven to be an oligarchy as described here
You know you're on the Internet when a single study counts as "official proof".
Now you just need someone to reply asking for confirmation, then a person to reply that it is confirmed, since they saw that the same study does in fact exist. (Needless to say, no involved parties have read the study.)
Yes. That is made clear. Almost all of the article is about the NSA's capabilities. Then, at the end, some text, including the quoted part, about how this is important even if you don't mind the actions of the NSA.
"Even if you're not inclined to view the NSA as an adversary ... America is hardly the only intelligence agency capable of subverting the global communications network. ... While it's cheap to hold China out as some sort of boogeyman, it's significant that someday a large portion of the world's traffic will flow through networks controlled by governments that are, at least to some extent, hostile to the core values of Western democracies."
No they don't. The likelihood of collisions depends only on the effective number of bits in the hash. If you have a 32 bit hash, and more than 2^32 images, you are guaranteed to have collisions.
Yes, they do, because the shortest available cryptographic hash has a 128-bit output.
I didn't say there weren't.
IWF, the organization named in the summary who is providing "hash lists" to Google, Facebook, and Twitter.
Its not a hash in the sense of MD5/SHA etc that hashes the file contents at the byte level.
It's MD5, SHA1, and PhotoDNA hashes.
The standard in most law enforcement forensic applications is MD5 / SHA1, despite the obvious limitations.
Sadly, it still is reasonably common to encounter byte-identical images that are on the relatively small "known-bad image" hash lists.
Well ... this is great if no two things can have the same hash.
With even the worse of the acceptable cryptographic hashes, it is essentially true that no two things can have the same hash*.
* Barring manufactured collisions, which are best avoided but may or may not be a problem depending on your application.
They almost exclusively use binary hashes (MD5, SHA-1).
Let's say that the chance of two unrelated images matching is, say, one in million. Great. That sounds amazing - and it is, that's ridiculously optimistic for phash alone, but we can assume they have something better involving composite hashes.
For the reasons you outline, a hash with a one-in-a-million collision rate (one in 2^20) is worthless for this purpose and for many purposes. Maybe this is an accurate rate for phash. That's because it's a fuzzy hashing algorithm. Typically, all of these law enforcement applications use MD5 or SHA-1, which have collision rates around 2^128 to 2^160 (not including manufactured hash collisions).
Now feed into that a sizable database of child abuse imagery - say, ten thousand images. And a copy of the facebook photo library for one day, which is 350 million photos. Yes, that's facebooks claim, do not underestimate the number of compulsive photographers. That's 3,500,000,000,000 comparisons, and at your optimistic one-in-a-million error rate, 3,500,000 false positives to investigate every day.
It can be done, but it's going to need a bit more than just perceptual hash comparisons.
The numbers are, of course, much different when your hash has collision rate that's many, many orders of magnitude lower.
The interesting thing is that your hash mechanism really is untrustworthy if any two images in your total pool (Facebook photos + child abuse photos) have a collision. Since the Facebook photos dominate that set, you really can just look at the probability of a collision within the Facebook set. MD5 hashes are good up to about a trillion different items, which is much larger than even years of Facebook photo data.
IWF, incidentally, claims they will be publishing MD5, SHA-1, and PhotoDNA hash lists. I can't comment on the collision rate of PhotoDNA hashes.
As paranoid as is sounds, these days I think it is entirely plausible that a national security letter or somesuch was used to say "if you tell anybody about this, we will put you in a deep dark hole ... whether it's for the rest of your life or marking the end of it is your choice".
NSLs are not magic. They are not for making arbitrary legal requests. Even the EFF will tell you that--as well as telling you that NSLs cannot possibly have anything to do with ProxyHam.
I've only been in Berlin (a big city, obviously), but the public transit there was incredibly convenient. Comparable to NYC--the parts serviced by subway. But the S-Bahn (surface small train) covers most of the major suburbs, which is as good as the outer subway coverage of NYC and better than, say, commuting by LIRR.
I currently live in a small city that has pretty good public buses--for a city of its size in America. I don't use it, because, as most Americans here will complain, it's painfully slow and inefficient given the realities of where one lives, works, and shops. It's unfortunate but it's true. In a big city that's designed appropriately, public transit is great: it's faster than getting through traffic and it's cheaper than parking. Outside of that, in the US, it's often just not economically possible to have good enough service to all the places where people need to go.
Do you think the US and UK treat journalists and human rights activists the same way they are treated in Egypt and Sudan?
"And you're transmitting your key to your car" ... yeah..."
"Yeah!"
"Aaaaand... constantly while you're walking around."
"Uh.... well,
"Whew. Glad mine doesn't inform anyone and everyone what key I use wherever I go. Someone bad might listen..."
Active keys transmit only when you press the button. Passive keys transmit only when a challenge is transmitted to them. That's why the latter only functions if you're fairly close to the vehicle.
So it is not constantly transmitting the key while you're walking around. It's transmitting the key to anything that can sufficiently imitate the key-request transmission of a car.
Most of these systems implement appropriate rolling-key or challenge-response protocols so that the transmissions are not easily replayable. There are certainly dysfunctional implementations, but most hacks against remote-entry systems attack weaker parts of the overall system than key transmission.
Of course, if you want to duplicate a physical key, all you need to do is get a high-resolution picture when the victim takes their keys out of their pocket.
Yes, that's what I was saying. Salting uses a non-secret nonce. You could set up some system with a secret nonce, but then it would be a different construction than "salt" (and hard to argue that it's better).
Having access to the salt does make it much easier to crack the password. In fact, it's basically necessary to crack the password. It is still considered non-secret, though.
Only if encrypted used data really wasn't exfiltrated.
That situation is partially the result of more widespread use of salt. It doesn't magically make bad password hard to crack (as you point out). But it used to be the case that, with rainbow tables, you could crack even moderately difficult passwords very quickly. It also adds a pretty substantial slowdown for large password breaches -- even though all the easy passwords will be cracked anyway, a factor of hundreds of thousands slowdown starts changing the "easily crackable" threshold.
Salting is nice, but when the attacker gets both the hash and the salt, they can attack specific users.
Of course they can. The entire purpose of salting is to make it so that the same password, hashed two different times, produces completely different hashes. This has two important consequences. First, it makes it basically impossible to precompute password hashes. That's a big deal compared to the "without salt" case, where rainbow tables make checking against precomputed hashes very easy. Second, if two users on a system have the same password, you can't tell without computation. Said another way, it means you need to crack passwords individually rather than in bulk. This isn't game-breaking, but it's significant when you have million-user breaches.
All of the typical ways of storing password hashes store the salt alongside it. It's expected that an attacker that obtains the hash will obtain the salt. It's within the design.
If you want the password hash separate from a piece of key password-validation data, at that point the extra piece of data is a secret and what you're basically making is a message authentication code. But, it's very difficult to argue that this is ever really more secure.
Still, the 100k rounds of SHA256 seem decent.
Would bcrypt be any better than PBKDF2 here?
100k rounds of SHA256 is decent. The longer SHA2 variants are better, sure. More rounds is always better, of course. 100k is better than what most people use. But, if the decryption is always happening client-side (which it should), then ideally you can afford and should use many more rounds of SHA1. Maybe if they're using JavaScript, that limits how high they can jack the number of rounds up and still get reasonable performance on low-end devices.
I don't know that bcrypt is necessarily much better than what they're doing. It may be, but at a "details" level, not a "major benefit" level. Both bcrypt and PBKDF2 support many rounds and prevent precomputation, which are major features.
What would be better, if the devices they want to support can run it, is something like scrypt, which is resistant to hardware acceleration and thus much harder to crack in practice.
Few people can memorize a large collection of high-entropy passwords.
Yes, I know there are strategies for getting away with memorizing fewer. They're not necessarily good ideas.
The radiation detectors at border crossings will trigger on cat litter.