A well designed private "cloud" is a couple orders of magnitude faster than anything you can provision from somebody like Amazon. Until Amazon gets around to installing 40 Gbit low latency switches and network adapters that is.
And even when Amazon does that, a large number of customers will be crippled by the lack of bandwidth between the Amazon cloud and their local operations - relative to what the most inexpensive Ethernet network can provide. Priced any high traffic multigigabit Internet connections lately?
The final problem is that if you need sustained computer power it is considerably cheaper to buy and maintain your own servers at a nearby colocation center than to purchase the same capacity from Amazon, at least at the economy of scale any medium size business is likely to require.
By the way, some databases support a feature called "asynchronous commit", which sounds like exactly what these folks needed. No disk wait on commit because you give up the durability guarantee. The other way to resolve that problem is to use solid state disks for your commit logs.
So, it turns out that most of the game logic was built off of complicated SQL stored procedures, triggers, logic, etc. Basically, they were using their hard drive as a processor.
For a game? That is positively insane. Most of the overhead of a well designed SQL database server is designed to preserve ACID properties for business transactions, on large databases that won't fit in memory. Without those requirements any well designed game algorithm should somewhere between a hundred and a thousand times faster.
A good optimizer could easily partition a traditional trigger into internal triggers that only ran when certain columns were updated, and in most cases maintenance would be much simpler.
Counter mode is clearly far superior to ECB mode. But if you are really worried, and want multi-core scalability you might break up the input into parallel streams and then run a something like AES in chaining mode on each one independently...
Absolutely. You want to get rid of any gratuitous headers, unless they are applied after the encryption process takes place, for the very reason you describe.
The other thing is that you really want a compression algorithm (or pre-pass) that is tailored to the type of data you are compressing. If you are compressing data in a known format (a file system image for example) you might have necessary header information (like tables of offsets known to be within a certain range) that a run of the mill compression algorithm is not going to compress very well.
So if you are maximally paranoid, you might write a pre-pass tailored to the format you are compressing that eliminates predictable aspects of the data (like zero bytes in certain places, or even a high probability of zero bits in certain places) as much as possible.
Well said. My only quibble is that in it is impossible (*) to make a "perfect" encryption algorithm, for the same reason that encrypted data in general isn't truly random. It is an information bearing signal, and that makes it vulnerable to some sort of attack. Theoretically speaking of course. Practically speaking, I think you are probably right.
The NSA is probably more qualified to answer that question. My point is that _in principle_ it is impossible to know when and whether a mathematical discovery is going to make (or has already made) attacking a particular encryption algorithm a practical possibility.
The reason why that is the case is that it is mathematically impossible to transform non-random data into maximally random data and back in a lossless manner. So whether it is thirty years or thirty thousand years, at some point a mathematician is liable to discover a practical means of attacking the algorithm using all the non-random data that any information bearing signal necessarily contains.
All finite encryption algorithms are _intrinsically_ flawed by their very nature, in that it is impossible for a encryption algorithm to do the impossible. An encryption algorithm that produces an output string strictly indistinguishable from random data is not just a physical impossibility, it is a logical impossibility.
Brute force is called "brute" for a reason. Realistic attacks generally use much more sophisticated techniques. Counting on the incapacity of mathematicians is a sucker's bet. Your only sure hope is if the value / sensitivity of your data decreases faster than the ability of cryptographers to analyze it.
There is no question is is computationally difficult, just not computationally impossible. The reason for that lies in computational complexity theory. You can get a basic summary of the theory here.
In summary, if the data being encrypted is compressible by practically any algorithm whatsoever, it has computational complexity less than its bit length, i.e. a smaller bit string can be used to recover the larger one. Likewise, the computational complexity of any encryption key is at most the length of the key in bits.
Suppose you are encrypting a 256K bit string that can be compressed by a factor of two by an _ideal_ compressor. And then you have a maximally random 1K bit key. The maximum Kolmogorov complexity of any finite deterministic function of those two inputs that is known to the attacker is 129 Kbits. Where the maximum computational complexity of a truly random string of the same length as the input is 256 Kbits.
The difference between those figures opens the door to a statistical attack, because the data is not _really_ random. It just looks that way, sort of like the output of a pseudo random number generator, which isn't really random at all. If you encrypt a string of zeroes with a significantly shorter key, the output of a pseudo random number generator is exactly what you will get, a pattern that is maximally vulnerable to attack.
The lesson to be learned here is if you want to minimize the risk of attacks from folks with far more computer power than you have available, compress it using the best available compression algorithm first. Then the computational complexity of the input string will approach the theoretical maximum for a string of that length, depending on how good the compressor is. Throughout I mean complexity relative to what the attacker knows (such as the encryption algorithm) of course.
No question our relationship with Saudi Arabia is an alliance of convenience. The big difference though is that Iran is a state sponsor of terrorism and Saudi Arabia isn't. If that ever changes...
AES encrypted data should be indistinguishable from random data, at least for a distinguisher running in bounded time
That is a mathematical impossibility. The computational complexity of any finite deterministic function of inputs of finite complexity is finite. That means that any necessary analysis can also be conducted in finite time. The only thing protecting AES is the lack of computational resources on the part of attackers. Does anyone really think that thirty years from now it won't be commonplace to break the encryption of data encrypted using the key lengths we use now?
This is categorically not true, unless the key is as long or longer than the data file (and never used again). There is indeed an attack vector against any encrypted data file if the key length is small by comparison. Statistical analysis plus the slightest idea of what type of data is being encrypted is more than adequate to mount a successful attack (given sufficient computational resources) unless the key is _much_ longer than what is typical today. The lack of computational resources is the only thing that keeps typical encrypted data secure.
Encrypted files have maximum entropy, just like absolutely random files
Not true, unless you have a randomly generated key that is as long as the file. You cannot apply any deterministic process to inputs of limited entropy and expect to get output entropy that is strictly speaking (computational complexity wise) higher than the sum of the entropy of the inputs.
Properly encrypted data is indistinguishable from random data.
That is the ideal, but hardly the reality. It is strictly _impossible_ for an encrypted version of a non-random data file to have as much real randomness as as a random data file, unless the key is as long as the file. Sure it can look pseudo random, much more random than the output of a pseudo random number generator (depending on the source file), but distinguishing it from real random data is just a matter of having sophisticated enough statistical tests. The larger the file, the easier it is to tell the difference.
How often does that happen? I must be in a low crime area, because I never had a card stolen, ever. If someone stole my wallet, replacing a twenty dollar item would be the last thing I would worry about. And of course in mass production these gadgets will probably be more like five dollars, not twenty.
...is that thanks to the lack of an IOMMU on consumer x86 computers, JavaScript exploits in the browser can now give you access to all the computer's memory, and along with it, ring 0.
You are claiming that the graphics drivers / host kernel give user processes the ability to read any location in the computers physical memory? Or is a separate vulnerability in the graphics driver required?
Regular phone networks have geographical addressing and routing (more or less). That means that a call within your metro area never leaves it. Where with IP typical backbones interconnect in only a handful of cities. I live in a metro area with about two million people. If I called my neighbor using VoIP the traffic is liable to traverse seven Western states. That is how bad it is.
If someone cut a few fiber lines the Internet in many states would be partitioned with no hope of recovery for days, because the backbones don't bother to interconnect anywhere in the state. VoIP over the public Internet for anything you expect to operate in an emergency is not something to be counted on.
If the FCC wanted to do something useful, they would require the major backbones / providers to fix that, in the name of national security. The side benefit would be that Internet latency for many peers would go way down, traffic would actually take something resembling a shortest route, and so on.
HP mostly competes in the x86 server market. Oracle/Sun mostly doesn't (and wants to less by the day). With the exception of high end IA-64 and SPARC machines, and perhaps a storage product here and there, there is very little direct competition at all.
It is as if one asked several years ago if Compaq and Sun were direct competitors. The question is enough to provoke laughter. The only stuff that is even close is the stuff that Compaq soon thereafter acquired from DEC, which is mostly on life support. From the original HP, that leaves IA-64/Itanium and HP-UX vs SPARC/Solaris as about the only significant area where Oracle/Sun and HP are direct competitors. Both platforms seem to be in serious danger of going out of existence. Dangerously close to irrelevant on all but the largest machines already.
One problem I have is that most of HP's PCs are generally constructed to some of the worst quality standards on the planet. I have a hard time thinking of any supplier where I could purchase a computer with flimsier, cheaper components than HP.
In my opinion, if HP wants to produce such (relatively) shoddy merchandise, it would be better served by adopting an entry level brand for people who are so price sensitive that they don't care about the trade offs, and reserving the HP name for somewhat more expensive items that are actually made to respectable quality standards.
And of course where HP goes, Dell usually follows. Although as of late, Dell's personal computers seem to be a step up from HP.
The higher end HP stuff seems to be pretty good. Not as good as it used to be, but certainly respectable. It is the "consumer" grade stuff that is worthless.
I like to blame all that on the merger with Compaq, which had a reputation for putting out really bad entry level hardware in the early 90s, but I don't know who there is really responsible for polluting a respectable brand with self destructing consumer equipment.
Don't you find it ironic that while everyone else you accuse is complaining about customer service issues and the like, you bring up a long list of factors that almost no one except spectators and speculators actually care about?
Trying to push Cisco out? Forced out their CEO? Strange reseller practices? Unless they are in danger of going out of business, most customers couldn't care less. Customer service, quality, reliability, longevity, device drivers, support, service information, replacement part availability. Those are things that _customers_ actually care about.
Slashdot Mirror
A well designed private "cloud" is a couple orders of magnitude faster than anything you can provision from somebody like Amazon. Until Amazon gets around to installing 40 Gbit low latency switches and network adapters that is.
And even when Amazon does that, a large number of customers will be crippled by the lack of bandwidth between the Amazon cloud and their local operations - relative to what the most inexpensive Ethernet network can provide. Priced any high traffic multigigabit Internet connections lately?
The final problem is that if you need sustained computer power it is considerably cheaper to buy and maintain your own servers at a nearby colocation center than to purchase the same capacity from Amazon, at least at the economy of scale any medium size business is likely to require.
By the way, some databases support a feature called "asynchronous commit", which sounds like exactly what these folks needed. No disk wait on commit because you give up the durability guarantee. The other way to resolve that problem is to use solid state disks for your commit logs.
So, it turns out that most of the game logic was built off of complicated SQL stored procedures, triggers, logic, etc. Basically, they were using their hard drive as a processor.
For a game? That is positively insane. Most of the overhead of a well designed SQL database server is designed to preserve ACID properties for business transactions, on large databases that won't fit in memory. Without those requirements any well designed game algorithm should somewhere between a hundred and a thousand times faster.
A good optimizer could easily partition a traditional trigger into internal triggers that only ran when certain columns were updated, and in most cases maintenance would be much simpler.
Counter mode is clearly far superior to ECB mode. But if you are really worried, and want multi-core scalability you might break up the input into parallel streams and then run a something like AES in chaining mode on each one independently...
Absolutely. You want to get rid of any gratuitous headers, unless they are applied after the encryption process takes place, for the very reason you describe.
The other thing is that you really want a compression algorithm (or pre-pass) that is tailored to the type of data you are compressing. If you are compressing data in a known format (a file system image for example) you might have necessary header information (like tables of offsets known to be within a certain range) that a run of the mill compression algorithm is not going to compress very well.
So if you are maximally paranoid, you might write a pre-pass tailored to the format you are compressing that eliminates predictable aspects of the data (like zero bytes in certain places, or even a high probability of zero bits in certain places) as much as possible.
Well said. My only quibble is that in it is impossible (*) to make a "perfect" encryption algorithm, for the same reason that encrypted data in general isn't truly random. It is an information bearing signal, and that makes it vulnerable to some sort of attack. Theoretically speaking of course. Practically speaking, I think you are probably right.
(*)Excepting a random one time pad of course.
The NSA is probably more qualified to answer that question. My point is that _in principle_ it is impossible to know when and whether a mathematical discovery is going to make (or has already made) attacking a particular encryption algorithm a practical possibility.
The reason why that is the case is that it is mathematically impossible to transform non-random data into maximally random data and back in a lossless manner. So whether it is thirty years or thirty thousand years, at some point a mathematician is liable to discover a practical means of attacking the algorithm using all the non-random data that any information bearing signal necessarily contains.
With a key length significantly smaller than the source text, of course.
Unless there are flaws in the algorithm...
All finite encryption algorithms are _intrinsically_ flawed by their very nature, in that it is impossible for a encryption algorithm to do the impossible. An encryption algorithm that produces an output string strictly indistinguishable from random data is not just a physical impossibility, it is a logical impossibility.
Brute force is called "brute" for a reason. Realistic attacks generally use much more sophisticated techniques. Counting on the incapacity of mathematicians is a sucker's bet. Your only sure hope is if the value / sensitivity of your data decreases faster than the ability of cryptographers to analyze it.
There is no question is is computationally difficult, just not computationally impossible. The reason for that lies in computational complexity theory. You can get a basic summary of the theory here.
In summary, if the data being encrypted is compressible by practically any algorithm whatsoever, it has computational complexity less than its bit length, i.e. a smaller bit string can be used to recover the larger one. Likewise, the computational complexity of any encryption key is at most the length of the key in bits.
Suppose you are encrypting a 256K bit string that can be compressed by a factor of two by an _ideal_ compressor. And then you have a maximally random 1K bit key. The maximum Kolmogorov complexity of any finite deterministic function of those two inputs that is known to the attacker is 129 Kbits. Where the maximum computational complexity of a truly random string of the same length as the input is 256 Kbits.
The difference between those figures opens the door to a statistical attack, because the data is not _really_ random. It just looks that way, sort of like the output of a pseudo random number generator, which isn't really random at all. If you encrypt a string of zeroes with a significantly shorter key, the output of a pseudo random number generator is exactly what you will get, a pattern that is maximally vulnerable to attack.
The lesson to be learned here is if you want to minimize the risk of attacks from folks with far more computer power than you have available, compress it using the best available compression algorithm first. Then the computational complexity of the input string will approach the theoretical maximum for a string of that length, depending on how good the compressor is. Throughout I mean complexity relative to what the attacker knows (such as the encryption algorithm) of course.
No question our relationship with Saudi Arabia is an alliance of convenience. The big difference though is that Iran is a state sponsor of terrorism and Saudi Arabia isn't. If that ever changes...
AES encrypted data should be indistinguishable from random data, at least for a distinguisher running in bounded time
That is a mathematical impossibility. The computational complexity of any finite deterministic function of inputs of finite complexity is finite. That means that any necessary analysis can also be conducted in finite time. The only thing protecting AES is the lack of computational resources on the part of attackers. Does anyone really think that thirty years from now it won't be commonplace to break the encryption of data encrypted using the key lengths we use now?
You cannot distinguish between the two.
This is categorically not true, unless the key is as long or longer than the data file (and never used again). There is indeed an attack vector against any encrypted data file if the key length is small by comparison. Statistical analysis plus the slightest idea of what type of data is being encrypted is more than adequate to mount a successful attack (given sufficient computational resources) unless the key is _much_ longer than what is typical today. The lack of computational resources is the only thing that keeps typical encrypted data secure.
Encrypted files have maximum entropy, just like absolutely random files
Not true, unless you have a randomly generated key that is as long as the file. You cannot apply any deterministic process to inputs of limited entropy and expect to get output entropy that is strictly speaking (computational complexity wise) higher than the sum of the entropy of the inputs.
Properly encrypted data is indistinguishable from random data.
That is the ideal, but hardly the reality. It is strictly _impossible_ for an encrypted version of a non-random data file to have as much real randomness as as a random data file, unless the key is as long as the file. Sure it can look pseudo random, much more random than the output of a pseudo random number generator (depending on the source file), but distinguishing it from real random data is just a matter of having sophisticated enough statistical tests. The larger the file, the easier it is to tell the difference.
When I get my credit card stolen,
How often does that happen? I must be in a low crime area, because I never had a card stolen, ever. If someone stole my wallet, replacing a twenty dollar item would be the last thing I would worry about. And of course in mass production these gadgets will probably be more like five dollars, not twenty.
...is that thanks to the lack of an IOMMU on consumer x86 computers, JavaScript exploits in the browser can now give you access to all the computer's memory, and along with it, ring 0.
You are claiming that the graphics drivers / host kernel give user processes the ability to read any location in the computers physical memory? Or is a separate vulnerability in the graphics driver required?
Regular phone networks have geographical addressing and routing (more or less). That means that a call within your metro area never leaves it. Where with IP typical backbones interconnect in only a handful of cities. I live in a metro area with about two million people. If I called my neighbor using VoIP the traffic is liable to traverse seven Western states. That is how bad it is.
If someone cut a few fiber lines the Internet in many states would be partitioned with no hope of recovery for days, because the backbones don't bother to interconnect anywhere in the state. VoIP over the public Internet for anything you expect to operate in an emergency is not something to be counted on.
If the FCC wanted to do something useful, they would require the major backbones / providers to fix that, in the name of national security. The side benefit would be that Internet latency for many peers would go way down, traffic would actually take something resembling a shortest route, and so on.
HP mostly competes in the x86 server market. Oracle/Sun mostly doesn't (and wants to less by the day). With the exception of high end IA-64 and SPARC machines, and perhaps a storage product here and there, there is very little direct competition at all.
It is as if one asked several years ago if Compaq and Sun were direct competitors. The question is enough to provoke laughter. The only stuff that is even close is the stuff that Compaq soon thereafter acquired from DEC, which is mostly on life support. From the original HP, that leaves IA-64/Itanium and HP-UX vs SPARC/Solaris as about the only significant area where Oracle/Sun and HP are direct competitors. Both platforms seem to be in serious danger of going out of existence. Dangerously close to irrelevant on all but the largest machines already.
Mark Hurd did not break any law and he only cheated on his expense reports
That is called fraud. It is more than enough to get people sent to jail in some cases.
One problem I have is that most of HP's PCs are generally constructed to some of the worst quality standards on the planet. I have a hard time thinking of any supplier where I could purchase a computer with flimsier, cheaper components than HP.
In my opinion, if HP wants to produce such (relatively) shoddy merchandise, it would be better served by adopting an entry level brand for people who are so price sensitive that they don't care about the trade offs, and reserving the HP name for somewhat more expensive items that are actually made to respectable quality standards.
And of course where HP goes, Dell usually follows. Although as of late, Dell's personal computers seem to be a step up from HP.
The higher end HP stuff seems to be pretty good. Not as good as it used to be, but certainly respectable. It is the "consumer" grade stuff that is worthless.
I like to blame all that on the merger with Compaq, which had a reputation for putting out really bad entry level hardware in the early 90s, but I don't know who there is really responsible for polluting a respectable brand with self destructing consumer equipment.
Don't you find it ironic that while everyone else you accuse is complaining about customer service issues and the like, you bring up a long list of factors that almost no one except spectators and speculators actually care about?
Trying to push Cisco out? Forced out their CEO? Strange reseller practices? Unless they are in danger of going out of business, most customers couldn't care less. Customer service, quality, reliability, longevity, device drivers, support, service information, replacement part availability. Those are things that _customers_ actually care about.