Vista computers connected to the same link auto-configure themselves to use 6to4 and then advertise over ICMP that they are willing to route traffic through their 6to4 net. However, it turns out that they just drop the traffic!
That is an interesting case. I wish I could point out what you should be doing differently, but it isn't obvious to me. When you have a public IPv4 address, you could run your own 6to4 gateway. But of course if you have a network with public IPv4 addresses to multiple machines, you usually don't want each and every one of them to run a 6to4 gateway. It would be more appropriate to have one machine being the 6to4 gateway, and your machine seems to be trying to find it, and fail at that.
So that leaves us with two questions. Why are those Vista machines behaving the way they do? And what can you do to make your own laptop work under those circumstances? You could get your laptop to ignore advertisements of 6to4 routers if it already has a public IPv4 address, and then just setup your own 6to4 gateway in that case (but don't announce it on the network unless you intend to forward packets in both directions). Doing that should work as long as the network you are on does not filter away the 6to4 packets. Possibly that is what is going wrong for the Vista machines. It is plausible that they are actually working correctly, and the network is at fault. It would be worth setting up your own 6to4 gateway just to find out if that is the reason.
The other question to ask is when there is a bunch of gateways all announcing an 6to4 segment, how do you find out if there is one among them, that works correctly. I don't have any better suggestion than just trying to send a handcomputerful of pings through each of them to servers known to respond.
What do you do when all you are provided by the network is an RFC1918 address? Are you using a tunnel broker? Perhaps you could have a white list of good 6to4 gateways and if you receive announcements from any 6to4 gateway not on your list, just pretend it does not exist, and do whatever you would do in that case.
and for an executable one would expected it to be "run."
Maybe it would be ok if you had to go through some additional steps the first time you run an executable that wasn't installed on the system. A dialog that asks you if you really want to run this program would be a step in the right direction. (But we know that such a solution is too convenient, many users really need something more inconvenient). You could open a window explaining that to run this program you have to right click on the icon and choose to mark it executable.
There are different ways to keep track of what files are ok to be executed. You could have an execute bit on the file. You could have a list of accepted file paths stored somewhere, or a list of accepted file hashes.
this, of course, assumes that your operating system has an execute bit; if it doesn't, that's a bigger problem
On Windows, the extension is the execute bit. At least that is more or less the case. I don't know at what layer that is implemented. It is a convention inherited from DOS. In DOS the semantics of the execute bit (extension) was implemented by the shell rather than the kernel, might still be the case in Windows. Might not be the best idea, it is just the kind of heritage that is hard to get rid of.
Using the same action (the double-click) both to "run" and to "open" -- which are two very different actions -- is probably responsible for the vast majority of user-propagated malware today.
Good point. I hadn't thought about it that way. But thinking about it a bit more I realize that when working from the command line, those two really are different. The main reason you don't unintentionally run an executable from the command line is that there is a difference between typing less virus.txt and./virus.txt.
have a cron script run that takes a snapshot of your filesystem, runs e2fsck on the snapshot
Can you get away with doing this kind of thing without LVM? LVM does add another layer where things can go wrong. I know it is a very simple layer compared to the file system, but still...
I have been playing with the idea of simulating the snapshot feature in a user mode program that reads from the block device and presents another block device through the nbd driver. Then run fsck on that nbd device. Has the advantage that it doesn't depend on any snapshot feature to exist inside your normal fs stack. The obvious drawback of my approach is, that it does not give an atomic snapshot, which could give false positives in the fsck check, but maybe that can be fixed.
The original reason I started thinking about this was not to do file system checks, but rather to have the user mode program backup all the metadata while the fsck is running. A backup of metadata is no substitute for a real backup, but there are people who think it is impractical to backup all their data, maybe they could be convinced to at least backup the metadata.
One idea I had to dealing with the change of the underlying file system as the check is running was to reread the accessed sectors to see which have changed and rerun the check a few times. After the first run you have a pretty good idea of which sectors the check needs to read. And after two runs you have a pretty good idea of which sectors change frequently. Then do two reads of all the sectors one read starting from the least frequently changed ending with the most frequently changed, and then another read in opposite order to verify that no sector had changed in the meantime. It is only metadata, so there is a chance to keep a lot of it in cache.
Of course this approach is a lot more complicated than an LVM snapshot. But the complicated code would be in user mode, far away from your file system stack.
Last time I checked FAT had terrible performance if you have a lot of files in a directory, performance was even worse if you tried to do random access to large files, and files larger than 4GB were not supported at all, neither were basic unix semantics like file permissions. It also doesn't have journaling. Given all those lacks of features it is amazing how much code it is. There are much simpler file systems that have most of those features and perform better than FAT. I'd rather restrict the number of operating systems my disks can be used with than live with such a bad file system. After all my primary operating system seems to support more than half the file systems that exists today.
you should be using UDF. Vista supports it, OS X supports it, Linux supports it
Maybe a few years down the line that will be an option. Currently the windows systems I would be most interested in exchanging data with are running windows xp. You won't find me pushing for adoptation of new microsoft systems. Maybe I should have made it clear in my previous comment that I was mostly talking about windows xp, and that I only had minor interest in windows 98 and windows vista. But if Linux and Mac OS X both support it out of the box, that would actually be sufficient argument for me to at least try it.
It is rarely an issue to me, but once in a while it is convenient to be able to plug an USB disk on a machine with Windows or Mac OS X. What portable file systems are there that will cover those cases? Last I did some looks around a few years back I ended up concluding that the best option for a file system supported on both Linux and Windows was ext2 (with third party drivers for Windows). The only other file system supported on both was FAT, which have several drawbacks.
Moving forward, what file system will be the most portable? Are we going to be stuck with ext2 and FAT for file systems that we need to access across multiple operating systems, or is there going to be some journaled file system with support for large disks and basic unix semantics?
If the data is stored redundantly, then a single chip failing does not render your data inaccessible. You would be able to go on without even noticing (which in some sense is bad, because you wouldn't replace it until it is too late). If data is not stored redundantly, then you cannot replace a chip to get your data back. It may be possible to replace a chip to make the card work again, but you'd have to reformat and start from scratch.
I guess the data is not stored redundantly. Making it redundant would either double the price per GB or quadruple the price per GB/s. And even if the data is stored redundant on the card, you could still lose the data for other reasons. The question really is, how frequently do the chips fail. You don't want to use RAID as a backup strategy. Besides, if you want RAID, you don't want it to be implemented by this card, since if the card fails you are in trouble anyway. You could do RAID-1 over two cards of this type if you really want to, but even with that, you still need a backup strategy.
or are they actually pushing some new flavor of interconnect that will require BIOS tweaks and/or special drivers?
BIOS tweaks shouldn't be necessary. ATA controller cards that you plugged in a PCI slot came with a ROM chip on the card containing a driver that would allow the BIOS to use the disk. You could even work around bugs in the BIOS own driver for the onboard controller that way. I'm sure this new card will come with a driver on the board that will allow the BIOS to boot from it. However such a driver is only useful for booting and for running DOS era operating systems. Any modern operating system will need its own driver for such a card, which means your question is still relevant. I don't know the answer though. The card could have been made compatible with existing ATA, SCSI, or SATA drivers. But it's a new kind of device, and possibly the most efficient way to use it is different from protocols designed with disks in mind.
Actually with SSD the performance should be even better. Sequential reads on RAID-5 with n+1 harddisks will give you n times the speed of a single disk. The reason you don't get a factor of n+1 is that you have to skip the parity blocks on all the disks, and for skipping this few sectors the cost of a seek is going to be the same as just reading them. But with SSD you don't have the cost of seeks, so you should be able to read just the sectors you need and get n+1 times the speed of a single disk.
Writes to RAID-5 are slow for a completely different reason, and using SSD does not eliminate that cost. Random access writes require two physical writes for each logical write, and you also have to do two reads, unless the data is already in cache. So a five disk RAID-5 will not be significantly faster than writing to a single disk. Using SSD of course speeds up random access, no matter if you use RAID or not. Sequential writes to RAID-5 can be quite fast though since you don't have to write parity multiple times, and if there is sufficient data in the write queue, you don't have to read from disks either, since the data is going to be overwritten anyway.
A RAID system that actually takes into account that the underlying storage is SSD could do things in a more efficient and/or more reliable way.
I have been wondering if it is possible to catch this with a local http proxy. If you run an http proxy on your own machine, and let all the https traffic go through that, then that proxy would be between your client and any man in the middle. Is it possible to inspect the https traffic and find out early enough, if the certificate is valid, and for the correct domain? (Asking because I don't know https well enough to say for sure myself). I was hoping that could also get rid of the annoying certificate warnings I always get when connecting to public access points, since they tend to hijack all traffic, including https, until you are logged in.
Sometimes it randomly wakes up. If I make it hibernate, when I reawake it chances are my trackpad will not work or I will not be able to connect to the wireless network again.
Do you think the randomly waking up is caused by the operating system? When the machine is at sleep, the operating system is not running, so waking up would have to be triggered by the hardware. Do you think your Mac have a hardware setting that the software can set to make it wake up at random times? That would be a pretty odd feature to implement. Well, Macs do wake up when they shouldn't. For example a Mac usually wakes up when you unplug a USB device, and that is when using Mac OS X. And the implementation of going to sleep and waking up have a load of race conditions. I'll give a few examples, which I experience frequently:
It takes a while for the system to go to sleep, and during that time you can actually make the system go to sleep. The result of that is, that when the machine is woken up from sleep, the first thing it does is to continue where it was, which was going to sleep.
When woken up it will show a password prompt where you must type your password. If it times out, it goes back to sleep. That's of course fine, but the timeout counter starts before it even shows the password prompt. Sometimes showing the password prompt is so slow, that it times out and go back to sleep before the prompt even appears.
If there is a wireless network around when the machine is woken up, it will open a password window behind the screen blanker, and it will get focus. The end result is, that as you are typing your password, some of the keypresses end up in the wrong password window.
When installing some security update, the keyboard shortcut for putting the machine to sleep stopped working.
So before blaming Linux, stop and think about all the problems Mac OS X give on a Mac. I'm not saying Linux is perfect either, after all I have not yet reinstalled the Mac with Linux.
MS could start writing drivers for those hardware vendors that provide specifications and let others write their own drivers.
It is not in Microsoft's interest to give hardware vendors incentive to provide specifications to the public. Getting vendors to provide specifications to just Microsoft would be of some value to Microsoft, but if Microsoft can get the vendors to just write the drives, then it costs less money. Of course the optimal for Microsoft would be, if the hardware vendors would give Microsoft the driver sources, and Microsoft can then do bugfixes and updates themselves.
What would happen if hardware vendors actually provided specs to the public? Well for one the specs would obviously be used to support that hardware in Linux and BSD. Better drivers would get more people to use those operating systems, something which is not in Microsoft's interest.
Are those the same people who came up with the...... idea that a bottle of water could be used to perform a terror attack? (If you open a bottle of water onboard an airplane, the oxygen and hydrogen is going to react and cause a huge explosion, or something).
Of course, whether the proof of Distributed.net is correct depends on how strongly they can prove their program actually covered all possibilities.
Interesting question. What if somebody were cheating? Maybe somebody wanted to give the impression that they did a large amount of computation, and modified the code to skip large parts of the solution space and report back no solution was found.
the TV has to wait for a reference frame before it can begin to decode video.
Even that I can imagine being solved if you have sufficient processing power in the TV. How often is a key frame sent? Every five seconds? If the TV would figure out which 10 channels you are most likely to zap to next and store the last five seconds of compressed video for each of them, switching channels would just boil down to how quickly you could decode those five seconds of video.
Look at what happened with teletext. With early televisions supporting it, you would have to wait 10s of seconds for a page to show up. Today they show up instantly. After all it would only take a few MB of memory to store every page as it was sent over the air and keep it just in case you wanted to see it. A few MB of memory was a lot when teletext was invented, today it is nothing. Buffering MPEG streams requires a few orders of magnitude more memory, but other than that it is pretty much the same.
In the case of DOS version numbers were actually used by the software for all kinds of compatibility checks. So vendors would basically be forced to make their software report the version number of the MS DOS version they were compatible with. If something was sold as DR DOS 5.6 and told applications that it was a 6.22, then consumers would have been confused. I can see why they would just follow whatever version numbers Microsoft happened to decide on.
BTW. As I recall it, it was not DOS 3.3 that was used for a long time but rather DOS 3.30. I think DOS 3.30 was the one version that was used for the longest period of time. And DOS 6.22 was the last version in widespread use. As for DOS 7.x I don't think I have ever seen anybody use the full system. A lot of people have used the stripped down version (without knowing) to run Windows 9x on top of it.
Exactly my thought. I think giving back addresses might actually be bad for the internet as a whole. Giving back addresses means more fragmentation in allocations, which means more headaches for routing. Giving back addresses also means, that by the time we really have to convert, there will be more systems to convert, hence the cost will be higher.
You clearly have some good ideas there. To protect against certain active attacks against encrypted devices, you would actually need a hash tree and a signature or MAC of the root of the tree. I know that ZFS already has some of it. And I agree that some of the integrity it provides is useful for the raid layer as well. And yes, journaling is one way to handle the atomicity issues. It applies to file systems as well as an encryption that does use the extra space. I am not entirely sure what the right kind of layering is. On one hand I feel that integrating all the layers with encryption, raid, journaling, and file system, could end up being a bad spaghetti. On the other hand. I can see what advantages it gives. As for the on disk layout, I don't think an area at the start of the disk for IVs and MACs is the way to go. I think you would get better performance by putting that data in single sectors spread across the disk. One such sector for every n physical sectors on disk, where it would be related to the sectors just before or after.
How much random data does it need? An amount equal to what's being written?
No, it doesn't need nearly that much. For each sector written it needs 16 bytes or randomness from the system random number generator as well as 16 bytes from its own PRNG. So that amounts to just 1/16th of the size of the data. For reading it need to generate the same 16 bytes using its own PRNG before it can decrypt the data. I am pretty sure it is not the PRNG that is going to be the bottleneck but rather the AES encryptions. (If you have hardware to do AES but not to do the random numbers, then it might be different).
As for your suggestions on how to fix the nonatomicity, I don't know the solutions you suggest well enough to say whether they will work. But they sound like plausible solutions.
The only real solution for the bitmap problem is modification of the file system to contain secure IVs for each block on the disk.
Since the encryption is done at the block device layer, it shouldn't be part of the file system but rather a part of the encryption layer. Except from that detail, I agree with you. However doing it is tricky. When you take a logical sector with 512 bytes of data and put it together with a small IV, it no longer fits in a 512 bytes sector. So you have to come up with a layout on disk that is efficient. And then you have the problem of updating it. Because you cannot write two sectors to disk atomically (with some disks you can't even write one). So you end up having the potential of data loss if an update does not complete because of a power failure. And the data loss is not guaranteed to affect only the logical sector you were writing at the time. I am still thinking about the problem. A raid layer have some similar problems, and I have been considering whether merging those two layers would help solving this problem. A can see a few other advantages you could get from merging a raid layer with an encryption layer.
Reusing IVs has *always* been shown to be a bad idea. It's broken more thought-to-be-secure schemes than I can count.
Indeed. And the damage depends on the kind of encryption used. CBC is actually dealing pretty well with reused IVs compared to other encryptions. RC4 and counter mode are two examples where reusing IVs is so bad that you might as well not be encrypting at all.
As an aside, once I actually found two files from a Slackware distribution that were detectable inside a truecrypt volume.
Interesting. Though the patterns are simple, I wouldn't have thought they would show up unless they were carefully constructed.
One problem truecrypt had back then was that they were using 512 byte CBC blocks, while most file systems used larger blocks, so it was trivial to insert a file that would guarantee that the resulting ciphertext would be vulnerable to detection by starting at an even sector boundary and having just simple bit differences.
That's the vulnerability I was referring to. And this is what is demonstrated by the file I linked to. It requires just one or two bits of difference between neighbor sectors depending on which parameters you used to set up the volume. My file contains just a lot of variants of that to cover different byte orderings so as to demonstrate the weakness not just in truecrypt, but also in some early Linux loopback encryptions.
The fact that a file system block consists of multiple disk sectors just makes the exploit slightly more reliable. Even if the sizes were the same, the file system would still try to put consecutive blocks from your file right next to each other on disk. (If it didn't do that, performance would go down by orders of magnitude).
Shouldn't it be relatively easy to replace the PRNG?
Depends on whether you are worried about breaking compatibility. There is nothing you can do for existing encrypted volumes. If you want to improve security for your existing volumes, a complete reencryption of the volume is needed. If you want to protect new volumes while remaining compatible with the existing implementation, a small change to the way the master key is generated would help. There is a 256 byte lookup table, in which having two identical bytes is a weakness. The table is generated randomly, which means master keys vary in quality. The best master keys are those were all 256 bytes are different, but the chance of that happening at random is negligible. If the key was instead generated by taking an array containing all the values from 0 to 255 and permuting those randomly, you would always get keys that are resistant to the known attack.
If you don't need compatibility with the existing code, you can do even better. The PRNG makes use of MD5. Before anybody starts talking about MD5 being broken, keep in mind that the known attacks against MD5 do not apply to the way GBDE use it. The input that GBDE passes to MD5 is 24 bytes, which have 128 bits of entropy (with the best keys). Obviously the output will have no more than 128 bits of entropy, but it could have less. Though the input is just 24 bytes, MD5 is going to add a length field and pad the result to a multiple of 64 bytes. So you could actually double or triple the key size and pay no extra cost in performance on the MD5 operation.
Since three byte quantities are difficult to work with, I'd only suggest to change the lookup table from having 256 8-bit values to having 256 16-bit values. The risk of having two identical values if you pick them randomly, is reduced significantly. But still generating it in a way that guarantees it is still better. This would produce a 40 byte input to MD5. If you wanted to make use of the full MD5 block size, you could append another random key to the 40 bytes such that you would make the input as large as what would fit in a single MD5 block. Doing all of this would increase the key size from 272 bytes (16+256) to 584 bytes (56+16+512), and would not spend any additional time on the cryptographic operations in the critical paths.
You could also ditch the PRNG altogether and use a standard PRNG. But that would mean a significant performance hit, so you'd have to reduce the size of the master key. So I am not sure that would really make it any stronger than fixing the known vulnerabilities in the current PRNG.
As for the potential data loss goes, I suppose you could live with it and just make sure to have a good backup strategy. When doing that of course you have to keep in mind that you shouldn't make backups by copying the encrypted container. The correct way would be to copy the files from the encrypted container to a different encrypted container. Or you could copy the files to an encrypted archive that does not support random access, like tar + gpg. Or you could back up your container by creating an encrypted copy of the container. If you use asymmetric keys, you could create a gpg encrypted copy of the encrypted container even while it is not mounted.
Problem is that the only way to detect that corruption has happened is by trying to open the files with applications understanding the format, and see them barf. However fixing the potential corruption is tricky. If I find a good solution, I am probably going to write my own storage encryption layer some day.
Slashdot Mirror
That is an interesting case. I wish I could point out what you should be doing differently, but it isn't obvious to me. When you have a public IPv4 address, you could run your own 6to4 gateway. But of course if you have a network with public IPv4 addresses to multiple machines, you usually don't want each and every one of them to run a 6to4 gateway. It would be more appropriate to have one machine being the 6to4 gateway, and your machine seems to be trying to find it, and fail at that.
So that leaves us with two questions. Why are those Vista machines behaving the way they do? And what can you do to make your own laptop work under those circumstances? You could get your laptop to ignore advertisements of 6to4 routers if it already has a public IPv4 address, and then just setup your own 6to4 gateway in that case (but don't announce it on the network unless you intend to forward packets in both directions). Doing that should work as long as the network you are on does not filter away the 6to4 packets. Possibly that is what is going wrong for the Vista machines. It is plausible that they are actually working correctly, and the network is at fault. It would be worth setting up your own 6to4 gateway just to find out if that is the reason.
The other question to ask is when there is a bunch of gateways all announcing an 6to4 segment, how do you find out if there is one among them, that works correctly. I don't have any better suggestion than just trying to send a handcomputerful of pings through each of them to servers known to respond.
What do you do when all you are provided by the network is an RFC1918 address? Are you using a tunnel broker? Perhaps you could have a white list of good 6to4 gateways and if you receive announcements from any 6to4 gateway not on your list, just pretend it does not exist, and do whatever you would do in that case.
Maybe it would be ok if you had to go through some additional steps the first time you run an executable that wasn't installed on the system. A dialog that asks you if you really want to run this program would be a step in the right direction. (But we know that such a solution is too convenient, many users really need something more inconvenient). You could open a window explaining that to run this program you have to right click on the icon and choose to mark it executable.
There are different ways to keep track of what files are ok to be executed. You could have an execute bit on the file. You could have a list of accepted file paths stored somewhere, or a list of accepted file hashes.
On Windows, the extension is the execute bit. At least that is more or less the case. I don't know at what layer that is implemented. It is a convention inherited from DOS. In DOS the semantics of the execute bit (extension) was implemented by the shell rather than the kernel, might still be the case in Windows. Might not be the best idea, it is just the kind of heritage that is hard to get rid of.
Good point. I hadn't thought about it that way. But thinking about it a bit more I realize that when working from the command line, those two really are different. The main reason you don't unintentionally run an executable from the command line is that there is a difference between typing less virus.txt and ./virus.txt.
Can you get away with doing this kind of thing without LVM? LVM does add another layer where things can go wrong. I know it is a very simple layer compared to the file system, but still...
I have been playing with the idea of simulating the snapshot feature in a user mode program that reads from the block device and presents another block device through the nbd driver. Then run fsck on that nbd device. Has the advantage that it doesn't depend on any snapshot feature to exist inside your normal fs stack. The obvious drawback of my approach is, that it does not give an atomic snapshot, which could give false positives in the fsck check, but maybe that can be fixed.
The original reason I started thinking about this was not to do file system checks, but rather to have the user mode program backup all the metadata while the fsck is running. A backup of metadata is no substitute for a real backup, but there are people who think it is impractical to backup all their data, maybe they could be convinced to at least backup the metadata.
One idea I had to dealing with the change of the underlying file system as the check is running was to reread the accessed sectors to see which have changed and rerun the check a few times. After the first run you have a pretty good idea of which sectors the check needs to read. And after two runs you have a pretty good idea of which sectors change frequently. Then do two reads of all the sectors one read starting from the least frequently changed ending with the most frequently changed, and then another read in opposite order to verify that no sector had changed in the meantime. It is only metadata, so there is a chance to keep a lot of it in cache.
Of course this approach is a lot more complicated than an LVM snapshot. But the complicated code would be in user mode, far away from your file system stack.
Last time I checked FAT had terrible performance if you have a lot of files in a directory, performance was even worse if you tried to do random access to large files, and files larger than 4GB were not supported at all, neither were basic unix semantics like file permissions. It also doesn't have journaling. Given all those lacks of features it is amazing how much code it is. There are much simpler file systems that have most of those features and perform better than FAT. I'd rather restrict the number of operating systems my disks can be used with than live with such a bad file system. After all my primary operating system seems to support more than half the file systems that exists today.
Maybe a few years down the line that will be an option. Currently the windows systems I would be most interested in exchanging data with are running windows xp. You won't find me pushing for adoptation of new microsoft systems. Maybe I should have made it clear in my previous comment that I was mostly talking about windows xp, and that I only had minor interest in windows 98 and windows vista. But if Linux and Mac OS X both support it out of the box, that would actually be sufficient argument for me to at least try it.
It is rarely an issue to me, but once in a while it is convenient to be able to plug an USB disk on a machine with Windows or Mac OS X. What portable file systems are there that will cover those cases? Last I did some looks around a few years back I ended up concluding that the best option for a file system supported on both Linux and Windows was ext2 (with third party drivers for Windows). The only other file system supported on both was FAT, which have several drawbacks.
Moving forward, what file system will be the most portable? Are we going to be stuck with ext2 and FAT for file systems that we need to access across multiple operating systems, or is there going to be some journaled file system with support for large disks and basic unix semantics?
If the data is stored redundantly, then a single chip failing does not render your data inaccessible. You would be able to go on without even noticing (which in some sense is bad, because you wouldn't replace it until it is too late). If data is not stored redundantly, then you cannot replace a chip to get your data back. It may be possible to replace a chip to make the card work again, but you'd have to reformat and start from scratch.
I guess the data is not stored redundantly. Making it redundant would either double the price per GB or quadruple the price per GB/s. And even if the data is stored redundant on the card, you could still lose the data for other reasons. The question really is, how frequently do the chips fail. You don't want to use RAID as a backup strategy. Besides, if you want RAID, you don't want it to be implemented by this card, since if the card fails you are in trouble anyway. You could do RAID-1 over two cards of this type if you really want to, but even with that, you still need a backup strategy.
BIOS tweaks shouldn't be necessary. ATA controller cards that you plugged in a PCI slot came with a ROM chip on the card containing a driver that would allow the BIOS to use the disk. You could even work around bugs in the BIOS own driver for the onboard controller that way. I'm sure this new card will come with a driver on the board that will allow the BIOS to boot from it. However such a driver is only useful for booting and for running DOS era operating systems. Any modern operating system will need its own driver for such a card, which means your question is still relevant. I don't know the answer though. The card could have been made compatible with existing ATA, SCSI, or SATA drivers. But it's a new kind of device, and possibly the most efficient way to use it is different from protocols designed with disks in mind.
Actually with SSD the performance should be even better. Sequential reads on RAID-5 with n+1 harddisks will give you n times the speed of a single disk. The reason you don't get a factor of n+1 is that you have to skip the parity blocks on all the disks, and for skipping this few sectors the cost of a seek is going to be the same as just reading them. But with SSD you don't have the cost of seeks, so you should be able to read just the sectors you need and get n+1 times the speed of a single disk.
Writes to RAID-5 are slow for a completely different reason, and using SSD does not eliminate that cost. Random access writes require two physical writes for each logical write, and you also have to do two reads, unless the data is already in cache. So a five disk RAID-5 will not be significantly faster than writing to a single disk. Using SSD of course speeds up random access, no matter if you use RAID or not. Sequential writes to RAID-5 can be quite fast though since you don't have to write parity multiple times, and if there is sufficient data in the write queue, you don't have to read from disks either, since the data is going to be overwritten anyway.
A RAID system that actually takes into account that the underlying storage is SSD could do things in a more efficient and/or more reliable way.
I have been wondering if it is possible to catch this with a local http proxy. If you run an http proxy on your own machine, and let all the https traffic go through that, then that proxy would be between your client and any man in the middle. Is it possible to inspect the https traffic and find out early enough, if the certificate is valid, and for the correct domain? (Asking because I don't know https well enough to say for sure myself). I was hoping that could also get rid of the annoying certificate warnings I always get when connecting to public access points, since they tend to hijack all traffic, including https, until you are logged in.
Do you think the randomly waking up is caused by the operating system? When the machine is at sleep, the operating system is not running, so waking up would have to be triggered by the hardware. Do you think your Mac have a hardware setting that the software can set to make it wake up at random times? That would be a pretty odd feature to implement. Well, Macs do wake up when they shouldn't. For example a Mac usually wakes up when you unplug a USB device, and that is when using Mac OS X. And the implementation of going to sleep and waking up have a load of race conditions. I'll give a few examples, which I experience frequently:
So before blaming Linux, stop and think about all the problems Mac OS X give on a Mac. I'm not saying Linux is perfect either, after all I have not yet reinstalled the Mac with Linux.
My experience (based on a fairly small sample of just four models) is, that Olympus works with standard USB storage drivers, and Canon does not work.
It is not in Microsoft's interest to give hardware vendors incentive to provide specifications to the public. Getting vendors to provide specifications to just Microsoft would be of some value to Microsoft, but if Microsoft can get the vendors to just write the drives, then it costs less money. Of course the optimal for Microsoft would be, if the hardware vendors would give Microsoft the driver sources, and Microsoft can then do bugfixes and updates themselves.
What would happen if hardware vendors actually provided specs to the public? Well for one the specs would obviously be used to support that hardware in Linux and BSD. Better drivers would get more people to use those operating systems, something which is not in Microsoft's interest.
Mac OS X can do it.
Are those the same people who came up with the ...... idea that a bottle of water could be used to perform a terror attack? (If you open a bottle of water onboard an airplane, the oxygen and hydrogen is going to react and cause a huge explosion, or something).
Interesting question. What if somebody were cheating? Maybe somebody wanted to give the impression that they did a large amount of computation, and modified the code to skip large parts of the solution space and report back no solution was found.
You can suspend to RAM and disk. Then you get fast wakeup most of the time, and losing power doesn't mean lost data, just a bit slower boot.
Even that I can imagine being solved if you have sufficient processing power in the TV. How often is a key frame sent? Every five seconds? If the TV would figure out which 10 channels you are most likely to zap to next and store the last five seconds of compressed video for each of them, switching channels would just boil down to how quickly you could decode those five seconds of video.
Look at what happened with teletext. With early televisions supporting it, you would have to wait 10s of seconds for a page to show up. Today they show up instantly. After all it would only take a few MB of memory to store every page as it was sent over the air and keep it just in case you wanted to see it. A few MB of memory was a lot when teletext was invented, today it is nothing. Buffering MPEG streams requires a few orders of magnitude more memory, but other than that it is pretty much the same.
In the case of DOS version numbers were actually used by the software for all kinds of compatibility checks. So vendors would basically be forced to make their software report the version number of the MS DOS version they were compatible with. If something was sold as DR DOS 5.6 and told applications that it was a 6.22, then consumers would have been confused. I can see why they would just follow whatever version numbers Microsoft happened to decide on.
BTW. As I recall it, it was not DOS 3.3 that was used for a long time but rather DOS 3.30. I think DOS 3.30 was the one version that was used for the longest period of time. And DOS 6.22 was the last version in widespread use. As for DOS 7.x I don't think I have ever seen anybody use the full system. A lot of people have used the stripped down version (without knowing) to run Windows 9x on top of it.
Exactly my thought. I think giving back addresses might actually be bad for the internet as a whole. Giving back addresses means more fragmentation in allocations, which means more headaches for routing. Giving back addresses also means, that by the time we really have to convert, there will be more systems to convert, hence the cost will be higher.
You clearly have some good ideas there. To protect against certain active attacks against encrypted devices, you would actually need a hash tree and a signature or MAC of the root of the tree. I know that ZFS already has some of it. And I agree that some of the integrity it provides is useful for the raid layer as well. And yes, journaling is one way to handle the atomicity issues. It applies to file systems as well as an encryption that does use the extra space. I am not entirely sure what the right kind of layering is. On one hand I feel that integrating all the layers with encryption, raid, journaling, and file system, could end up being a bad spaghetti. On the other hand. I can see what advantages it gives. As for the on disk layout, I don't think an area at the start of the disk for IVs and MACs is the way to go. I think you would get better performance by putting that data in single sectors spread across the disk. One such sector for every n physical sectors on disk, where it would be related to the sectors just before or after.
No, it doesn't need nearly that much. For each sector written it needs 16 bytes or randomness from the system random number generator as well as 16 bytes from its own PRNG. So that amounts to just 1/16th of the size of the data. For reading it need to generate the same 16 bytes using its own PRNG before it can decrypt the data. I am pretty sure it is not the PRNG that is going to be the bottleneck but rather the AES encryptions. (If you have hardware to do AES but not to do the random numbers, then it might be different).
As for your suggestions on how to fix the nonatomicity, I don't know the solutions you suggest well enough to say whether they will work. But they sound like plausible solutions.
Since the encryption is done at the block device layer, it shouldn't be part of the file system but rather a part of the encryption layer. Except from that detail, I agree with you. However doing it is tricky. When you take a logical sector with 512 bytes of data and put it together with a small IV, it no longer fits in a 512 bytes sector. So you have to come up with a layout on disk that is efficient. And then you have the problem of updating it. Because you cannot write two sectors to disk atomically (with some disks you can't even write one). So you end up having the potential of data loss if an update does not complete because of a power failure. And the data loss is not guaranteed to affect only the logical sector you were writing at the time. I am still thinking about the problem. A raid layer have some similar problems, and I have been considering whether merging those two layers would help solving this problem. A can see a few other advantages you could get from merging a raid layer with an encryption layer.
Indeed. And the damage depends on the kind of encryption used. CBC is actually dealing pretty well with reused IVs compared to other encryptions. RC4 and counter mode are two examples where reusing IVs is so bad that you might as well not be encrypting at all.
Interesting. Though the patterns are simple, I wouldn't have thought they would show up unless they were carefully constructed.
That's the vulnerability I was referring to. And this is what is demonstrated by the file I linked to. It requires just one or two bits of difference between neighbor sectors depending on which parameters you used to set up the volume. My file contains just a lot of variants of that to cover different byte orderings so as to demonstrate the weakness not just in truecrypt, but also in some early Linux loopback encryptions.
The fact that a file system block consists of multiple disk sectors just makes the exploit slightly more reliable. Even if the sizes were the same, the file system would still try to put consecutive blocks from your file right next to each other on disk. (If it didn't do that, performance would go down by orders of magnitude).
Correct.
Depends on whether you are worried about breaking compatibility. There is nothing you can do for existing encrypted volumes. If you want to improve security for your existing volumes, a complete reencryption of the volume is needed. If you want to protect new volumes while remaining compatible with the existing implementation, a small change to the way the master key is generated would help. There is a 256 byte lookup table, in which having two identical bytes is a weakness. The table is generated randomly, which means master keys vary in quality. The best master keys are those were all 256 bytes are different, but the chance of that happening at random is negligible. If the key was instead generated by taking an array containing all the values from 0 to 255 and permuting those randomly, you would always get keys that are resistant to the known attack.
If you don't need compatibility with the existing code, you can do even better. The PRNG makes use of MD5. Before anybody starts talking about MD5 being broken, keep in mind that the known attacks against MD5 do not apply to the way GBDE use it. The input that GBDE passes to MD5 is 24 bytes, which have 128 bits of entropy (with the best keys). Obviously the output will have no more than 128 bits of entropy, but it could have less. Though the input is just 24 bytes, MD5 is going to add a length field and pad the result to a multiple of 64 bytes. So you could actually double or triple the key size and pay no extra cost in performance on the MD5 operation.
Since three byte quantities are difficult to work with, I'd only suggest to change the lookup table from having 256 8-bit values to having 256 16-bit values. The risk of having two identical values if you pick them randomly, is reduced significantly. But still generating it in a way that guarantees it is still better. This would produce a 40 byte input to MD5. If you wanted to make use of the full MD5 block size, you could append another random key to the 40 bytes such that you would make the input as large as what would fit in a single MD5 block. Doing all of this would increase the key size from 272 bytes (16+256) to 584 bytes (56+16+512), and would not spend any additional time on the cryptographic operations in the critical paths.
You could also ditch the PRNG altogether and use a standard PRNG. But that would mean a significant performance hit, so you'd have to reduce the size of the master key. So I am not sure that would really make it any stronger than fixing the known vulnerabilities in the current PRNG.
As for the potential data loss goes, I suppose you could live with it and just make sure to have a good backup strategy. When doing that of course you have to keep in mind that you shouldn't make backups by copying the encrypted container. The correct way would be to copy the files from the encrypted container to a different encrypted container. Or you could copy the files to an encrypted archive that does not support random access, like tar + gpg. Or you could back up your container by creating an encrypted copy of the container. If you use asymmetric keys, you could create a gpg encrypted copy of the encrypted container even while it is not mounted.
Problem is that the only way to detect that corruption has happened is by trying to open the files with applications understanding the format, and see them barf. However fixing the potential corruption is tricky. If I find a good solution, I am probably going to write my own storage encryption layer some day.