Slashdot Mirror
Oracle Adds Data-integrity Code To Linux Kernel
jazir1979 writes "ZDNet is reporting that Oracle has added code to the Linux kernel for ensuring data integrity. The code has been developed in partnership with Emulex and was recently accepted into the 2.6.27 kernel release." According to the article, "The code adds metadata to data at rest or in transit, to monitor whether that data has been corrupted. It helps make sure that I/O operations are valid by looking at that metadata — which acts as verification information — exchanged during data transmissions."
Seems like Linux is going to have a great momentum in these coming months.
How badly does this affect performance?
It adds a 2nd layer of metadata that is used to verify the first layer of metadata wasn't corrupted so you can be EXTRA confident that your original data was actually handled correctly.
No sig for you!!
Nice one, some time ago people used magic numbers to distinguish structures in the dumps, now they have meta data for that.
As long as it applies to every data in transit, it should bring stability into the kernel.
Does one OR two layers make you feel more confident?.. Why not put more effort in back checking I/O operations rather than adding more data to the stream? Is this a band aid fix? Apparently not in the eyes of Oracle..
Did anyone else read the headline and think "Integrity" in terms of security (you know, as in the good ol' CIA Triad)? And then if you did, did you also knee-jerk a response like this: "What?! Oracle adding security code to the Linux kernel? With THEIR security reputation of their products!? $#@%"
It pre-corrects a future corruption in the as-yet-unimplemented third layer of metadata. Kernel developers have decided to add the third layer and accept the patch on the grounds that the corruption might still have occurred even if Oracle hadn't said anything.
I thought I'd be uber pimp after using XFS, boy was I wrong. That file system becomes corrupt after about every six months for me, with no way to repair it (nothing I've tried works). I'm running an ext3 filesystem now for my root partition and data being on an ext3 and another on a seperate xfs drive (luckily this filesystem never became corrupted). No problems so far for about a year.
I wonder if these patches make a difference in different filesystems or what exactly the meaning is? It says transactions "in transit" well if the hardware is bad I wonder how the software knows whether what it thinks is valid really is?
Sounds like a layer of code to slow things down without really providing anymore protection against errors.
Block integrity patches were discussed in excellent article on LWN in July 2008. Kernel 2.6.27 was released in October 2008. This is old news.
:wq
You've invented the Checksum
On a more serious note (yes I did RTFA), somebody please explain where this fits. Other than network or disk errors (which generally already have error detection schemes), I'm not sure what the target problem is that this is supposed to fix. The article says "the code helps maintain integrity as data moves from application to database, and from Linux operating system to disk storage", that it checks I/O operations, and that "code contribution includes generic support for data integrity at the block and file-system layers". That's still not clear what they think the problem is. Don't most of the modern file systems already check data operations?
This is obviously pure PR. It reads like a press release - the journalist understood even less about it than (s)he usually does.
Seems like Oracle is being clever at creating buz about their participation in Linux development - while using the words that they use to sell there products. Having "Linux", "Oracle" and "Data integrity" in the same sentence, without paying for it, is very nice.
Now, it doesn't mean that they haven't done something good. But until someone explains me what it is, I'll take it with a grain of salt.
Sure it might bloat the size of the source code, but hopefully you can choose to include or not include it during a recompile. So it's not so bad.
link is skim on details. any word if there is error correction or is it just detection? what does this add that say erasure coding (reed-solomon) lacks?
Is this as much of a paranoiacs nightmare as NTFS Alternate Data Streams?
What exactly is the "Data Integrity Payload" that we will be adding to these files... how can this be abused?
and most importantly... Most people won't need this so, Will this sucker be turned on by default? Will there be an easy way to purge your files of unwanted metadata?
How long will it be before systems start requiring certain metadata attached to files before accepting the file? Is this the foot in the door for DRM in the kernel?
I don't know where it fits either, but ZFS and eventually BTRFS actually have checksums at the block level, and can heal over corrupted blocks using redundant copies whose checksums do work. That alone is enough reason to use ZFS for a file server, but similar improvements could be made inside the Linux stack without a new filesystem on top. However ZFS' reliability also comes from copy-on-write updates which is not trivially installed into an existing filesystem.
Sam ty sig.
Excuse me if this is a dumb question, but how does this differ from the journal in many existing filesystems?
I'm not certain but it appears to be checksumming data while it is moving around the kernel after a write or read call is made.
Seems like something that should be handled in hardware with ECC, but what do I know.
... whatever that means.
Info
I'm not certain but it appears to be checksumming data while it is moving around the kernel after a write or read call is made.
Seems like something that should be handled in hardware with ECC, but what do I know.
Kernel bugs can cause data to get corrupted and hardware ECC won't correct that. Likewise with transfers from memory to disk. Ultimately it'll need to be a hardware/software thing but the software portion is needed as well.
"When you sit with a nice girl for two hours, it seems like two minutes. When you sit on a hot stove for two minutes, it
This is industry standard checksumming to ensure end to end data integrity from the disk to the storage system to the HBA to the OS to the app. I'm quite stoked for this since my SAN vendor (Xiotech) has the first system to support the standard (Emprise 5000/7000) and we have Oracle 10/11G already in our environment.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
RAM errors.
Spontaneous bit flips change data in transit.
It also helps against errors in kernel code or malicious data injection attacks
- Michael T. Babcock (Yes, I blog)
Im confused...
did oracle just remove the whole point in having ECC ram in the first place? Could this also possibly lead to the kernel making the system much more resistant to say... radiation? (certain particles can cause a bit to get flipped when they pass through the computer, causing kernel panics and various unpleasantness) I am but a lowly college student from Mississippi so I know very little about this subject, any idea from the more educated slashdotters?
One of the problems that this is supposed to detect is blocks getting written to the wrong place or being read from the wrong place. I think it's one of those rare problems that stops being quite so rare when you have huge amounts of data stored on cheap hardware.
Mea navis aericumbens anguillis abundat
Maybe linux will break .8% of the market with this groundbreaking advance.. *snicker*
Windows 37.4% (268), Linux 34.6% (248), Unknown 19.2% (138), Macintosh 7.6% (55), FreeBSD 0.5% (4), Solaris 0.4% (3).
I think it depends on which market you're talking about.
ECC usually covers specific paths or devices, but it doesn't give you an end-to-end integrity check. A similar situation happens with IP packets. You can disable packet checksums if you like to live dangerously. Then, all you need is a bit of noise or a hardware problem to silently corrupt data that flows over the network.
Mea navis aericumbens anguillis abundat
I don't know where it fits either, but ZFS and eventually BTRFS actually have checksums at the block level, and can heal over corrupted blocks using redundant copies whose checksums do work. That alone is enough reason to use ZFS for a file server, but similar improvements could be made inside the Linux stack without a new filesystem on top.
Without changes to the filesystem where do you put the new checksum for each filesystem block?
You might not understand why we need it, but trust me, it is needed. Not all storage device drivers are created equally, and some will happily report to the kernel that the write operation was successful even if it wasn't, and you end up with corrupted data. When Oracle operates on a trusted environment like Solaris on Sparc, this type of integrity is built in to the operating system and it's not necessary to do a read after write to verify the data was written correctly. On Linux, and other untrusted operating systems where this doesn't happen, Oracle has to do a read after write to make sure the data was written correctly. This slows things down quite a bit.
"When the president does it, that means it's not illegal." - Richard M. Nixon
The article I read about this states that Linux is the first operating system to implement these standards, T10 PIM and DIE. It says that they are looking to implement the same technology in Windows, Solaris, and other Unixes. It also states that this is implemented both in hardware and software. In what way does this differ from Solaris/SPARC's data integrity implementation?
Time makes more converts than reason
It may not be the same implementation, but Solaris on Sparc has built in ECC across all data paths, including CPU -> memory, CPU -> I/O, etc. I believe it is a combination of hardware and software (kernel) that does this.
The article, I believe, is talking about implementing the same level of software checking in x86 versions of the operating systems mentioned.
"When the president does it, that means it's not illegal." - Richard M. Nixon
Below the filesystem. ZFS can export zvols, which are just block devices whose storage maps on to ZFS blocks. They get checksumming and copy-on-write semantics, even snapshotting, and yet still support any filesystem on top, a whole partition table, a virtual machine disk, or whatever. They're just block devices, but they still get most of the reliability advantages of ZFS itself.
It's not as efficient as storing a ZFS filesystem which can track its used/free space and feed that in to the reliability systems, but if you want reliability, you get a lot of it for zero effort.
What I'm saying is that, as far as I understand, the same volume reliability features can be implemented for Linux without a new filesystem. You'd just have a new integration of existing RAID and block mapping features, plus block level checksumming and copy-on-write (feeding in to the device mapper). Even RAID-Z could be implemented in this way without needing a new filesystem. You can already run Linux filesystems over iSCSI to a zvol, so why not just implement the zvol directly in Linux?
Sam ty sig.
SAN vendor (Xiotech) has the first system to support the standard (Emprise 5000/7000
Which standard?
Has Xiotech added Emulex HBAs to the compatibility matrix for those systems yet? ROFL.