Ext4 Filesystem Enters Experimental Kernel Tree

An anonymous reader writes "Looks like the next version of the venerable Linux 'ext' filesystem is just around the corner. Andrew Morton has added an early version of ext4 to his 2.6.19-rc1-mm1 tree, enabling Linux to support storage volumes up to 1020 petabytes in size, and to write files in 'extents,' or contiguous, reserved areas. According to an article at Linux-Watch, ext4 will be ready for production use within six to nine months, if all goes well. On the downside, the new ext4 filesystem will offer only limited backward compatibility with ext3-aware Linux kernels."

Comments

Reiser4

[Anonymous Coward]

Unfortunately, this will just murder Reiser4.

Re:Reiser4

[Profane MuthaFucka]

http://geekz.co.uk/lovesraymond/archive/so-i-marri ed-a-kernel-programmer

Re:Reiser4

[osee]

How come everyone makes these murder comments as Anonymous Coward?
Are you afraid of something?

I found the following on the talkback page:
"Yesterday came the announcement of Hans Reiser's arrest for the murder of his missing missus. Yeah, I know: "innocent until" and whatever, but OJ and Scott Peterson have already blown that curve.

Today comes news of a new and potentially powerful Ext4 fs. I find this welcome and timely news.

I'm presently, happily running linux via knoppix on my Toshiba laptop. As of yesterday's news, however, I'm feeling kindof dirty; my little 60Gb hard drive is running on reiserfs 3.

I will, as soon as I can practically do so, upgrade my OS and its file system to something, ext4 or otherwise, that is not associated with a suspected murderer.

Genious does not excuse immoral or illegal behavior: I don't care if reiserfs4 can read/write files at petabytes-per-millisecond; wrong is wrong.

I encourage all other linux users to boycott using resierfs as a statement that morality still means something.

We don't allow this sort of behavior from our politicians or other public officials. We don't excuse it for our football or basketball heroes. Why should we do so for our open-source gurus?

I say we shouldn't.

Linux has its own mountains to climb without having to be bogged down with salacious crap like this.

We should, as a community, put reiserfs behind us. Let us bury this now in the dirt where it belongs, then salt the earth."

This is so much bullshit I could cry. Hans does not benefit from me using reiserfs... Why should I drop it?
Even if he is convicted that doesn't mean he is guilty. Even if he was he can still be a great programmer.
I don't want him to marry my sister, but that's whole another matter.

Re:Reiser4

[Anonymous Coward]

the police were wishing that ext3 had murdered his wife instead, because then the body wouldn't have been cut up into so many small pieces.

Re:Reiser4

[Anonymous Coward]

This ext4 looks nice and everything but everyone knows that Reiser4 is the real killer here.

Re:Reiser4

[Leadmagnet]

I wonder what advantages EXT4 has over the newest version NTFS?

Re:Reiser4

[udippel]

I will, as soon as I can practically do so, upgrade my OS and its file system to something, ext4 or otherwise, that is not associated with a suspected murderer.

As much as I agree with you questioning the many 'funny' ACs, I can't support this statement. One day you, yes, you, might also become a suspect. And then, suddenly, you would see the world with different eyes. Especially, when you're found innocent and you also find that your friends have deleted all memories of you since they don't want to be associated with you; as suspect.

I am sure many of us would feel similar once the investigation is over and Hans eventually found guilty. But as civilised people, we better wait. How old are you ? Have you never ever been suspected of something ?

Re:Reiser4

[Anonymous Coward]

Hate the sin, not the sinner.

Re:Reiser4

[BrokenHalo]

Well said. I, for one, am going to continue using ReiserFS until someone comes up with a damn good technical reason why I shouldn't.

Re:Reiser4

[drinkypoo]

You are what you do. Only Christians and other orthodox religious wingnuts believe that there is a difference between the so-called "sin" and "sinner". We all make our own decisions in life and are ultimately responsible for them. Orthopraxy all the way!

Re:Reiser4

[FST777]

Do you actually READ a comment before you reply? I mean, it's bad enough that folks don't RTFA, but replying to a comment without even READING what he said is just pathetic.

I suspect even the mods who modded GP down didn't read properly. He was QUOTING and then stating why he disagreed with said quote.

Now go read the GP's comment again, and take heed to spot the quotation marks.

Re:Reiser4

[HiThere]

You have more faith in the US court system than I do. Whether he is found guilty or innocent, I will consider the matter "undecided" until I have some personal need to decide. Then I will decide based on what I know...and a decision of the court would be only one factor to be considered. An important one, perhaps, but not in and of itself decisive. Presumably evidence will be brought forwards during the trial that won't be sealed. That should be the main factor.

I would consider that "People are usually killed by close friends or lovers" together with the arrest would be good evidence if those were independent facts. They aren't. It may well be that the MAJOR reason the police have for suspecting him is that he is an estranged husband. That together with her disappearance is sufficient for suspecting him. They have some other evidence, characterized as "circumstantial", whatever that means. Some "biological materials". What this will actually show is uncertain.

Given what I know I see three main possibilities:
1) Hans did it
2) The other guy did it
3) She went into hiding for some reason (which may be totally independent of this case)

In general terms, and based only on publicly available information that I happened to notice, I listed them in order of my estimate of decreasing probability. Please note that the I would consider the residual probabilities to be 40% or higher, and they include several corner cases together with "exogenous factors" ranging all the way down to
n) a MS hitman "removed" her to disrupt linux development (at considerably less that 0.01% chance).
At
4) Something I haven't thought of (around 20%)
I realized that this wasn't something I could reasonable come to even a tentative conclusion on. If I guessed now it would be because circumstances forced me to.

Re:Reiser4

[Anonymous Coward]

I see what you did there.

Re:Reiser4

[philwx]

You are what you do. Only Christians and other orthodox religious wingnuts believe that there is a difference between the so-called "sin" and "sinner". We all make our own decisions in life and are ultimately responsible for them. Orthopraxy all the way I'm pretty sure they believe sinners will be judged, just not by people. And while that may not be sufficient to you, it doesn't have to be; there is still a government to punish crimes. Best of both worlds. Happy to be a wingnut.

Re:Reiser4

[blank axolotl]

offtopic and gossip.. but you inspired me to to check up on that

The police say they found his wife's blood splattered in his house and car. He had tried to hide the car from the police and took the back seat out, and bought books about homicide investigations (though that is understandable, him being a suspect).

link

Doesn't look promising... but as you point out 'innocent until proven guilty'. (and I'm guilty of gossip here)

Re:Reiser4

[udippel]

Thanks for the pointer, you are right.

Though, the author must be a programmer and the reader must count opening and closing quotation. > 80% of the comment is quotation, right, and in the end the author of parent says Even if he is convicted that doesn't mean he is guilty.

That makes my comment bad, but not the mod. It makes the comment redundant for copy & paste of another post to finally say "I don't agree with this".

Re:Reiser4

[Nutria]

4) Something I haven't thought of (around 20%)

She died in some off-beat accident that only Gil Grissom could deduce.

Re:Reiser4

[Nutria]

(and I'm guilty of gossip here)

How is repeating a newspaper story considered "To run about and tattle; to tell idle tales"?

Re:Reiser4

[try_anything]

You are what you do. Only Christians and other orthodox religious wingnuts believe that there is a difference between the so-called "sin" and "sinner". We all make our own decisions in life and are ultimately responsible for them. Orthopraxy all the way!

I think you're a bit mixed up. Orthopraxic religions are more relaxed about what people say and believe theologically, but this has nothing to do with hating people who do evil things. There's no "conservation of religious nastiness" that forces them to hate murderers to compensate for tolerating heterodoxy. Buddhism, for instance, counsels against hatred and makes no exception for hatred of murderers.

Furthermore, while some people believe that moral responsibility should be enforced through the hatred of believers, this is unfortunately a recurring pathology in all religions, not something limited to a specific class of religions.

Re:Reiser4

[try_anything]

I agree with your skepticism 100%. The evidence they've come up with is easily explained. (I don't mean to imply that you implied the evidence is conclusive, when in fact you said the opposite, but this is a convenient place for me to play devil's advocate.)

The police say they found his wife's blood splattered in his house and car.

A recurring theme in weak and eventually overturned convictions.

He had tried to hide the car from the police

He isn't cooperating with them, so they will characterize anything they have difficulty finding, or find in a weird place, as "hidden."

and took the back seat out

It's harder to deal with that one. I've known people who had (and even drove) cars with missing seats, but they were rural teenagers and college students, not prosperous urban software developers. On the other hand, the guy dropped out of junior high school and got into Berkeley anyway, so he's an odd duck.

What makes me most skeptical is that the police investigated for weeks and then said everything they found led back to him. That seems mean that they didn't find any evidence that pointed any other way, and the less evidence they found, the more confident they felt in their first guess. (Data can make you pretty sure of something, but no amount of data can make you feel as sure as you would if you didn't have any data at all.) Next they order a forensic examination of his house and car and frame what they have in the most damning way possible for the press. That projects confidence, which may prompt the suspect to confess and will go a long way toward convincing a judge and/or jury if no further evidence is discovered.

But I'd still bet on him being the killer :-)

1020 Petabytes?

[skrew]

This will lead to overly cluttered hard drives, with never having to delete files it'll take more work to keep it organized.

Re:1020 Petabytes?

[Angstroem]

Oh, please.

By now you don't even now what to do with 1024PB. Just as we couldn't imagine filling a 250GB harddrive 15 years ago when 500MB were considered huge.

What will happen? We store our digital photos in raw format, not JPEG. We store our songs in raw format, not artificially crippled. We will store high-definition video, possibly even in raw format, not MPEG4 or the likes.

And, woosh, 1024PB will be nothing leaving us wondering how we could ever survive with a measly 250GB drive -- just as we ask ourselves today how life was with nothing but 170kB disk drives.

Re:1020 Petabytes?

[tomhudson]

What will happen? We store our digital photos in raw format, not JPEG. We store our songs in raw format, not artificially crippled. We will store high-definition video, possibly even in raw format, not MPEG4 or the likes.

Come on, admit it ... what you really mean is you'll store your pr0n in raw format ... and you'll STILL be complaining that you don't have enough disk space.

Re:1020 Petabytes?

[HerrEkberg]

So the solution is to limit the storage capacity of computers, to ease the work of keeping data organized? Wow, why didn't anyone think of that when we had 500 MB hard disks? Organizing data would be a breeze for us now; but no, instead we are stuck with our large capacity systems.

Re:1020 Petabytes?

[astralbat]

And, woosh, 1024PB will be nothing leaving us wondering how we could ever survive with a measly 250GB drive -- just as we ask ourselves today how life was with nothing but 170kB disk drives.

I'm not convinced by this myself. I do however see a need for super computers who need to work with filesystems spanning perhaps hundreds of disks. As for the desktop user, even if they did store their files in raw format, I doubt they'd use more than a few 10's of terabytes at the most.

Re:1020 Petabytes?

[Rakshasa Taisab]

There's a pun in there about being exact, I think that pun might even be constructed with several layers of punyness, but I'll leave that as an exercise for the reader.

Re:1020 Petabytes?

[CastrTroy]

This is exactly how I'm filling up my space. I got a new computer with a 160 GB drive, thinking it would be "enough". Started storing all my CDs in FLAC, and I'm currently transfering all hte movies I downloaded in AVI to DVD so I can watch them easily on my home theatre. Once you start working with video and sound that isn't compressed to nothing, you start to realize just how fast you can use up all that space. If my camera did RAW i'd probably use that to store my photos. I usually save any edits I do in PNG or TIFF so that I don't have to worry about the lossy encoding. Granted I still have space to spare, but I could see very easly using up a Terabyte drive if I had it, and a faster internet connection.

Re:1020 Petabytes?

[RAMMS EIN]

Meh.

find "$dir" -amin +"$time" -print0 | xargs -0 rm -fr

Re:1020 Petabytes?

[HerrEkberg]

Well, that depends on what your expectations for the future are. I don't think it is impossible that demands on multimedia will reach high enough sometime. Let us as an example consider a movie file from the Future (tm). Given better and bigger screens (perhaps covering whole walls) a frame dimension of 3000 x 2000 pixels is not inconceivable. Each pixel might consist of three RGB values of 16 bits each. Such a movie, if two hours long and running with 25 frames / second, would require about 6.5 TB in raw format.

framedimensions = 3000 x 2000
framebytes = framedimensions x 6
moviebytes = framebytes x 25 x 60 x 120

moviebytes / 10^12 ~= 6.5

Re:1020 Petabytes?

[astralbat]

Interesting.. That'll be UHD-DVD then

Re:1020 Petabytes?

[tomstdenis]

You don't need to store your audio as FLAC. Only weirdo audiophiles do that [unless you own the masters then it's ok]. Use 192 to 256kbps M4A and be done with. As for the movies, you should be able to convert ON THE FLY to DVD so you don't need to store uncompressed frames or whatever. As for the camera, just use high bitrate JPEG.

Unless you are publishing your audio, videos or photos you don't need a 100% representation. Often that 97% quality but 1/10th the size copy will be fine for your own enjoyment and use.

Tom

Re:1020 Petabytes?

[tomstdenis]

You forgot the 16 channels of 192KHz 32-bit audio you need to. That's 84GiB on it's own!!! :-)

Tom

Re:1020 Petabytes?

[Eivind]

Actually, there's natural limits to that kind of storage.

The limits are set by our senses, more concretely, our ears and our eyes.

Our ears are only capable of hearing up to about 20Khz (less than that for most people) and 16-bit samplings are enough that most people cannot hear the difference with anything more. Thus CD-quality is, if not perfect, then good enough that further improvements are ignorable for most people. CD-quality losslessly-compressed music is around 300MB/hour.

In a year, there's 8760 hours, so you'd need on the order of 2.5 TB to store a year worth of around-the-clock never-repeateing losslessly-compressed music. If computers keep getting replaced at the current rate, this means you'll never need more than about 10TB to store sound. This assumes you don't store more than you listen to, if you choose to for example store all music ever produced for convenience, despite never listening to more than a tiny fraction of it, then this requirement goes up by a couple of orders of magnitude. Still, there's good reason to suppose that 10TB will suffice for most peoples sound-storage needs. (even if you wanted to store all the sound you've *ever* heard in your life, including traffic at nigth, that'd still only be 200TB or so)

The real killer is video. We can take in a *lot* more data with our eyes. 10GB/hour is in the ballpark of what you'd need for the sort of quality a modern cinema can deliver. (and there's no particular reason we couldn't go higher.) That works out to 100TB/year, more or less. A lifetime of high-quality video is thus on the order of 10PB.

In short, it is unlikely that an individual (or family) will be able to fill a 1000PB disc with sound and video-recordings. Infact it's unlikely they'll be able to fill it with anything, if that anything is to be consumed only trough their 2 eyes and 2 ears.

That doesn't mean it won't happen. Only that it'll be filled with something more. Once we fire up the holodecks all bets are off. I don't even want to try to estimate the bandwith needed for that kind of immersive experience.

Re:1020 Petabytes?

[Anonymous Coward]

Your reasoning is flawed by a couple orders of magnitude..

Re:1020 Petabytes?

[AccUser]

it's unlikely they'll be able to fill it with anything

People always used to ask me if I had the internet at home. Maybe when I can get my hands on a 1020 PB hard drive, I will be able to download it all for local access... :-)

Re:1020 Petabytes?

[CastrTroy]

I don't need to store any uncompressed video on my hard disk, but having a couple 15 GB of VOBs floating around that I haven't had time to burn yet takes up well, 15 GB. As far as i've found the easiest way to go from AVI (divx, etc) to DVD requires encoding to MPG, while the files are 3-4 GB, and then creating the VOB files from that, again 3-4 GB, and then burn those. I don't think I could go straight from AVI to burn, because it usually takes a few hours to go from AVI to MPG.

Re:1020 Petabytes?

[CastrTroy]

Also, I store my audio as flac, not because I can hear the difference between 256 kbps M4A, but because i lose/scratch the CD, I still want to have a way to get a perfect copy back.

Re:1020 Petabytes?

[tomstdenis]

VOBs are just MPEG-2 streams. Also nothing says you have to do your movie as one huge 15GB VOB. You could break it into 1GB parts and then delete the temps as you go.

mencoder can transcode pretty much anything, unfortunately only into AVI [the MPG output sucks bad], but ffmpeg can do some other streams [notably though it doesn't like 5.1 AC3 streams though ...].

Tom

Re:1020 Petabytes?

[tomstdenis]

If you can't hear the difference between a high bitrate m4a and the original ... does it matter?

That's the thing, the "difference" is only in your mind [not sensory]. Recall the whole point of psychoacoustic based encoders is they take advantage of the disparity of S/N ratios on various bands. If you have a 10dB masking on a given band, encoding it with full 96dB range [e.g. 16-bit PCM samples] doesn't make sense.

Imagine you can't see the colour red. Would adding more bits to the red channel make the picture better? If you can't hear that masked tone, adding more bits won't make it any better.

At the point where you are encoding at >=192kbps the encoder doesn't have to cut out resolution you can perceive so you're essentially home free.

Tom

Re:1020 Petabytes?

[entrylevel]

Insightful? Yes. Informative? Certainly not. Finally the Funny mod hits (what took so long?) This is the funniest thing I've read on Slashdot in ages, on so many levels.

On second thought, maybe it is Informative, since I was not previously aware you could cram that many puns into so few words.

Re:1020 Petabytes?

[glsunder]

If you can't hear the difference between a high bitrate m4a and the original ... does it matter?

Yes, it does, mainly for re-encoding. What if he wants to convert them to 128kbps for a portable player. Or what if in 5 years a different compression is used? Personally, I ripped about 120 of my cds to mp3 this year and it's not something I plan on doing again soon. I can completely understand someone never wanting to have to redo it.

Re:1020 Petabytes?

[Bwian_of_Nazareth]

When you need to edit the damn thing, it is quite important to have it in a lossless format. If you loose 5 % everytime you save the file, by the fifth save the quality may be lost forever.

Re:1020 Petabytes?

[tomstdenis]

Use open source software.

As for going down in bitrate ... um ... if the 256kbps audio is INDISTINGUISHABLE from the original ... it might as well be the original. Sure the bits are different but the sound produced is perceptible as the same. I've re-coded stuff before for my portables and frankly I can't really tell the difference. Specially since you typically listen to portables in noisy environments where the noise floor masks most subtle noises.

Tom

Re:1020 Petabytes?

[Anonymous Coward]

I could see only 1 pun.
What are the others?

Re:1020 Petabytes?

[NerveGas]

This reminds me of a conversation I had about ten years ago with a coworker. One of us brought up the new 1-gig drives that had been released. He let out a slow whistle, and with a grin on his face, said "Boy.... that is big. It would take me at least three weeks to fill that up."

Re:1020 Petabytes?

[MMC Monster]

84 GB is a drop in the bucket compared to the 6.5 TB of video data. Crazy, but true. :-)

Re:1020 Petabytes?

[OmnipotentEntity]

It doesn't matter if you use OSS or Closed Source, it's a lossy format. Sure you might not be able to hear the encoding on a 256kbps audio file, if you've only just encoded it, but try transencoding a file from aac to ogg to mp3 all at 256kbps and see if you can't hear the compression.

FLAC is the only logical choice if you think you might have to reencode later. Or burn back to a CD, or remix the track, or make a home movie with it, etc. etc. etc.

Re:1020 Petabytes?

[John Nowak]

It isn't nearly that simple. Oftentimes, in audio production, you work with audio that's much higher than 16-bit... maybe 24-bit, or often internally in software, 32-bit. Can I hear the difference between 16-bit and 24-bit? No. However, if you're going to be processing it heavily, you can hear the difference after it's finished. This is especially true when doing dynamics processing. The more bits an audio encoder has to work with, the better (within reason).

Re:1020 Petabytes?

[squoozer]

If anything I would say that your resolutions are way to low. For a full wall display of ultra high quality I would guess at 10000x4500 (for roughly anamorphic). I would also up it to 50 FPS for a really smooth display at 48 bpp a two hour movie is ~100TB for the video alone. Trouble wouldn't be storing it - it would be getting it off the device fast enough.

Re:1020 Petabytes?

[ndsbriand]

BenBenBen wrote:

1 Exabyte!

You are hereby notified to refrain from using our trademark in such a manner. If such pratices continue, we may be forced to take legal action.

Regards,


Exabyte Corporation

Re:1020 Petabytes?

[springbox]

By the time people have hard drives this huge, I bet there will be even better lossless compression algorithms available for video and audio data. I don't know why someone would want to store huge amounts of A/V data WITHOUT using some sort of compression. It just becomes silly after a certain point.

Re:1020 Petabytes?

[entrylevel]

I can't believe I'm answering this, but...

• Excitebike sounds like exabyte
• 1020 petabytes != 1 exabyte
• 1 exbibyte != 1 exabyte
• In terms of scale, 1 exbibyte and 1 exabyte are completely different, the difference in this kind of mistake could equal several hundred thousand copies of Wikipedia, as opposed to confusing KiB/KB or MiB/MB, which are different, but not earth-vs-sun different.
• The post was moderated Insightful and then Informative before it was modded Funny
• I'm counting the excellent sig too

That's 6, and I'm sure there are more I'm missing... Sorry if you think I'm stretching, but I really think this is one finely crafted layer-joke, and I just enjoy that kind of thing.

Re:1020 Petabytes?

[tuffy]

As for going down in bitrate ... um ... if the 256kbps audio is INDISTINGUISHABLE from the original ... it might as well be the original.

And what about going up in bitrate? Or switching to a different codec?

128kbit CBR MP3 used to be good enough for everybody and "indistinguishable from the original." Then it was 192kbit VBR mp3. Now it's m4a. What will it be next year? When will the next big lossy audio player roll around with an even better psychoacoustic model? Who knows, but I'm sure there'll be one.

FLAC means never having to re-rip my CDs. No matter what the next big lossy format is, I'll be ready for it. Considering that hard drives cost about $.50/gigabyte and I can fit about 3 losslessly-encoded albums per gigabyte, I opt to store them as-is and avoid throwing away data until I need to transcode my music for portables and so forth.

Re:1020 Petabytes?

[smallpaul]

Given petabytes of storage, we would take a very different approach to media. For example, we might make much more significant use of webcam capture, perhaps putting webcams in every room to catch anything interesting that might happen (especially for people with children). We also might capture everything that comes over the airwaves or television cable. We also might cache HD movies forever. We might move to a software caching mode whereby we have all of Microsoft's or Red Hat's software on our computer (auto-updated during the night) and we just turn it on when we need it. A lot depends on the ratio of bandwidth to disk space.

Re:1020 Petabytes?

[LarsG]

Think of it in terms of archive vs use format. If storage capacity is cheap, it makes sense to store all your media in the highest quality available (e.g. FLAC or other lossless if you rip from CDs, VOBs from DVDs). While MP3 and MPEG2 will be with us for a long time, when the codecs of choice for player devices change it makes sense to have the highest possible quality source available when you re-encode. It is perhaps even more important if you want to edit the media (trying to use YouTube or Google Video as source for a mashup is ugly).

With HD content available just around the corner, the storage requirements for full quality archive copies will go up a lot. A PVR set to capture digital broadcasts of all your favorite TV shows can easily fill a lot of disk.

(which reminds me - anyone have a howto on how to build a cheap and reliable Linux NAS?)

Re:1020 Petabytes?

[jackbird]

You're over in all your assumptions:

Most Hollywood movies using digital negatives are burned to film at 2k (~2000 x 1250, varying slightly based on aspect ratio). Postproduction people might use 4k or 6k frames while working with the footage, but that's downsampled on output most of the time. Note that this is not terribly different from the size of a 1080p24 HD frame.

Movies are also generally scanned, mastered, and encoded in 10-bit or 12-bit per channel log. 16 bpc really isn't necessary, as at the top of the brightness range there's not a lot of subtlety in the colors anyhow. Again, post-production people might work with 32-bpc float internally, but there's no reason to have that much color definition in the output.

Also, movies are shot at 24 fps, not 25 (which is the PAL framerate). I'll spot you the 120 minute assumption to keep my answer in the same order of magnitude as yours, but it really should be 90 minutes.

So for most movies, (assuming uncompressed 1.75:1 2k frames):

framedimensions = 2048 x 1170
framebytes = framedimensions x 4.5
moviebytes = framebytes x 24 x 60 x 120

moviebytes / 10^12 ~= 1.8

Re:1020 Petabytes?

[Anonymous Coward]

Yeah especially since on excitebike, despite what the menu selection says, you can't save shiat.

Re:1020 Petabytes?

[Surt]

Your estimate is a bit off. People are already working on screens with horizontal pixel counts in the range of 8k. Let's assume we don't go any higher than that, and use a 16:9 format, so that's 8192x4608

We're also fairly likely to see 8 Byte/pixel screens, as 64 bit/pixel is an increasingly popular format.

60 frame progressive displays are also quite popular (already), so let's guess that we'll see recording at that speed not far into the future.

And a lot of movies run up to about 2.5 hours, so let's give 150 minutes to cover most movies without having to go to a second disk.

8192 x 4608 x 8 * 60 * 60 * 150 = 163074539520000 = 151875 GB = ~148 TB

I guess a 1020 Petabyte drive will hold a decent number of these.

Re:1020 Petabytes?

[evilviper]

Thus CD-quality is, if not perfect, then good enough that further improvements are ignorable for most people.

The thing about standards is that they can't just be good enough for "most people" they have to be good enough for EVERYONE to get traction. CDs will be around for a while to come, but I can see it being replaced with something better.

CD-quality losslessly-compressed music is around 300MB/hour.

Higher sample-rate, more channels (surround sound) etc. Maybe even include a direct copy of the studio multichannel recording, so people can either listen to the pre-mixed version, or mix their own...

10GB/hour is in the ballpark of what you'd need for the sort of quality a modern cinema can deliver. (and there's no particular reason we couldn't go higher.)

Not just higher, but MUCH higher. Film has great resolution, but 24fps is ridiculously low. Upgrading to 120fps seems an obvious step even now.

Besides that, like DVDs, it's not unreasonable to assume that with every movie you get, you also have to keep many GBs worth of special content and features you don't really want... When there's enough space, expect the dozens of commentary tracks to include video instead of just audio. And in-depth coverage of every aspect of every minute of every film...

And all that's not to mention what will happen to video data-rates when holographic displays become practical, and movies go 3D. (holodecks are much further off)

Re:1020 Petabytes?

[arth1]

In terms of scale, 1 exbibyte and 1 exabyte are completely different, the difference in this kind of mistake could equal several hundred thousand copies of Wikipedia, as opposed to confusing KiB/KB or MiB/MB, which are different, but not earth-vs-sun different.

In the eyes of doctrinaires, this is true. Us others continue to use power-of-two units for data storage, and power-of-ten units for other purposes. This is no more difficult than having "mile" mean different things on land and at sea.

1 kilometer = 1000 meter
1 kilobyte = 1024 bytes
1 kibibyte = 1 pedant

--
*Art

Re:1020 Petabytes?

[mandrakethepenguin]

He can say exabyte without referring to your corp.

Re:1020 Petabytes?

[tomstdenis]

And the percentage of people touting FLAC vs. the # of people doing mixing is?

I agree there are uses for FLAC [and the like]. I just disagree that most people need to use it [and thus require TB storage arrays].

Most people are probably like me, buy a CD or DVD, make a rip for their digital archive [e.g. multimedia box] and play it back in their homes or portables. I seriously doubt most people rip stuff then make a DJ mix, etc...

Tom

Re:1020 Petabytes?

[tomstdenis]

It depends on various things like your speakers/amp, sound card, room acoustics, noise floor, etc.

For most people lame with q=2 and 192kbps joint-stereo is as good as the original CD even with good acoutics and gear. 128kbps never cut it as far as I'm concerned for anything but portables or noisy environments. M4A is just a step up [e.g. better quality at the same bitrate] but even without it MP3 is fine.

The problem I have with storing mass amounts in FLAC is when I want to share or transfer the music it takes forever and you really have to plan ahead. I'd rather hack a stockpile of m4a's and just cp them as required.

Tom

Re:1020 Petabytes?

[Fallingcow]

The reason that most people who rip to flac do it is so that they can FORGET about the original CD.

Need an MP3? Just use the flac. Need a .rm? Use the flac. Want to burn it to CD? Use the flac. Gave the CD to a friend, and they want to make OGGs? They can use your burned CD, without having to boost their bitrate to compensate for the cumulative loss of going CD->MP3->CD->OGG.

With hard drives available in the sizes that they are now, and at their current prices, it's plenty worth it to rip to flac and not have to worry about the original CD at all. 300-350MB is NOTHING. I regularly download individual episodes of TV shows that are that size or larger.

Re:1020 Petabytes?

[tomstdenis]

Yeah, until you hit multi TiB collections then have to sort out how you backup, or transfer RAIDs when the drives are old.

Also, if you just encode it once you don't have to worry about "OMG I NEED A RM FILE!! OH NOES" cuz you already have an mp3 or m4a or whatever. Why would you need the same audio in more than one or two formats?

Tom

Re:1020 Petabytes?

[tomstdenis]

You can fill your drive with a few really really big rips, or be like me and fill it with many many quality rips.

"drives as big as they are" is no excuse unless you only plan on having like 50 albums in your collection...

Tom

Re:1020 Petabytes?

[tomstdenis]

Most people touting FLAC are not editing or mixing. They're just really impressed that they can waste space and pretend they can hear signals below the S/N levels. :-)

Tom

Re:1020 Petabytes?

[Samrobb]

I do however see a need for super computers who need to work with filesystems spanning perhaps hundreds of disks.

Super computers? Once, maybe - not today, and not for the last decade or so. There are a bunch of companies (I'm working for one of them, now) that will quite cheerfully sell you a storage system that spans hundreds of disks. Assuming your OS won't flake out when it sees a 500+ TB volume, you could mount it on your desktop, if you want. There's absolutely no need to conflate processing power (super computers) with storage capability (NAS, SAN, etc.)

Re:1020 Petabytes?

[Shawn is an Asshole]

Also, I store my audio as flac, not because I can hear the difference between 256 kbps M4A, but because i lose/scratch the CD, I still want to have a way to get a perfect copy back.
I store my music as mp3 encoded with LAME's "standard" preset. Most of the files are ~224 bit and sound excellent. The file size is much smaller than a flac, which is important when your music collections gets pretty large.

For losing cds, though, you should try what I do as it saves lots of space. When you buy a cd, rip it and make copy directly from the wavs. Take the original and store it somewhere safe. Only carry around the copy. That way if you lose it or it gets scratched, you still have the original to make another copy from.

Re:1020 Petabytes?

[Cygnus78]

And do not forget the future-future where movies store 3D information.

Re:1020 Petabytes?

[hankwang]

Thus CD-quality is, if not perfect, then good enough that further improvements are ignorable for most people. CD-quality losslessly-compressed music is around 300MB/hour.

I agree that more than 44 kHz/16 bits doesn't make much sense for music playback, but I could imagine an upgrade from stereo to 40-channel surround sound that enables a full immersion into a three-dimensional sound field. Of course, most people wouldn't want to mount 40 loudspeakers in their living room. :)

Re:1020 Petabytes?

[hankwang]

When there's enough space, expect the dozens of commentary tracks to include video instead of just audio.

As a movie producer, you will still need to pay a human or a team of humans to do the additional recording and editing. I don't expect the producers to increase their investments in the future uxhd-dvd with a factor 1000 just to fill up the available space.

Re:1020 Petabytes?

[dreamlax]

You're meant to divide by 2^40, not 10^12, it yields a very different result on larger numbers . . . or unless you intended metric TB and not TiB (tebibytes).

Re:1020 Petabytes?

[John Whitley]

Our ears are only capable of hearing up to about 20Khz (less than that for most people) and 16-bit samplings are enough that most people cannot hear the difference with anything more. Thus CD-quality is, if not perfect, then good enough that further improvements are ignorable for most people.

And before anyone steps in and notes that there's audio hardware and software that deals in higher sampling rates (e.g. 88.2kHz up to 192kHz) and higher dynamic ranges (e.g. 24-bit vs. 16-bit fixed, or 32-bit float)... these rates are targeted at quality improvements during the production process, not so much at improved quality of distributed music.

Re:1020 Petabytes?

[entrylevel]

Did you miss that I was explaining a pun?

You just go right ahead using the same word to mean different things and calling it pedantry. I'll call it a pun when it's funny, and a crime against humanity when you are actualling making important calculations based on exabytes without a care in the world whether it's base-2 or base-10.

Re:1020 Petabytes?

[Alan]

I think by that point we'll be digitizing our pron *people*, not pictures and storing them..... think holodeck.

Re:1020 Petabytes?

[Anonymous Coward]

He obviously meant TB, since that is what he wrote, if he meant TiB he would have written TiB.

Nitpicking is bad, but you take it to a new level.
You are picking on somebody who does it right and you tell him he should do it the wrong way.

Please respond with the same determination every time you see somebody use tera- when he really means tebi-.

Re:1020 Petabytes?

[gringer]

What about 3D content? Assuming your calculations are correct, add another 2000 for z resolution, and you're getting to around 10PB for a 2 hour movie.

I'd expect that to fill up such a hard drive fairly fast...

Re:1020 Petabytes?

[Angstroem]

By the time people have hard drives this huge, I bet there will be even better lossless compression algorithms available for video and audio data. I don't know why someone would want to store huge amounts of A/V data WITHOUT using some sort of compression. It just becomes silly after a certain point.

I don't now why I should bother with *lossless* compression especially on streaming media. That will, if ever, give compression rates of 25-50%, taking a buttload of computation power to compress. Lossless compression works only on data with low levels of entropy such as text or stupid zero-filled formats coming from Redmond.

All what lossless compression can do is finding a more suitable alphabet to code this very data into, and identifying identical/repeating regions so that only "repeat this!" is stored instead of the actual data.

There was a time when even 25% compression rate was a great thing cause it made things fit on a 170kB/360kB/720kB/1.44MB disk drive, but today it's just not worth the hassle anymore -- only for very big data, which usually are multimedia files and therefore a very bad application for lossless compression algorithms.

Do you care whether a file has 2MB or 1MB these days? I don't. It takes a second longer to download, but apart from that you don't feel the file size anymore. You do care, though, if a movie can be shrunk from 4GB down to 700MB -- cause then it fits on a CD, cause then it takes just a fraction of time to download, to burn, plus 3.3GB is a difference you still feel on today's harddrives.

As for lossy compression, there is always a catch. I'm one of the people out there who are able to actually hear MP3 artifacts. I also easily spot DCT-related artifacts. Besides, lossy-compressed data is harder to postprocess. (If you want to hear some really strange effect, do a 128kbit-MP3-coding of Art of Noise's "Moments in Love" and feed it through a Hush surround sound processor. However, these side-effects are not always that pleasant/interesting.)

So, to make it short: you don't want to archive in a lossy format, cause when the newer, better format appears, you transcoding makes no sense anymore cause you're stuck with the limitations of your original format. Whatever bad that format did to your data will remain. You can't "repair" lossy-compressed data. And no, low-pass-filtering ("blurring" for you video guys) is no option.

In a couple of years we all are networked with gigabit fibers, we have several Terabytes home storage. Then we just don't care about these 3.3GB as we today don't care whether a file is 1 or 2MB in size. By then we probably care that a HD++ video takes just 50GB instead of 300GB...

Re:1020 Petabytes?

[springbox]

I know the point was that with larger hard drives was that we can store larger amounts of data. In this case, we're talking about storing images and audio without compression so they're archived perfectly. Lossless compression does make a difference and conserves space.

You might already know about these but: FLAC for audio (works great), ffvhuff (from ffmpeg) for video in the YUV colorspace. Those are the two I can think of that currently work pretty well.

Especailly with ffvhuff (a modified Huffyuv) for a 704x480 stream at 24 minutes, I will get a file that's about 14GiB (results will vary, of course.) The uncompressed size of that would be (using a YV12 colorspace at 29.97 FPS, that's 1440 seconds or about 43158 frames) 506880 bytes/frame * 43158 frames = 21875927040 bytes = ~20.37GiB. So, no, you won't be putting that onto a single CD, but for lossless compression of video, that's pretty significant. Even more space is wasted as the total number of frames and resolution increases without some form of compression.

Re:1020 Petabytes?

[springbox]

Oops, wrong numbers. 34 minutes of video with lossless compression (704x480, 29.97 FPS, still YV12 with ffvhuff): almost 14GiB (results will vary.) Same thing without compression: 506880 bytes/frame * 61140 frames = 30990643200 bytes = ~28.86GiB. Quite a difference. Didn't get this right the first time because I was in a hurry.

Re:1020 Petabytes?

[r3m0t]

Yes, but will we be able to conveniently carry around that much data on small cards, send it over a network quickly, etc?

If not, then compression will still have its uses.

Re:1020 Petabytes?

[Nutria]

By now you don't even now what to do with 1024PB

I can barely fill a 250GB drive, and most of it is taken up by barely-touched pr0n.

The people who use massive disks now and in the future will be businesses and government agencies, if for no other reason than bureaucracies like to keep records.

Re:1020 Petabytes?

[Nutria]

as 64 bit/pixel is an increasingly popular format.

Where?

But even if it were 32-bit pixels, that would mean the movie would "only" be 74TB.

Re:1020 Petabytes?

[Eivind]

And as long as they don't there's no value to them of having the 40 channels required. The overhead is simply not going to be worth it.

OK, fine, I too can perhaps see an upgrade to say 5.1 sound becoming common, but even that is "only" a factor of 5 larger.

Besides, I was assuming lossless compression. That is, frankly, overkill for end-users. A CD (uncompressed) is on the order of 10MB/minute. With modern codecs you can compress that down to 320kbps or even 256kbps and the quality is still such that even the critical listener, in a silent environment, with excellent components, is unlikely to hear the difference.

Re:1020 Petabytes?

[Eivind]

No. That is completely wrong. The landscape is littered with standards that are merely good-enough. If 99% of all users see no reason to upgrade from one standard to another, teorethically sligthly superior one that cover the wishes of the last 1%, then the new "standard" will remain marginal and irrelevant.

Witness SACD. There's no doubt whatsoever that it is "better" than standard audio-cd. But the advantages are irrelevant to 99% of all listeners. I predict it'll *never* become dominant over CD, despite being technically superior. The same may very well be true for Ogg versus Mp3. (If Ogg *does* win, it'll likely be because of political issues more than because of any property of the codecs used or flexibility of the file-format)

I too can see CDs being replaced with "something better". There are lots of disadvantages of CDs that *are* relevant to normal people, if a new format comes along that fixes a significant numer of those, without introducing new worse problems, there's no reason it couldn't take over. Problems include:

• Physically fragile. Most people have scratched CDs.
• Small capacity. 70 minutes of music is tiny in an age where people are used to 1000 hours in ipod-size thingie.
• Requires moving-parts to read, problematic for mobile usage.
• Too physically large. Problematic for mobile usage. Leads to shock-sensitivity and increased battery-consumption.
• Inadequate support for additional content beyond music. Not even *songtitles* and *artistnames* are universally coded in a way that all players understand. (yes I know about CDTEXT)

Witness how SACD fails to solve basically all of these real-world problems. In contrast, for most people, the 16-bit sampling-depth and the frequency-limitation of 22Khz are *NOT* significant problems, nor is the 2-channel limitation. The fraction of people that *do* consider these three serious problems is shown in the uptake of SACD-players versus mp3-players. (the latter solves most of my problemlist, but actually typically *degrade* soundquality)

The logical choice that solves all of these and then som problems is the non-physical downloaded unencumbered mp3 or ogg-file. People love those. Even enough to use them in millions despite a very limited array of legal ways of aquiring them. Hell, most people I know that *do* still buy CDs do it explicitly for the purpose of converting them into files for their mp3-players.

I agree with you that it's quite reasonable to expect video-bitrates significantly higher than 10GB/hour in the next few years.

Re:1020 Petabytes?

[evilviper]

If 99% of all users see no reason to upgrade from one standard to another, teorethically sligthly superior one that cover the wishes of the last 1%, then the new "standard" will remain marginal and irrelevant.

I wouldn't just call 99% "most", I'd call that an overwhelming majority, or "almost everyone".

"Most" is a much lower percentage... And when you have several millions people unwilling to accept the lower standard, the higher standard eventually gets accepted, baring major limitations.

I predict it'll *never* become dominant over CD, despite being technically superior.

It's technical improvements are hampered by it's natural and artifical limitations.

Hell, most people I know that *do* still buy CDs do it explicitly for the purpose of converting them into files for their mp3-players.

Actually, I'd say audio CDs are a very good example of my point... Though many people are willing to buy a few tracks of lossy compressed AAC files, their much poorer quality will prevent them from eliminating CDs. The lossy compression, or the bitrates, will need to be improved significantly before it's suitable enough for almost everyone to be happy with them. And until that happens, the standard will remain decades-old CD technology.

Re:1020 Petabytes?

[Eivind]

I am pretty confident you are just plain wrong.

Of the people I know that refuse to buy (or in practice don't buy) anything from iTunes, I've only *once* heard quality-concerns even mentioned. What I *have* heard, repeatedly, is:

• Poor value. $0.99 is *more* expensive than buying the CD for older music. *much* more expensive than buying used CDs. People expect to see *some* of the savings of digital distribution.
• Doesn't work on all player-devices, forcing you to essentially use iTunes and iPod. (I know this ain't strictly true, buy the general impression is in this direction)
• No resale-value whatsoever. Nor is it trivial to do simple innocent stuff that people are used to, like for example give music that you are tired of to your friend that isn't. (again, I know this ain't strictly true, nevertheless, people don't feel confident they'll be able to do these things)
• AAC ? What's an AAC ? I know mp3s, may even have heard of Ogg. But AAc ? What's that ? (simply lack of brand-recognition, even the iPod is refered to as an "mp3-player" not an "aac-player")
• Some may have heard arguments from their nerdier friends involving the word DRM and sounding negative. (if they understood the arguments is a different matter, but they're aware that there's something there that many of the "experts" say will bite you)
• Many (especially older people) are still not ready for non-physical distribution. This will change over time, but presently it's so. I can't see my grandmother buy iTunes-songs for the birthday of any of her grandchildren, she frequently buys CDs if such are on the wishlist though.
• Some (the more technologically clueful) *do* know what DRM is and what it does and just flat out refuse. "You can have my unencumbered digital music-files when you pry them from my cold hands" sort of people. These aren't many, but there's been a *sharp* increase the last 2-3 years.
• It's a monopoly. In practice, going for an AAC-library means marrying apple. Apple players. Apple Software. Apple-approved operating-systems. If you instead go with mp3, you have literally hundreds of players on a much more even playing-field.
• Last, there's also a few that consider the current music-business a near-mafia, and see it as *more* unethical to support them financially than it is to ignore copyrigth-law.

Notice how none of this has anything to do with sound-quality. If sound-quality was the huge selector you make it out to be, 128Kbps mp3 would *never* have been the stellar success it actually was (and is)

Re:1020 Petabytes?

[evilviper]

Notice how none of this has anything to do with sound-quality.

The fact that you failed to mention sound quality... or more specifically that the people you heard from failed to mention it... doesn't mean it's not an issue.

If sound-quality was the huge selector you make it out to be, 128Kbps mp3 would *never* have been the stellar success it actually was (and is)

MP3s are a bit like copying songs off the radio. People accepted the quality because they were getting them for free. Unencumbered 128k MP3s didn't replaced CDs, nor will they.

1020 Petabytes?

[BenBenBen]

1 Exabyte!

Not to be confused with Excitebike, which is something entirely different.

Thinking of switching

[Anonymous Coward]

I was thinking of switching from reiser3 to ext4, but before I do, I just want to know... If enough people switch, is the developer of ext4 going to go out and murder his wife?

fsck

[tttonyyy]

EXT4-fs warning (device sdb1): ext4_journal_start_sb: Detected tasteless ReiserFS jokes - hahahaha!

Re:FSCK

[walt-sjc]

Yeah - it's brutal to fsck a 1T array, or even just mkfs a partition that large with ext3. XFS and JFS are MUCH MUCH faster. So besides just huge filesystems, what else is better about ext4?

Re:FSCK

[Abcd1234]

Well, given ext4 (like ext3) is journalled, I can't see why you'd ever perform a full fsck (unless you're paranoid).

Re:FSCK

[rubycodez]

sometimes the extX force you to do an fsck before it'll mount, let's face it, it's not as sophisticated as other more advanced fs

Re:FSCK

[Anonymous Coward]

You can change how long it goes between checks or even disable it, although it is advised you don't.

Re:FSCK

[kwark]

'man tune2fs' is your friend

spoiler: see -c and -i
workaround: see 'man mke2fs'

Re:FSCK

[ncc74656]

Deleting large files (1 GB or larger) also takes an obnoxiously long time with ext3. The same operation with ReiserFS or JFS completes right away. (I've never used XFS, so I can't say anything about its performance one way or the other.) I'd think MythTV users (like me) would be interested in how long ext4 will take to delete a large file.

Re:FSCK

[rubycodez]

I'm talking about when something bad happens and it is REQUIRED before mounting is possible.

Re:FSCK

[rubycodez]

no, that's not the issue. the fact is there are times when extX is in an inconsistent state, and if you lose power then you WILL have to fsck before mounting is even possible (assuming that the filesytem is recoverable, and there is a finite but small possibility that it is not). If you work with a large enough group of linux servers running ext3 you'll see such problems. rare but not zero probability.

Re:FSCK

[makomk]

Deleting large files (1 GB or larger) also takes an obnoxiously long time with ext3. The same operation with ReiserFS or JFS completes right away. (I've never used XFS, so I can't say anything about its performance one way or the other.) I'd think MythTV users (like me) would be interested in how long ext4 will take to delete a large file.

In fact, I think MythTV recently introduced incremental file deletion (slowly truncating the file more and more before finally deleting it) in order to avoid some kind of possible IO starvation problem if you deleted a file whilst recording. (Of course, since I followed the recommendations and put my recording store on JFS, I've got it disabled.)

Re:FSCK

[Anonymous Coward]

christ, you fucking paupers - get a real unix and use vxvm w/ vxfs.

Re:FSCK

[kwark]

Sure, but setting mount count and days between to 0 makes sure you don't have unwanted fscks.

Corruption can happen on any filesystem, in my cases so far always due to some bad hardware (or its setup). I have lost data on XFS and reiser (though it has been many years since I used the latter). I can't remember when and if I lost data on ext3, but that's because I only use it for small, nearly static filesystems (such as /).

If recovery is of the essence I guess the recovery tools are the most important reason to choose a f.s. (second to the frequency of corruption offcourse). I'm no expert but I'm under the impression that ext2/3 excels in that area.

FSCK

[Pegasus]

Wake me up when chunkfs hits the kernel. I don't even want to think about fscking all those petabytes ...

backups?

[Chrisq]

It could make doing backups interesting too.

Re:backups?

[MBGMorden]

Why to people keep thinking we'll be backing up the hard drives of 15-20 years from now with current tape drives? By that time the tape drives (if we even still use tape, which we probably will) will have increased in capacity/performance too, and it won't really big that big of a deal.

Re:backups?

[Anonymous Coward]

thats a fuck load of 360K disks!

Re:backups?

[randomalias]

Honestly it's a right pig.

Most of the databases I work on are 10-50Tb.

Initially we built 3Gb filesystems - we couldn't back then up, the sequential file pointer in HPUX can only address 2Tb, which meant we couldn't copy the whole filesystem to tape. I had to rebuild with max 2Tb filesystem.

Then, Veritas Netbackup can only parallelise backups in different directory trees so I was taking ages to perform a full back up - 18 - 19 hours (bit of a bugger in a 24hr backup window).

We don't do incremental backups because restoring takes an age with incremental. Even with fulls, if the backup takes 10 hours, the restore takes 15 - its to do with how different simultaneous backups are stiped across the tape. Therefore we always do fulls.

Eventually (and fortunately during test phase) we rebuild the filesystems with 500Gb limits.

Which makes growing the database really sticky because we have to allocate new filesystems, rather than grow the ones we've got.

There's a major move towards disk-based archival storage, but tape's reliable and extremely portable so its generally what we end up using.

Re:backups?

[suggsjc]

I don't understand all of the bitching about "how are we going to backup ALL of this data up..."

Filesystems aren't a problem, bigger hard drives aren't a problem, in fact there is no problem at all except people that don't understand the concept of what doing a "backup" actually means.

When we had 1024K drives it would require 1024K to do a FULL backup.
When we had 1G drives it would require 1G to do a FULL backup
...
When we had 1TB drives it would require 1TB to do a FULL backup

If you can't notice a trend, then you don't need to be anywhere near my data backup process. It is just plain, SIMPLE math. When we increase our storage capacity by X, we must also increase our backup capacity by X as well (or a percentage of X if we compress our backups). Its like people think that new drives/technology only apply to active storage, but they must rely on last year/decades technology for backups.

Let me repeat it again...there is no problem. Stop bitching!

Re:backups?

[mikeee]

Yes, there is a problem.

First of all, disk drives are advancing faster than tapes.

But the problem is worse than that. Different aspects of disk drives are advancing at different rates. Capacity is increasing faster than interface speed is increasing faster than access speed is increasing faster than block reliability.

Consider an old 500MB drive from the mid-90s; it takes maybe a couple hours to read every block on the drive, and odds are that you won't have bad blocks before the disk dies entirely.

The new 1000GB disks we'll have soon, though... you're pretty much guarenteed some of the blocks will go bad before the disk fails; there are just so many of them. And if you can read at 10MB/second, sustained (pretty respectable, I think?), it literally takes 24 hours just to dump all the data from the drive, never mind back it up to something else.

The whole model of 'back up your fixed media to removable media' is not going to work anymore. It does not scale up to modern hardware. I think the answer will be some wacky combo of raid and file-system version control, but YMMV.

Re:backups?

[Anonymous Coward]

Yay! That's about too much tapes!
Puts the fun back in backing up

Re:backups?

[dhasenan]

USB2.0 allows for 480Mbps, which is 55MBps. A high-quality SAS drive will offer 3Gbps (375MBps) at introduction, allowing eventually for 750MBps.

That means, well before we have common petabyte storage, a terabyte disk will take about 22 minutes to clone. A petabyte disk will then take about fifteen days to copy. But by the time we commonly have petabyte storage, it'll likely take no more than a day at the outside to clone a petabyte disk.

As for bad blocks, use an internal RAID.

Re:backups?

[suggsjc]

I'll give you that i ranted and oversimplified the situation, but its absurd to complain about technology advancing too fast.

Now, first (for the time being) terabyte storage and certainly petabyte storage is primarily a business (not consumer) requirement/market. In that arena and if you are already going to be paying $$$ for the diskspace, it is almost a no-brainer to be using some sort of RAID. I could just be out of touch with current storage trends, but for *most* systems that I know of they are either fading out or dropping completely tape stoarge in favor of either multiple on-line copies of data in different locations (multiple mirrors) or I know one place that has roughly 250GB of active storage that they put onto 500GB drives that use more like tapes. Plug them in, back up, un-plug them, store them.

As this parent points out, the interface speeds are certainly increasing...and by the time that we (consumers) start requiring terabytes and petabytes of storage the transfer speeds will increase as well.

Again, I don't see this as a problem. If your system is currently at X storage requirements just because technology allows for X^N storage capability doesn't change that you still only use/need X. So, what works for you now will continue to work. The new capabilities will only affect you once you start *needing* those capacities. Then, and only then do you need to start looking at what your backup process is/will be. If it ain't broke, don't fix it. Oh yeah, and still stop bitching.

Re:backups?

[mikeee]

No.

That's the bus speed; the disk can't actually sustain I/O at anything approaching those rates. (If you're doing all reads serviced out of the little internal cache on the HDD, sure, but try writing a few hundred megabytes of data and see if you get rates approaching the interface bandwidth.)

Re:backups?

[WuphonsReach]

Neither SAS or SATA-II drives will come close to filling a 3Gbps channel in the forseeable future. Not until perpendicular storage boosts their data density by about 4x (and possibly even longer).

Modern 750GB SATA-II drives top out at around 75-80MB/s, with 40-50MB/s as a more realistic number across the entire disk. Assuming that we could spin that disk at 15k, that's still only 150-160MB/s peak or 80-100MB/s average.

Maybe once the 2TB 7200rpm drives or the 1TB 10k or the 600GB 15k drives arrive...

Re:backups?

[swillden]

And if you can read at 10MB/second, sustained (pretty respectable, I think?)

A little low, but in the ballpark. Most drives these days can sustain ~50MBps, and even laptop drives can usually manage ~25MBps. High-performance drives are probably closing in on 100MBps. Doesn't affect your point, though.

Re:backups?

[Nutria]

Initially we built 3Gb filesystems - we couldn't back then up, the sequential file pointer in HPUX can only address 2Tb, which meant we couldn't copy the whole filesystem to tape. I had to rebuild with max 2Tb filesystem.

Then, Veritas Netbackup can only parallelise backups in different directory trees so I was taking ages to perform a full back up - 18 - 19 hours (bit of a bugger in a 24hr backup window).

This has always puzzled and concerned me about Unix.

Mainframes and minicomputer OSs like OpenVMS have had parallel database backups for years.

10 CPUs, 50 disk packs, and 10 tape drives each on a separate SCSI card. Whoosh, you pump your database out there in parallel using a vendor utility.

Which makes growing the database really sticky because we have to allocate new filesystems, rather than grow the ones we've got.

Why grow your existing filesystem? That just puts all your eggs in one basket.

1020 petas

[blantonl]

>> 1020 petabytes

My porn collection will now be complete.

Re:1020 petas

[Zaatxe]

>> 1020 petabytes
My porn collection will now be complete.

Liar... there is no such thing as a complete porn collection!

Re:1020 petas

[tonigonenstein]

My porn collection will now be complete

In fact, 315 TB should be enough for anybody.

Here is why:
Suppose you want to watch porn 24 hours a day from the age of 15 till 75. Thats 60 years = 60 * 365.25 * 24 * 60 * 60 s = 1.89 * 10^9 s
A DivX is around 600 MB / hour = 600 * 1000000 / (60 * 60) = 1.67 * 10^5 B/s
So for your lifetime porn collection you need 1.89 * 1.67 * 10^14 B = 315 TB.

Re:1020 petas

[Dr_Barnowl]

You're neglecting the possibility for VR formats.

I'm sure full-immersion sensory data will require far more data bandwidth.

Re:1020 petas

[Wiz]

Ah, don't you mean the index to your porn collection with be complete? ;)

Re:1020 petas

[nuzak]

> In fact, 315 TB should be enough for anybody.

No, 2^128 bits should be enough for anybody. Everybody, in fact.

Re:1020 petas

[ArwynH]

What SD Porn?! With artifacts?! True Pron addicts will only settle for 1080p HQ porn!

Mind you, even that won't take up 1024 PB... maybe if you used raw video with no compression...

Re:1020 petas

[Anonymous Coward]

Talking about Godel's porn incompleteness theorem. ;P

Re:1020 petas

[joto]

You forgot something. You need to skip the boring parts. Assuming you only watch about 10% of the porn, and skip past the rest, you need 3150TB. And of course, you may want something a little higher definition than the typical 2006 DivX. In twenty years time, I'll probably want a fully immersive experience, with a tactile VR suit. This will need much higher bandwidth (or is it called "wider bandwidth"? ;-)

Re:1020 petas

[Anonymous Coward]

Given the average porn actress' lack of physical perfection, I'm guessing the more visually satisfying experience would come from watching SD than watching HD and seeing every skin imperfection, every bit of cellulitis, etc...

Re:1020 petas

[Chineseyes]

Yes but what about those of us who have our walls covered in 30 inch monitors with a different porn flick on every monitor?? Surely we'd need more than 315TB.

performance

[bioglaze]

How does ext4 perform when compared to, say, reiserfs 3.6 or 4? What new features there are?

Re:performance

[Anonymous Coward]

ReiserFS just kills it.

Oh god, I'm sorry. So, so sorry. I just had to do it... oh god why did I go through with it?

Re:performance

[Anonymous Coward]

maybe just this time, your wife may have the better answer...

Re:performance

[wyohman]

A lot like beta software.

hardware.slashdot.org?

[Anonymous Coward]

I thought file systems were typically constructed in, for and with software.

Re:hardware.slashdot.org?

[creepynut]

Not ext4, it's too cool for software.

Ext4 Features

[sacx13]

I'm happy to hear this. I just waiting to see the new features ... Proposed features of ext4 are delayed allocation, higher resolution timestamps, and support for larger volumes and file sizes. Regards

Advantages over XFS, for example.

[Tribbin]

My question is why they don't mention why it is better to use ext4 then XFS.

XFS can do 9 exabytes (exabyte = 1024 petabytes).

They mention that ext4 is not faster than other filesystems.

Ofcourse people can do whatever they want, but why not spend their time making XFS easily resizable for example?

Re:Advantages over XFS, for example.

[Zarhan]

Ofcourse people can do whatever they want, but why not spend their time making XFS easily resizable for example?

I would also appreciate block journaling for XFS.

Re:Advantages over XFS, for example.

[orzetto]

(exabyte = 1024 petabytes).

You mean, "exbibyte = 1024 pebibyte". An exabyte is exactly 1000 petabyte.

I used to think this was just fussy, but I am quite tired of guessing which system producers of hard disks/CDROMs/DVD+±×RWs use to figure out if that is enough for my needs.

Re:Advantages over XFS, for example.

[Anonymous Coward]

I'm an XFS fan and I use it and have for years. I've also shipped products that used XFS on Linux.

It has 2 problems that I can see. 1) SGI isn't really funding it's development anymore. That's not a terrible problem as filesystems should kind of reach a "done state" While it is incredibly safe, from my experience, there are still regular bugs and patches that get applied. It hasn't been abandoned but I'm not sure what the level of committment is to keep it working correctly and I doubt we'll see any major changes to it. It's built to last quite a while though so I'm not sure what those changes would be. 2) The code base is fairly complex and that kind of keeps people out. By contrast, Ext(2,3,4) has been completely done from the bottom up by guys that are actively working on Linux still. If you're a budding filesystem guy, go have a look at the codes for the various filesystems. Ext2,3,4 is fairly accessable and looks quite a bit like what you see in OS textbooks. I don't think XFS is going to die or anything like that but it's just not clear what the future holds. OCFS2 is getting a lot of attention I could also see GFS2 become a standard FS at somepoint.

If ext4 uses extents and has more addressability, I'm not sure what other short-commings it would have. Performance is comparable, it doens't get blown out of the water at anything while it may not be the fastest at some things. It supports extended attributes for SELinux and quota and what have you, I'm somewhat surprised at how Reiserfs doesn't do this as well yet. It's also incredibly reliable. There is a bit of hype in filesystems regarding ZFS and Reiserfs4 but aside from log structuring and journaling, there has not really been too much innovation in filesystems. ZFS added block checksums which is nice but other than that it's just kind of rehashing the same shit with more addressability. RFS4 adds arbitrary plugins but you'll never dynamically change the semantics of a filesystem with a plugin without reformatting. The end user experience doesn't change that much and hasn't changed that much.

Re:Advantages over XFS, for example.

[Spirilis]

Can't you grow an XFS online already? In fact, that's the only way you can grow an XFS filesystem... with it mounted.

Re:Advantages over XFS, for example.

[TheOrquithVagrant]

I think the main advantage over XFS is that the codebase won't make kernel developers vomit.

From a user POV, XFS is my favorite FS, and I've had no bad experiences with it whatsoever. However, from what I've heard about the code, I can fully understand why it's not "popular" among kernel devs, and why none of the enterprise distros favor it (because doing so would inevitably force _their_ kernel hackers to try to debug XFS-related errors).
I'm hoping now that SGI is officially "retiring" IRIX, XFS will be cleaned up and properly ported to Linux, rather than be essentially a non-native module with a "portability layer" around it, which from what I hear is the main reason the code is so ugly to work with.

Re:Advantages over XFS, for example.

[Anonymous Coward]

Remember that XFS was written by SGI to handle big files: video streams, audio streams, and high-res pictures. It's heavily optimized for it. Last time I used XFS it was slooooow when handling many small files.

Re:Advantages over XFS, for example.

[Anonymous Coward]

Why XFS? ZFS has even bigger limits, and more features. Why not help getting ZFS running under FUSE?

Re:Advantages over XFS, for example.

[Anonymous Coward]

You mean, "exbibyte = 1024 pebibyte". An exabyte is exactly 1000 petabyte.

Argh! Wash your mouth out with soap and water.

Re:Advantages over XFS, for example.

[Firehed]

Well not only that, but on this scale you're talking a difference of FOUR THOUSAND TEBIBYTES (well, 24,000 in this example, 4000 in the earlier comments saying 1020PB=1EB). To put that in perspective, that's enough storage to make the oceans run dry because of the boost in lubricant production.

Re:Advantages over XFS, for example.

[jrumney]

I am quite tired of guessing which system producers of hard disks/CDROMs/DVD+±×RWs use to figure out if that is enough for my needs.

It really isn't that difficult to "guess". System producers use whichever units make their product look better.

Re:Advantages over XFS, for example.

[Anonymous Coward]

If that were the case, 700 "MB" CD-Rs would be marketed as 735 MB, and 8 "GB" iPods as 8.5 GB.

Gee....

[slightcrazed]

....now I can make use of those 1400 1 terrabyte drives that I have sitting in my basement. I knew they'd be useful for something someday. How long until the RIAA sues because Ext4 will allow for even MORE music to be illegally downloaded and stored?

Re:Gee....

[kimvette]

Kazaa is not available for Linux and runs poorly under wine. I don't think you have to worry too much about that.

Wow

[knipknap]

enabling Linux to support storage volumes up to 1020 petabytes in size

Now, is there anybody who still believes that porn does not drive innovation?

Re:Wow

[porkmusket]

I think you misread it... it's petabytes, not paedobytes.

What kind of pr0n

[Anonymous Coward]

do you need 1020 pedobytes for? ;)

also in linus tree

[alonso]

From lwn current issue(you have to subscribe for the full article ;):

Also merged is the developmental ext4 filesystem, which includes a number of enhancements, including support for extents and 48-bit block numbers. See the ext4 documentation file if you are interested in playing with ext4 (and have good backups).

And how...

[porkchop_d_clown]

Will we back all this data up?

I'm honestly more interested in someone coming up with cheap, long term archival storage. Hard disks have gone so far past our ability to archive information it's beyond comprehension.

Re:And how...

[CastrTroy]

Well, you could just back everything up on other hard drives, instead of using tapes or whatever else you are used to. Sure hard drives crash, but not when you just write the backup to them and then put them on a shelf. I don't think they'd be any less dependable than tapes.

Re:And how...

[Eivind]

I don't see the problem.

Harddiscs are, as you say currently superior to other storage-technologies.

So, you make backups to hard-disc then. Simple. Quick. Affordable.

Yes, the lifetime is limited, so you should make sure to have atleast 2 independent backups (that's true for any media, all media can go bad) and you should change them every 3 years or so.

The thing is, capacity is growing so rapidly, that in 3 years, what is now a hard-disc full of backup will be a hard-disc 10% full of backup.

I need a single hard-disc now to backup all my data. I have two, and store the two in physically separate buildings to ensure against fire, robbery or similar.

I expect to be able to make do with 2 hard-discs for the rest of my life. By the time the current 2 are old, I'll be able to buy 2 new ones with 10 times the capacity for the same price.

Where is the problem ?

Miss messing around with unreliable, slow, crappy, noisy, sensitive tapedrives all that much ?

Re:And how...

[Zaatxe]

Will we back all this data up?

20 years ago you would ask the same thing about 500 MB hard disks. And don't forget the article (or at least the sumary, as a good slashdotter I don't RTFA) says about the file system capacity, not the real capacity of hard disks. Petabytes hard disks are something for the future, not for now.

Re:And how...

[holistah]

haha, at the top of my page their is an ad for a 1 exabyte automated tape-backup system... it's uncanny how relevant the ads are on /.

Re:And how...

[jdavidb]

You buy ten of them and do a super-redundant RAID.

Re:And how...

[ObsessiveMathsFreak]

I'm honestly more interested in someone coming up with cheap, long term archival storage. Hard disks have gone so far past our ability to archive information it's beyond comprehension.

Once, I was like you. Backupless. Then I went down to by local PC store and happened by a 320GB external USB HDD for around 200. Needless to say my current backup woes have been solved. If I ever get around to building that terabyte fileserver, all I need do is lash together four of these little beauties and use something like DAR, to span the backup over multiple disks. Or in lieu of that mount all the disks as one filesystem. It may well be painful, but at least it will be cheap.

Re:And how...

[eln]

RAID won't help when you accidentally delete half of your database. Tape backups protect against more than just hardware failure.

Re:And how...

[Abcd1234]

Indeed. Just ask my wife, who accidentally did an "rm -rf *" in her mirrored home directory...

Re:And how...

[MrNaz]

What's wrong with RAID5 ? Linux does it in software now so you can't say controllers are expensive.

Re:And how...

[networkBoy]

Dude, who had 500MB drives in 1986?
I saw 50 meg drives that were 18" platters and scarry big.

I'd think your point still is valid though...
Anyone in 1986 faced with the prospect of backing up 500Mb would likely turn to the intern and say: "Start memorizing".

BTW, in 1986 if someone came up to you and told you that there would be hard drives in 20 years that are 750GB and were faster than your current system memory what would you have said to them?
-nB

Re:And how...

[Amazing Quantum Man]

You: Ha! My RAID system can handle hardware failures!

Luser: Dude, uh.... um.... I think I umm.... accidentally deleted the database.

You: Doh!

Re:And how...

[Zaatxe]

Dude, who had 500MB drives in 1986?
Hmmm... aliens? Well, who has 1020 petabytes drives today?

BTW, in 1986 if someone came up to you and told you that there would be hard drives in 20 years that are 750GB and were faster than your current system memory what would you have said to them?
I'd say "to the time machine, Robin!"

Re:And how...

[MrNaz]

I have that covered. Luser: Can I have root access to anything remotely important? Me: No.

Re:And how...

[TheRaven64]

It does with ZFS or with FreeBSD's UFS2. Most relatively modern filesystems support snapshots, so all you need to do is set a cron job to regularly mount snapshots and then you can roll things back easily. ZFS is nicer than UFS2 for this, since it allows you to snapshot any part of the directory hierarchy at any point easily, and the copy-on-write mechanism allows for much easier updates.

If it weren't for the GPL, Linux could just port ZFS over and everyone would be happy.

Re:And how...

[Zapman]

What are your needs?

There are 3 kinds of 'backups' out there. Disaster Recovery, User Protection, and Archival.

Disaster Recovery provides protection for 'this building isn't here anymore'. Requires an 'off site' copy of all the data, that's kept up to date regularly. Ususally this is a combination of 'full', 'incremental' and maybe 'differential' backups. Or it could be a full, real time, copy of the data (if you have the $$$$$$ to support this). Direct backup to (cheap) disks works well for this, or tapes (which are easier to take off site).

User Protection provides protection for 'uh... I dropped this table from the database last week, but this monthly process is still expecting it... can we get it recovered?' type situations. This requires regular backups of your data, and the copies are kept for a relativly significant piece of time (6 months? a year?). Tapes are almost always used for this.

Archive backups are what's required when, say, the IRS says 'keep payroll tax data for 7 years'. Depending on the size of the data, CDs/DVDs may be a good choice, or often tape.

A real backup strategy has to deal with all these issues, while keeping the cost down. Real backups cost a LOT of real money.

--Jason

Re:And how...

[Ponga]

Exactly. Or if your box catches on fire. Off-site backups, my friends. Off, site, backups.

Re:And how...

[Cyno]

We'll use a ZFS backup server that's hooked up to a few racks of fully automated dvd burners.

640k

[malzraa]

Who needs ext4? I'm perfectly satisfied with my 640k, and so should you!

Re:640k

[Zaatxe]

Who needs ext4? I'm perfectly satisfied with my 640k, and so should you!

Have you heard of porn? Yeah, I didn't think so...

Re:640k

[Anonymous Coward]

You use a filesystem for your main memory? Yow!

Re:640k

[James McGuigan]

Back in my day, we didn't have these fancy jpegs and xvid, we had to make do with ASCII porn, you insensitive sod.

Re:640k

[freeweed]

1020 Petabytes should be enough for anybody...

Re:640k

[MrNaz]

"Clod" Peppe, it's "Clod".

Re:640k

[Zaatxe]

I was a little luckier... In my college days we had 320x200 gif porn. And we were happy! Now get out of my lawn!

extents

[slapout]

"and to write files in 'extents,'"

Aright! 1970s mainframe technology, here we come!

Re:extents

[rjamestaylor]

Exactly. How well will EXT4 handle CI/CA splits? What userspace tools will exist to tune VSAM, I mean, EXT4 extents?

With EXT4 having extents we'll finally have the joy of defragmenting a hard drive like Windows people. Yea, progress!

Re:extents

[gstoddart]

Aright! 1970s mainframe technology, here we come!

Given the reputation of 1970's mainframe technology for being bomb-proof, reliable, stable, and useful, I'd say it's about fscking time.

Just because something was a good idea then, doesn't mean it isn't a good idea now.

Cheers

Re:extents

[splutty]

Now if someone would just port RACF to Linux, I'd be back where I started! Whoopy! Builtin CVS in your filesystem, gotta love it :)

Interesting in light of OpenSuSE's decision...

[Saint Aardvark]

to no longer use ReiserFS as its default FS (orig. reported on OSNews.com...don't think I've seen it here yet). I think this came out before the whole Hans Reiser affair, BTW.

SuSE contrasted the ease of upgrading ReiserFS and ExtFS versions:

ReiserFS v3 is a dead end. Hans has been pushing reiser4 for years now and declared Reiser3 in maintenance mode. Any changes that arent bug fixes are met with violent resistance. Reiser4 is not an incremental update and requires a reformat, which is unreasonable for most people.... Ext3 has a clear upgrade path. There is quite a bit of interest in the community in improving ext3, and ext4 is already under development. Like the upgrade path from ext2 to ext3, the path to ext4 is clearly defined. Existing file systems can be updated easily, and new files will be able to take advantage of the new features.

Re:Interesting in light of OpenSuSE's decision...

[D4rk Fx]

Perhaps Hans' wife suggested a Fix to Reiser 3 that wasn't a bug?

Re:Interesting in light of OpenSuSE's decision...

[r3m0t]

"Any changes that arent bug fixes are met with violent resistance."

Yes, it was *very* violent... :)

Experimental??

[scsirob]

What I don't onderstand is that this is merged into the 2.6 kernel tree today. What has happened to the concept of -stable (2.6) and -experimental (2.7) trees? This would be aperfect opportunity to open the next experimental branche..

Re:Experimental??

[andrewd18]

It's not in the main 2.6.x kernels, it's in the -mm kernel. Andrew Morton's kernel patchset has always been bleeding edge.

Re:Experimental??

[Slashcrap]

What has happened to the concept of -stable (2.6) and -experimental (2.7) trees?

http://www.lkml.org/

You can use the archive function to go back to about two thousand and fucking four. That way you can answer the question yourself instead of everyone else having to go through this tired old shit yet again.

Re:Experimental??

[Prof.Phreak]

What has happened to the concept of -stable (2.6) and -experimental (2.7) trees?

They're all equally stable now :-)

Re:Experimental??

[camperdave]

Well, this was news to me too, but...

The development model for Linux 2.6 was a significant change from the development model for Linux 2.5. Previously there was a stable branch (2.4) where only relatively minor and safe changes were merged, and an unstable branch (2.5), where bigger changes and cleanups were allowed. This meant that users would always have a well-tested 2.4 version with the latest security and bug fixes to use, though they would have to wait for the features which went into the 2.5 branch. The 2.5 branch was then eventually declared stable and renamed to 2.6. But instead of opening an unstable 2.7 branch, the kernel developers elected to continue putting major changes into the 2.6 "stable" branch. This had the desirable effect of not having to maintain an old stable branch, making new features quickly available, and getting more testing of the latest code.

However, the new 2.6 development model also meant that there were no stable branch for people just wanting security and bug fixes, and not needing the latest features. Fixes were only put into the latest version, so if you wanted a version with all known bugs fixed you would also get all the latest little tested code, and risked that things which had previously worked suddenly broke. A partial fix for this was the previously mentioned fourth version number digit (y in 2.6.x.y), which are series of point releases created by the stable team (Greg Kroah-Hartman, Chris Wright, maybe others). The stable team only released updates for the most recent kernel however, so this did not solve the problem of the missing stable kernel series. Linux distribution vendors, such as Red Hat and Debian, maintain the kernels which ship with their releases, so a solution for some people is to just follow a vendor kernel. Wikipedia

Re:Experimental??

[kasperd]

What I don't onderstand is that this is merged into the 2.6 kernel tree today.

When it is just about merging a new file system, there is not much of a risk. Ext4 will be optional like almost every other file system in Linux. Include it in the kernel if you want to, build it as a module, or don't build it at all. As long as you don't include it in your kernel or don't load the module, it will do no harm to your system. And even if you have it in your kernel, it is unlikely to do any harm unless you actually mount something with it.

The main reason Reiser4 didn't go into the kernel as easy as this was, that Reiser wanted to make changes to the VFS layer to support his new file system. So it was not just about adding a new file system, but changing how the kernel handles file system, something that could bite other users who didn't even include Reiser4 in their build.

Novell Suse prefers Ext3/Ext4 over ReiserFS 3 / 4

[maggard]

Other Reiser issues aside, the SuSE folks at Novell are looking to leave the nearly unsupported reiserfs3 (in maintenance support, which isn't enough for them) and move to ext3 as their default FS. Why? They feel ext3 is a lot more mature & better/wider supported then reiserfs4, is an easier migration, and appreciate that there is a solid roadmap from ext3 to ext4.

Of course this would also be the week that (coincidentally) Andrew Morton gives reiserfs4 the green light for eventual mainline kernel inclusion.

duh

[Achoi77]

Just mirror it over the internet, you silly goose! Bandwidth is cheap(ish)

features isn't everything

[oohshiny]

My question is why they don't mention why it is better to use ext4 then XFS.

Simple: ext4 is a backwards compatible, evolutionary change from ext3, while XFS is a different file system and codebase. XFS doesn't offer sufficient advantages to overcome that built-in advantage of ext4 (after all, neither XFS nor ReiserFS managed to succeed even against ext3).

Re:features isn't everything

[kwark]

Well, XFS was good enough for me. I only keep ext around for small (root) filesystems (hysterical reasons).

Re:features isn't everything

[ZERO1ZERO]

(root) filesystems (hysterical reasons). Hahahahahahah haha hahahahahah hahah (hysterics) Didn't you mean historical?

Re:features isn't everything

[Rich0]

Depends on application. I'm using xfs for just about everything these days. One thing it does REALLY well is deletion of large files (multiple GB) - I run mythtv and that happens quite a bit - often while blocking user input. What can take 20 seconds in ext3 takes less that one second in xfs.

The main gripe I've heard about xfs is that it tends to delay writes quite a bit - so you can get in trouble if you lose power.

As far as Reiser goes - well, I'm running on amd64 and I like keeping my data intact... :)

extents oh boy!

[davidwr]

Now my p0rn won't be fragmented!

Short 4

[camperdave]

Oh! If only they could have squeezed an extra 4 petabyes on so that we could get a true exabyte. What am I to do with my last 4PB of... um... art?

Convert avi to DVD

[Laur]

Incidentally, what tools do you use to convert mpeg4 avis's to DVD? This is something I've been meaning to investigate for a while now.

Re:Convert avi to DVD

[CastrTroy]

I'm using the tovid tools, which is easily installed with URPMI under Mandriva. Not sure if your using Windows or another Linux distro, so getting the tools installed could have varying difficulty. I followed the instructions found here there is a couple of mistakes (possibly due to things changing with the tools) but it gives you pretty good starting point. I may post something on my blog (link above) about how I'm actually doing the process.

Re:Convert avi to DVD

[Laur]

Ah, thanks a lot, the torvid tools look pretty simple to use, exactly what I was looking for!

Not sure if your using Windows or another Linux distro

Heh, I use both, so I can be very flexible in the tools I choose.

Re:Convert avi to DVD

[Shawn is an Asshole]

I've been doing videos of my band and am working on a DVD. What I've been doing is using Avidemux2 for the editing saving it all in huvyuv (lossless video codec). When I'm ready to encode it to MPEG2 I use Avidemux2 to save it as an uncompressed AVI (fucking huge, btw) and then load it up in TMPGEnc under Wine and use it's excellent multipass encoder to create the final file (progressive 23.976fps w/ "3:2 Pulldown when playback"). I've tried all of the open source mpeg2 encoders but TMPGEnc does a significantly better job and the price isn't bad. The downside is when running under Wine your only option is uncompressed video as the source.

Next NTFS?

[Lost Penguin]

I would guess the "next NTFS" will be Reiser 4 renamed; now that Hans Reiser is busy on "other matters", Microsoft can complete their plan. /Yeah, I can believe MS would commit murder to steal code.

Re:Next NTFS?

[krewemaynard]

"/Yeah, I can believe MS would commit murder to steal code."

...

REALLY??!

So funny

[augustz]

The article says "On the downside, the new ext4 filesystem will offer only limited backward compatibility with ext3-aware Linux kernels."

Ext4 is going to be the MOST compatible with Ext3, relative to ANY other option out there.

Upgrading to Ext4 is NOT going to involve a dump and restore from Ext3, likely a tunefs -j or similar command, just as the ext2 -> ext3 migration worked. Ext4 will be able to mount ext3.

If older versions of software could use the new format, you wouldn't need the new format. Yes, upgrading to Ext4 means your 120 petabyte raid array will not be compatible with your old "ext3 aware kernel". But it is PRECISELY because such an array is not possible under ext3 that ext4 is going to be introduced.

And does this submitter think other fancy new filesystems magically work on old kernels? Of course not. Does the submitter know if ext4 will be backported and made available to older releases? It doesn't look like they gave that much thought either.

Please read this for a more detailed description of what is happening.

Slashdot's always good for a smile.

Re:So funny

[l4m3z0r]

The article says "On the downside, the new ext4 filesystem will offer only limited backward compatibility with ext3-aware Linux kernels."

Well if you had a clue, you would know that this is in reference to the fact that you can mount a ext3 file system as ext2 from a kernel that doesn't have support for ext3. It just doesn't journal. They were simply stating that this rather useful feature that made ext2->ext3 migration so painless will likely not be as simple in ext3->ext4 migration.

With that in mind, the rest of your post can be safely ignored or moderated into oblivion, thanks for playing!!

Re:So funny

[augustz]

It is not as easy as ext3->ext2 granted, but you can still remove extents I believe if you want to get your file system into ext3 shape. Probably would involve copying them all over themselves though! Blah.

But unless we want to keep essentially ext2 forever, at some point the forward compat is going to be broken. And while ext3 is ext2 is cool, ext3 is itself available, at one time all the way back to the 2.2 series of kernels. My suspicion is that ext4 will be available for older kernel series such as 2.4 in one form or another.

In other words, a tradeoff, but not an unexpected one given a full version change.

Re:So funny

[l4m3z0r]

You can mount any cleanly unmounted ext3 filesystem as ext2 by any kernel with ext2 support(it doesnt need any ext3 code at all, compiled without ext3 support, years before ext3 ever came out). ext3 is backwards compatible, meaning using it does not preclude you from using that filesystem as ext2 with ext2 drivers. It has nothing to do with moving ext3 drivers into an older kernel.

ext4 conversely will not be completely backwards compatible, which means you cannot simply mount a ext4 filesystem as ext3.

I repeat this has nothing to do with moving code into older kernel versions. This is about backwards compatibility.

Re:So funny

[kobaz]

I think you have some terms mixed up.
Backwards compatibility means that version 2 will work with data from version 1. Forwards compatibility is when version 1 will work data from version 2.

In terms of the ext fs:
- ext2 is forwards compatible with ext3 since the file storage structures were not changed and the only addition was journaling; you can mount an ext3 fs with ext2 drivers
- ext3 is backwards compatible with ext2; you can mount an ext2 fs with ext3 drivers.

Re:So funny

[augustz]

"ext4 conversely will not be completely backwards compatible"

Backwards compatibility will be maintained. The ext4 team has made that crystal clear.

We are talking about forward compatibility here. I think you have your terms confused as well. Forward compatibility is difficult to maintain.

I'm talking about tradeoffs. If ext4 is a simple upgrade from ext3 (no dump and reload) and ext4 drivers are relatively easy to obtain for older systems, and ext4 provides some improvements, and it is clear (by the names alone) that they are different file systems, then for most practical needs (not the I only restore systems using a 2.2 kernel variety, or I only use 2.4 kernels, but refuse to load the ext4 drivers) ext4 is going to be fine.

Forward compatibility usually matters most if older tools cannot be updated to deal with new datastructures etc. The real physical world has more of these situations then the it world, but both exist. However, in the case of the ext4 filesystem, in common use that is not the case. And even if you are stuck using a 2.4 kernel for some reason, I was pointing out that it is likely (if ext4 follows the ext3 model) that someone will post ext4 patches against the 2.4 series as well.

Anyways, save your cute signoffs like "thanks for playing" or "if you had a clue" for someone else. Seriously, resist the urge for that junk.

Re:So funny

[oracleofbargth]

There are some newer features available in the ext3 codebase in newer 2.6.x kernels that will not work in some older kernels with ext2, regardless of whether it is unmounted cleanly, namely directory hashing, large inode support, extended attributes, fast extended attributes, and acls (which embed themselves in the extended attributes).

There may be more, I haven't been digging into the code myself lately, not for the last couple of years, anyway.

Re:So funny

[sjames]

It sounds like migrating will be quite simple, just that falling back won't be an option. That will slow the adoption for some applications, but for most people the easy way forward will be enough.

Groups of crows?

[Ayanami Rei]

Or do you mean murderers...

Or XFS/ext3 on LVM

[Ayanami Rei]

Woo... snapshots. *blink*
(okay, so apparently yes, ext3 now comes with the 'lvm enhancement' patches that allow it to flush and lock operations during snapshot creation to avoid inconsistency after the fact)

As I said, long-term archival.

[porkchop_d_clown]

For example: I used to work in a business that felt it was reasonable to store information for a 100 years or more; just because medical research was done in the 19th century doesn't mean the results might not be useful today. But modern research generates far too much data to simply print it out on acid free paper. Technology for their needs simply doesn't exist today.

More personally: I've already had CD-Rs fail after just a few years, showing me that my digital photos are at at least as much risk as my old film negatives, if not more. I want to make sure my great grandkids can look at these shots, but hard disk interfaces and tape drives all become obsolete in just a few years.

Re:As I said, long-term archival.

[Zapman]

We as individuals are kinda screwed. The best thing is to keep a rolling hard drive set:

get two large hard drives, keep one off site (safe deposit box, at the office, whatever). Use each to get a regular (monthly or so?) backup of all your data.

As size of hard drives increase, keep getting new ones (or use multiple in each of your two sets).

Once every 2-3 years or so, re-evaluate everything.

On the enterprise level, it's a LITTLE easier... certain tape and drive makers promise compatibility for 7-10 years. And EMC has there Sentera (sp?) product line which they promise upgrades and compatibility for the next 30 years... but these things are rather expensive.

Uh huh.

[porkchop_d_clown]

And what happens when the luser is you?

No one can call themselves a "real" sysadmin until they've accidentally reformatted the root partition.

Re:Uh huh.

[MrNaz]

Been there, done that, and won't be doing it again. In any case, I have critical data backed up to a separate physical disk that doesn't get used except for backups.

I knew a guy...

[porkchop_d_clown]

who wanted to delete some of the preference files in his home directory, so he typed

rm -rf .*

It never occurred to him (or me) that ".." matched that pattern. He worked his way right up the directory tree and back down again...

Re:I knew a guy...

[porkThreeWays]

I've heard this a million times but have yet to see a modern system actually have this behavior. I'm fairly certain bash has protected against this behavior for awhile now...

Re:I knew a guy...

[jesboat]

No, actually, bash doesn't. Try something like echo .* , and you'll see that . and .. are, in face, there. What protection that does exist is that on many modern systems, rm will refuse to remove . or .. (though the truly motivated can foil that with a rm `pwd`)

read the jargon file, dude...

[Paul Crowley]

I think they mean hysterical raisins.

I don't understand...

[Almahtar]

This is a bit OT, but I don't understand why people would abandon a filesystem whose creator is suspected of murder, but they're totally cool with a filesystem whose creator is convicted of multiple patent infringements, false testimonies in court, and anticompetetive business practices.

Re:I don't understand...

[Anonymous Coward]

Is that Kill Bill Vol.3?

LOL!

[Anonymous Coward]

/. for ~10 years, and this is my first time I modded something +1 funny

That would explain

[porkchop_d_clown]

why the incident in question occurred in 1994, on a Data General AViiON, that wasn't running Linux and was using csh as the default shell.

You seem awfully sure of yourself for someone who doesn't even realize there are other shells besides bash.

Doing a global update...

[nu1x]

will be a bitch tho :)

unsophisticated is good

[r00t]

The forced fsck is because hardware and unrelated software can make errors. (failing disk, bad RAM, loose cable, buggy controller, an unrelated buggy driver like audio, electrical noise from the power supply, overheating CPU, a cosmic ray, an alpha particle emmission...)

If you totally trust the journal, eventually your filesystem will be trashed.

Very few filesystems are more durable than ext3. We will expect the same of ext4. Of the non-journalling filesystems, ext2 was relatively durable. This is a design team that knows their shit.

Much less durable: reiserfs, reiser4, jfs, xfs, hfsplus, ntfs
Slightly less durable: ufs, ufs2
Possibly more durable: zfs

With ext2/ext3/ext4 you get data structures in fixed locations on the disk. This makes them much harder to trash. With most filesystems, the loss of a single critical bit (in a pointer to the root of a tree) will wipe out the whole filesystem. The odd beast here is zfs, using replication to protect an otherwise-fragile tree structure.

Re:unsophisticated is good

[rubycodez]

ufs with journalling less durable than ext3? any evidence to back that claim? My experience with hundreds of linux and solaris servers says much the opposite, but I can't point to any study.

Re:unsophisticated is good

[r00t]

I was primarily thinking of BSD systems. These systems combine "soft updates" or weak journalling with a low-quality fsck program.

With ext3, you get a high-quality fsck and 3 different journalling choices. (4 choices if you count ext2) By default you get the mid-level journalling option. You can enable full data journalling if you wish; this is usually slower but can be the fastest for fsync-heavy workloads like email servers.

When you say "solaris", my first thought is "hardware that isn't remotely similar to a $400 PC". That counts for a lot. You probably get better UPS hardware too.

In any case, it's a minor difference. The ufs/ufs2 structure is very similar to the ext2/ext3 structure, and moderately similar to the ext4 structure. The main differences are fsck quality and log/journal/ordering behavior. All these filesystems have a fairly simple and static layout.

Agreed.

[porkchop_d_clown]

It's also been said that high quality DVD and CD ROMs last a lot longer than the cheap ones, so my current strategy is a single hard disk paired with DVD roms, but I'm getting ready to switch to a NAS with a mirrored array.

Re:Agreed.

[Zapman]

I guess it depends on how important your data is. That solution doesn't give protection in the case of a house fire or theft.

Discipline in backups is key... my personal failing is taking the second hard drive to the office.

First benchmarks of ext4

[Anonymous Coward]

First benchmarks of ext4 vs ext3/reiser4 reveal that it's already performing better in some areas. Even though it's still under heavy development.