How To Move Your Linux Systems To ext4

LinucksGirl writes "Ext4 is the latest in a long line of Linux file systems, and it's likely to be as important and popular as its predecessors. As a Linux system administrator, you should be aware of the advantages, disadvantages, and basic steps for migrating to ext4. This article explains when to adopt ext4, how to adapt traditional file system maintenance tool usage to ext4, and how to get the most out of the file system."

Comments

But does it run...

[RiotingPacifist]

reiser4?

Re:But does it run...

[2muchcoffeeman]

Reiser4 will absolutely kill ext4.[/badtastedaemon]

Re:But does it run...

[Sentry21]

From what I've read, Reiser4 completely kills Ext4 in performance... then it disposes of ext4's kernel module, removes one of its redundant drives, and then cleans the free space left on its array.

Re:But does it run...

[electricbern]

But it is too verbose, and that ends up being a problem.

Re:But does it run...

[CatOne]

How would you know? It's as if the old Ext4 system just... vanished.

Even if you found a platter under the front seat.

Re:But does it run...

[erlehmann]

Not only that. It killed my backup copies of bride.ru , i needed for, err science.

Re:But does it run...

[Anonymous Coward]

sounds like EOL with no possibility of fork.

Re:But does it run...

[erlehmann]

You apparently didn't get the whole picture. It's not about single files - Reiser4 is just a better choice for partitioning your wife.

Re:But does it run...

[Anonymous Coward]

Try looking amongst the random debris in your fanny pack.

Re:But does it run...

[apt-get moo]

In Soviet Russia, files find and kill YOU!

This never seems to get old.

Re:But does it run...

[Trogre]

Speaking of which, since all this Reiser ugliness, has anyone else noticed the surge of slashdot ads for Russian mailorder brides?

Coincidence?

Re:But does it run...

[Chrisq]

It has a built-in wash disk facility, which will hose down your old data but leave an inch of water.

Re:But does it run...

[AftanGustur]

Well, you open up your computer to find that the hard drive has been removed ..

And the rest of the PC interior has been carefully cleaned.

Re:But does it run...

[Corwn of Amber]

Just to ask in a high thread, is it any faster than Reiser4?

No, not flamebait. But Reiser3 was so much faster than everything, and Reiser4 squashed those perfs with one hand tied. So I'd like to know. Benchmarks anyone?

About "moving your data" : backup, format, restore. (tar, mkfs.ext4, tar -x. See? Polyglot!) How hard is that and why does it need an article and how is that news?

Re:But does it run...

[mav[LAG]]

Although there's still an inch of water sloshing around in the bottom of the case and two books on why you should remove hard drives...

Mere Mortals?

[db32]

"largely unnoticed by mere mortal Linux users and administrators" strikes me as a strange phrase to find on this IBM page. Is there some other IBM project more interesting than ext4 being revealed here?

Re:Mere Mortals?

[mweather]

Do you think the public response would have been different had he been in a bar?

Re:Mere Mortals?

[JackieBrown]

You know, I thought I was missing a post and others got it, until I realized that you were responding to yourself.

Nice trick.

Not for the casual user

[halivar]

ext4fs is designed to be used in systems requiring many terabytes of storage and vast directory trees. It is unlikely the common desktop (or even, for that matter, the common server) will see appreciable performance increase with it.

Re:Not for the casual user

[Anonymous Coward]

Do you realize how much porn some people have?

Re:Not for the casual user

[Vellmont]

It is unlikely the common desktop (or even, for that matter, the common server) will see appreciable performance increase with it.

Disk sizes are going up. In a few years you'll see a terabyte on a single drive. I'd also say that features like undelete, and online de-frag are important to anyone.

So while you may not see any real performance increases, that's really beside the point.

Re:Not for the casual user

[Anonymous Coward]

Instead of waiting a few years, go to your local computer store. They should have terabyte drives now.

Re:Not for the casual user

[A nonymous Coward]

Disk sizes are going up. In a few years you'll see a terabyte on a single drive. Unlike those two 1000 GB (or is it 1024) drives I have on my desk now.

Re:Not for the casual user

[miscz]

I can't wait for faster fsck. It takes something like an hour on my 500GB ext3 partition. Terabytes of storage are not that far away.

Re:Not for the casual user

[Uncle Focker]

Disk sizes are going up. Since last year we've seen a terabyte on a single drive. Fix'd it for you.

Re:Not for the casual user

[XenoPhage]

All you young kids want these days is a faster, more convenient fsck.. What about the old days where fscking was about the technique, not the speed or the size...

Re:Not for the casual user

[stuporglue]

I have the following disks in my computer:

1 TB
500 GB
300 GB

When they decide to fsck at the same time, it can take 1/2 hour or longer to get to the login screen.

Re:Not for the casual user

[dpilot]

It might be a win for me even today on my meager 300G MythTV media partition. I'm currently using xfs for that, but every now and then I hear about bad things on xfs with a power failure, and other times I hear that it can be physically hard on the hard drive. (excess head motion?) Of course other times I hear that xfs is the best thing since sliced bread, and is usable for ANY purpose with just a little tuning.

I transcode my Myth stuff on an ext3 partition, and occasionally get complaints about the large data size without having the right options set. But it works.

Re:Not for the casual user

[LWATCDR]

But EXT4 because really useful when you have many terabytes of disk storage. With just one or two EXT3 is probably good enough.
Now when we have ten TB drives....
Good grief people Yea just keep a few thousand TV shows on your desktop.

Re:Not for the casual user

[LWATCDR]

No file system likes a power failure. Bet a UPS that will shut down the PC. They are cheap.
And if you care about that data make a backup and even better run a raid.

Remember EVERY HARD DRIVE IS GOING TO FAIL SOMEDAY.

Re:Not for the casual user

[trolltalk.com]

ext4fs is designed to be used in systems requiring many terabytes of storage and vast directory trees. It is unlikely the common desktop (or even, for that matter, the common server) will see appreciable performance increase with it.

Really? My (comparatively) cheap laptop has 640gig of storage, and when you start getting into video, 640g is NOT enough!

You can buy 2 x 500gig desktop hds for the grand total of $150.

At that price, a terrabyte will be the "new pink" within a couple of years. Just like 2 gigs of ram is now the "norm".

Re:Not for the casual user

[EvilRyry]

This is why we have XFS. I fscked a 9TB partition is under 10 minutes. Hopefully they've done some improvements for ext4 in this area. A volume that takes days to fsck might as well just die completely.

Re:Not for the casual user

[techno-vampire]

I can't wait for faster fsck.

I can tell you're a slashdotter. When most people fsck they want it to last as long as possible.

Re:Not for the casual user

[drinkypoo]

What about the old days where fscking was about the technique, not the speed or the size...

I'm just happy when it's done for me, and I don't have to handle it manually. When fscking fails at the beginning, it can ruin your whole day if you're not an expert.

Re:Not for the casual user

[dpilot]

My /home is nfs4, the server is on a UPS, and the disk(s) on the server are raid-1.

I don't really care that much about TV, but would prefer not to lose it. If it were on ext3 I'd have it journal=data, or is that data=journal, anyway, full data journaling instead of just metadata. Incidentally, my raid-1 is set up that way, both for reliability and because I'd read that it's actually faster for an nfs server.

Re:Not for the casual user

[Daimanta]

I fscked a 9TB partition is under 10 minutes. That's what you get when you throw a harddrive in a bathtub with water.

Re:Not for the casual user

[diegocgteleline.es]

Ext4 has a lot of performance improvements, like extents or delayed allocation. Desktop users will notice that ext4 is much faster

That said, ext4 is unstable. It can easily eat your data. Just say NO to moving your filesystem to ext4 - for now.

Re:Not for the casual user

[instagib]

Correct, and that's why outsourcing your fscking needs is the way to go. These days it is more and more common to not maintain fscking resources inhouse; instead you choose from a variety of service level agreements who will take care of everything around fsck and related issues.

Re:Not for the casual user

[CrazedWalrus]

No kidding. Sometimes I'm just not into fscking either, but if you're failing at the beginning, you may want to call an expert.

Usually my problem is that my fsck gets a "fsck-completed-normally", when the media is really only half fscked.

But don't worry -- fscking takes practice. If you got a quality media, you can half-fsck it many times before the media fails completely.

May I also suggest fscking aids? There are many tools on the market that can help when your fscking routinely fails doesn't complete. They're usually lightweight and easy to use, and can help to save your media from getting fscked elsewhere.

As you said, when all else fails, sometimes you really do just need to handle it manually.

Re:Not for the casual user

[stanleypane]

You obviously haven't been married ;)

Re:Not for the casual user

[ewanm89]

May be they should parallise it, so they actually happen simultaneously, surely the most expensive part is actual disc access speed.

Re:Not for the casual user

[SpaceLifeForm]

That should be 'dick' sizes if you're going to be waving your hard drives around here.

Re:Not for the casual user

[Ucklak]

That's not bad compared to Windows.

I have a 160GB, and 2 500GB and when it CHKDSK, it can take a couple of hours.

Re:Not for the casual user

[erlehmann]

Good grief people Yea just keep a few thousand TV shows on your desktop.

This could make for RIAA settlements in an order of magnitude of the GDP of a small country !

Re:Not for the casual user

[glwtta]

ext4fs is designed to be used in systems requiring many terabytes of storage and vast directory trees

Well yeah, but Slashdot seems like a pretty good place to find people who administer multi-TB systems, no?

A terabyte isn't what it used to be (hell, 1TB SATA disks are pretty common) and ext3 sucks pretty hard even on a measly TB.

Does the sub-TB desktop crowd even care about filesystems? I mean, they all pretty much work and these days the popular ones have pretty similar performance (on a single spindle, at least).

Re:Not for the casual user

[Sancho]

Why not? When storage density gets so high and the drives get so cheap, why not rip all of your movies and store them on disk? I'm lazy, and don't want to get up to change the disk.

Re:Not for the casual user

[LWATCDR]

Well just how many times do you watch a movie? That is one of the things I never could get.
I know people that collect movies. They have a netflix account and as soon as they show up they make a copy. How much free time do people have to watch movies or is just the thought of having them?
But right now you can store a few hundred movies a moderate sized HD. What will you store when you have many TB of storage. And of course we have the old slashdot joke about the porn collection.
But back to your lazy part. I guess I am too lazy to rip and store a few hundred or even a few dozen DVDs. I will just grab one from the box and go.

Re:Not for the casual user

[EvilRyry]

There is more to computing than desktops. To many businesses, 16TB is almost nothing.

Re:Not for the casual user

[Sancho]

I have scripts to automate any digital conversions that I perform, so time isn't really a factor. It's easy enough to grab a stack of discs, set them by my workstation, and do the ripping while you're working on other tasks. If you have a media PC that you use to play back DVDs, you can perform the ripping there the first time that you want to watch the movie. Any time someone else wants to watch it, they can just watch the cached version.

As for your other points, it's largely a packrat mentality. There have been times when I wished that I could rent a movie, only to find that it's no longer available on DVD. Sometimes Netflix has it, sometimes they don't, but when production stops, it stops, and as copies of the movie wear out, it becomes much harder to come by. That makes purchasing them and backing them up on magnetic media more attractive, even if I may not get around to watching the films again any time soon.

Re:Not for the casual user

[whmac33]

from
man fsck
"filesystems on different drives will be checked at the same time to utilize parallelism available in the hardware."

If / is small it will be quicker. If / is a large partition then it hinders any parallelism.

              The sixth field, (fs_passno), is used by the fsck(8) program to deter-
              mine the order in which filesystem checks are done at reboot time. The
              root filesystem should be specified with a fs_passno of 1, and other
              filesystems should have a fs_passno of 2. Filesystems within a drive
              will be checked sequentially, but filesystems on different drives will
              be checked at the same time to utilize parallelism available in the
              hardware. If the sixth field is not present or zero, a value of zero
              is returned and fsck will assume that the filesystem does not need to
              be checked.

Re:Not for the casual user

[Xabraxas]

Ext4 has a lot of performance improvements, like extents or delayed allocation. Desktop users will notice that ext4 is much faster

XFS has both extents and delayed allocation. I really don't know why we need Ext4. XFS has been a very solid fs for quite some time now, it's sad that more attention hasn't been payed to it from kernel hackers. The whole idea behind Ext4 seems to be more of a NIH syndrome than anything else. I could understand if it was radically different but it isn't.

Re:Not for the casual user

[whmac33]

Err... man fstab

Re:Not for the casual user

[LWATCDR]

Yes I have thought about setting up a media PC but then I would have to set up an NAS and then I would want to move over to GigaE. Once I did that I would want to get an IPod Touch and write a WiFi based Universal remote program for it....
It is just more productive to grab the DVD.

Re:Not for the casual user

[rainhill]

I bet fscking was done faster in older days, have you seen our forked ancestors fscking on tree branches in African jungles?

Re:Not for the casual user

[oddfox]

One of many reasons right here

I messed around with Ext4 for a little while on my machine (Like a couple days, just toying with it and seeing how its performance compares to Ext3 and Reiser4) a while back, like maybe a little bit before it was merged as experimental in the mainstream kernel. It is fast, backwards-compatible and extremely featureful. XFS is not a bad filesystem, but it has some problems, in my eyes. Metadata-only journaling, aggressive caching that makes it a potentially dangerous choice if you don't have a UPS, very slow metadata and deletion operations.

That's great that XFS has a lot of features Ext4 is bringing to the playing field, and has had them for a long time. To pretend, however, that the developers of Ext4 simply have a NIH syndrome is just silly and disregards the fact that there is a lot that Ext4 already provides that XFS doesn't, and even more that it will soon. You might not see what the big deal is, but really, I can assure you that it won't be very long before the new ideas Ext4 employs are in widespread use.

Here's an interesting article that really caught my eye with this: "Storage snapshot: The financial firm has more than 14 Petabytes of active storage and plans to add "several more Pbytes" within the next 12 months."

Re:Not for the casual user

[Erpo]

It is unlikely the common desktop (or even, for that matter, the common server) will see appreciable performance increase with it. Extents!

Re:Not for the casual user

[Awptimus Prime]

do this mean if i upgrade, i can actually perform 'ls' in my mp3 folder? it seems my computar do not like 8k files in one directory.

Re:Not for the casual user

[turing_m]

Several points as to why the need for massive TB of storage of video (backups of course):
1) People wanting a lossless copy of the original media. i.e. purists.
2) Bonus content.
3) HD video.
4) Kids - they will want to watch different movies, over and over and over, at different ages.
5) Watching a movie with friends.
6) Keeping it all out of sight, out of mind.
7) Wanting to keep the NAS within a small power threshold, i.e. more easily done with higher storage density/fewer HDD. Better for the environment too with fewer HDD.
8) Never underestimate the bandwidth of a stationwagon full of HDD.
9) As someone else said, the packrat mentality. Who knows when that movie you want to watch will be out of production or perhaps banned?

Re:Not for the casual user

[sydneyfong]

As a sibling poster has mentioned, using XFS is dangerous if you don't have a UPS.

It's well known that if you get a hard shutdown (like, sudden power failure) when accessing files in a XFS partition, the contents of those files will be zeroed out.

For expensive server farms this won't be a problem. UPS, redundant servers, backups, etc will generally render the problem moot. But for my desktop that's backed up like once every few months (I know, I know...) data integrity is much more important than whatever advantages XFS could provide.

Re:Not for the casual user

[TractorBarry]

Whilst it is undoubtedly important (someties vitally so) "Undelete" is the wrong answer to the wrong problem.

What we need are file systems that allow multiple versions of files. Whenenver a files contents is altered (e.g. by an editor) the output should be to a new version of the which has the same logical name but which has an implicit higher generation number. The generation number being another file attribute which is held at the directory level.

So for instance I should be able to keep multiple generations of, let's say "/etc/X11/xorg.conf" (yeah I have loads of problems with that) without have to resort to manually adding sufffixes to files such as "/etc/X11/xorg.conf-2008-01-01", "/etc/X11/xorg.conf-2008-01-02" etc. etc.

By using this scheme then editing xorg.conf would results in two files which would show up from, say the output of "ls" as follows: /etc/X11/xorg.conf(2) /etc/X11/xorg.conf(1)

n.b. The bracketed generation number whon at the end of the file name is not part of the name, it is only displayed when using "ls etc." By default you'd just use commands such as "nano /etc/X11/xorg.conf" and, if 2 were the highest generation, you'd start editing generation 2. If you wanted to you could obviously do "nano /etc/X11/xorg.conf(1)". Both of these edits would result in generation 3 of the file being created.

We've had this sort of file system in VME (Virtual Machine Environment) mainframes since the 1970s and quite frankly I find using *NIX and WINDOWS file systems to be utterly primitive and archaic.

The whole "only one generation of a file at any one time" paradigm is ludicrous and needs replacing with a better model. Undelete is simply an, admittedly very useful, but poor "hack".

Just my periodic "primitive PC file systems" rant :)

Re:Not for the casual user

[Random Walk]

XFS has that nasty 'security' feature that it will zero files that were open when the power failed. Never use XFS on hardware that has no battery backup to shutdown properly if you trip over the power cable.

Re:Not for the casual user

[aproposofwhat]

Psssst!

If you've got fscking aids, stop fscking the wrong partition!

Re:Not for the casual user

[at0mjack]

On a 64-bit system with lots of memory, fine. I had a 2TB XFS partition get corrupted, and it was not possible to recover it as the fsck tool required more memory than is possible under a 32-bit OS.

Re:Not for the casual user

[at0mjack]

One big problem with XFS is that it's not usable for decent-sized partitions unless you have a 64-bit kernel. If you have a filesystem bigger than around a terabyte you can't run fsck on it, as fsck will require >3GB of RAM.

Re:Not for the casual user

[Anonymous Coward]

5.76TB, excluding the donkeys.

Re:Not for the casual user

[Crayon Kid]

Actually, I expected to see parent modded Insightful.

Re:Not for the casual user

[Crayon Kid]

Oooh, love the subtle touch, how you said "have been" ie. not "are".

Re:Not for the casual user

[MMC Monster]

I buy movies when the run $5-15 at the local store. I like having the physical disc. I may watch a movie just once, but good ones I'll watch at least a few times or more.

There's nothing better than to sit down with the wife one evening and use the remote and flip through the DVD collection on the TV and start the movie without having to get up and figure out what to watch.

It also means we're more likely to watch something we both enjoy rather than whatever happens to be on the TV at that time.

Re:Not for the casual user

[hackstraw]

When they decide to fsck at the same time, it can take 1/2 hour or longer to get to the login screen.

Why do they do this?

The defaults for mke2fs are unfortunately wrong. The check after X mounts and Y amounts of time are not done by any OS installers that I know of. man tune2fs on how to stop these checks on boot. I've turned these off on over 1000 disks in my time and it has yet to of come to haunt me since, doing a reboot and having to wait an unknown amount of time really sucks.

Re:Not for the casual user

[Abalamahalamatandra]

Gosh, without even trying I'm up into the 3+ terabyte range on my home server, and that's just for casual whole-house use for me an my family. An easy 6-7 gigabytes of that alone is a HUGE tree of JPG files for family photos accumulated since I started with digital photography in 1999. Another terabyte is raw video that I've been accumulating that needs to be edited down, just have to get around to it.

So, given those kind of figures, I think the "common server" could easily start getting there, if not now, surely within the next 10 years. When I get into that kind of storage, I'll want a filesystem that's been around awhile and has stood the test of heavy production use. No offense and no jokes here, but I never considered ReiserFS for that reason - I'm not going to trust my data to unproven filesystems.

Re:Not for the casual user

[Fatalis]

epic fail.

Re:Not for the casual user

[Abalamahalamatandra]

Something tells me you worked at DEC at some point in your career.

I don't think we'll be seeing things like file generation identifiers showing up in the actual filename any time soon - that's pretty primitive. I would like to see snapshotting of the type that's been available on NTFS for quite awhile, though, to allow for separate restoration of previous versions and proper backups of open files.

Re:Not for the casual user

[gosand]

After you have kids, you aren't so much interested in how long it lasts, but the MTBF. (Mean Time Between Fscks)

Re:Not for the casual user

[civilizedINTENSITY]

I don't know, but it rather sounds as if you are being *very* productive in the former scenario, and just watching a movie in the latter.

Re:Not for the casual user

[Inner_Child]

The RIAA is in charge of suing people that pirate TV shows now? They really are branching out, aren't they?

Re:Not for the casual user

[halivar]

but I never considered ReiserFS for that reason - I'm not going to trust my data to unproven filesystems.

Yeah, you don't want it killing your system.

*ducks*
*runs*

Re:Not for the casual user

[Grishnakh]

Disk sizes are going up. In a few years you'll see a terabyte on a single drive.

Hello from May 2008! You appear to be communicating with us through a wormhole from the recent past, perhaps 2005. I wish I could offer you some words of encouragement about the future, but other than terabyte drives now being a reality, I don't really have much good to say. Assuming you're in the US, gas prices are way up, we've selected three really awful candidates to run in the November elections, and the realty bubble has collapsed, and we're entering a recession. I recommend you sell your house(s) now while the prices are still high, because they're about to crash.

Re:Not for the casual user

[rtechie]

As others have pointed out, ext4 offers undelete and online defrag, features users have wanted for many years. However, I think a better case can be made to move the the extremely stable XFS file system rather than mess with the buggy ext4. XFS has more features anyway. The only really credible competitor seems to be ZFS.

Once ext4 is more stable everyone should move to it (or, I would argue, XFS).

Re:Not for the casual user

[medlefsen]

this has been beaten to death, but just so ya know, a TB is 10^12 bytes so it is correct. A TiB (tibibyte) is 2^40. for more info: http://en.wikipedia.org/wiki/Binary_prefix

Re:Not for the casual user

[amRadioHed]

Forget about the computer, I couldn't stand to have 8k files in one directory. I've got ~10k music files, but they are sorted by artist and album. That way the largest directory, Artists, has only ~500 files.

Re:Not for the casual user

[Awptimus Prime]

Yeah, I know it sounds bad, I used to keep them in the same fashion. I'd have everything sorted out all nice, but then I got a macbook and let itunes import my stuff. Of course, it did the crazy sorting and got everything set up in impossible to manage directories. This was fine, since all my original stuff was on the other system, raid5 and such. Well, as luck would have it I lost a drive and a spare on the same day due to a faulty PSU. Then a few days later, my position was eliminated and I had like 2 hours to get everything off the laptop-- thus copied the whole mess into one directory on one of my linux boxes. Being impatient, I just went and imported them with Rythmbox and Twonkymediaserver, which both work great since they use a db to store the file's locations. To cut down on resources, I finally just broke the directories off into chunk1 chunk2 and so on. To keep things maintained, I make sure everything has good id3 data so it doesn't matter how jumbled the files are as long as I've got some frontend to deal with them.

Yes, a far cry from the days of using mplayer on the command line. It was essential to keep everything organized very well back then, but modern technology has turned me into a file slob.

Re:Not for the casual user

[amRadioHed]

Well, at least you've got a good reason for the mess they're in! At least you were able to recover the files. It's my worst nightmare to lose the RAID1 I have all my music on and have to rip everything again.

Re:Not for the casual user

[Awptimus Prime]

If only I still had my original CDs. I'm stuck lugging around these bits on various backups until I get the mad money for a better solution. It's sadly near the bottom of my stack of to-dos which means I'll probably have disaster strike.

Wikipedia entry

[drgould]

Link to Ext4 entry on Wikipedia for people who aren't familar with it (like me).

Re:Wikipedia entry

[miscz]

Because nobody on Slashdot knows that primary filesystem used on Linux is called Ext3 and we're too stupid to figure out what Ext4 might be. Come on.

Re:Wikipedia entry

[discord5]

Because nobody on Slashdot knows that primary filesystem used on Linux is called Ext3

Now now, don't give us too much credit

we're too stupid to figure out what Ext4 might be

It's like ext2 times two, stupid.

Re:Wikipedia entry

[Anonymous Coward]

My Linux box goes to ext11.

Re:Wikipedia entry

[Applekid]

Because nobody on Slashdot knows that primary filesystem used on Linux is called Ext3 and we're too stupid to figure out what Ext4 might be. Come on. I, for one, saw the news item and immediately thought, "How to move to ext4? What's in it for me?"

Thank you, GP, for saving me the seconds of typing with a convenient link. And shame on you for wanting to put out a candle that might be used to light the darkness.

Re:Wikipedia entry

[ajayrockrock]

It's like ext2 times two, stupid.

No, wrong. it's the next stable release of ext. remember people, odd numbers are development, even numbers are stable!

okay, mod me down, I should really come up with new jokes.

Re:Wikipedia entry

[muszek]

Who knew Debian stable was so bleeding edge all these years (my grandma says "decades")?

Re:Wikipedia entry

[hostyle]

wow. a real life gentoo user! i feel privileged.

Re:Wikipedia entry

[Sentry21]

It's like ext2 times two, stupid. No, it's ext2 squared! Sheesh, seriously.

Re:Wikipedia entry

[Chrisq]

Yes, we all count 1,2,3, 8.1, 95, 2000, 2003

Re:Wikipedia entry

[Mr. Jaggers]

Hey, now! No need to be hostile, we like our Gentoo.

Now, excuse my while I read the handbook and figure out how to keep an old version of portage around to use Manifest1-hash digests for manifests from my overlays...

Seriously, though, I have always liked the Gentoo approach, having always admired the BSD zen of package management, but liking the Linux kernel, especially the 2.6 build system, which totally rocks... and I was thrilled with the kernel 2.2 build scripts when they were released. 2.4 was an improvement from that, and 2.6 has easily twice that improvement.

Plus, being a Gentoo-person, I'll probably be inclined to try out ext4 first amongst my peers, and we do have two machines that will be hosting networked multi-terabyte scratch-space for automated software builds. So, there we have 1) the big filesystems, 2) the networking component (probably via NFS), 3) transient nature of the data, as they're all build outputs, and the source code is in a repository, elsewhere. So, it could completely crap out, and nothing would be lost except that us engineers would have to all spin our own builds, locally. Sounds like a good way to test out ext4. The only problem I have is that I don't really know how to post a useful post-mortem report to LKML if badness happens... hmm.

So, there. I'm not off-topic either. Ha!

Preempting the prefix war

[A beautiful mind]

Yes, Terabyte is not entirely correct according to SI, but Tebibyte just sounds lame and language is a tool, to facilitate written and oral communication.

Of course, in this case you have to balance the confusion stemming from the Tera in IT context meaning 1024 in some cases. To be honest, people insisting on the new naming, they should have come up with a sensible sounding name and promoted that. You have to remember that language, even technical language is for the people. There are lots of ways to craft a beautiful, logical, symmetrical language that no sane person would use because it just doesn't sound convenient.

Maybe a linguist can pitch in to explain why tebibyte sounds so awful?

Re:Preempting the prefix war

[DahGhostfacedFiddlah]

Is it too late to suggest *aybyte? Kaybyte, Maybyte, G...

Dammit.

Re:Preempting the prefix war

[Dachannien]

Maybe a linguist can pitch in to explain why tebibyte sounds so awful? Tebibyte-buh: It's bad-buh because-buh it makes-buh you sound-buh like Mushmouth-buh.

Hey hey hey!

Re:Preempting the prefix war

[GreatBunzinni]

I'm no linguist but the notion of anything that resembles a Tera binary byte doesn't compute all that well.

Re:Preempting the prefix war

[RiotingPacifist]

I think the only place you need to use it is in the abbreviations, there KiB vs KB is sort of useful

Re:Preempting the prefix war

[Bralkein]

Yeah, but now the problem is that KB is now ambiguous - it could either be 1000 bytes, or it could be 1024. Before anyone mentions HDD manufacturers, it isn't ambiguous there, either. 1 KB is 1000 bytes, yeah that's because they're fleecing you, it sucks but oh well.

I just hate the mindset that comes up with all of this stuff, it reeks of the sort of person who alphabetises everything and writes into newspapers to complain that they misused the apostrophe one time on one page. I mean for god's sake, take exbibyte. The word was actually used in the article. Look at it, you can bear to. Say it, if you can figure out how to. The worst thing is that it seems to be becoming more popular. Sigh. I really shouldn't get so worked up by this.

Re:Preempting the prefix war

[mpathetiq]

I think you meant....

Tebibyte: Ubit's bubad bubecubause ubit mubakes yubou subound lubike Mubushmubouth.

Re:Preempting the prefix war

[CastrTroy]

They're only fleecing you if you keep on insisting that 1 KB is 1024 bytes. If you define that 1 KB is 1 billion bytes, then they are are really fleecing you. The only reason that 1024 was used as the size of a KB was because it was much easier, not because we were trying to standardize things, or because it made things simpler to understand. It completely went against all the other standards, just because it made the code a little simpler to write.

Re:Preempting the prefix war

[A beautiful mind]

To put it into more accurately worded language, I was hinting at the phonetic characteristics of tebibyte that makes the word harder to pronounce. Another poster mentioned that "quark" must have sounded weird when it was introduced the first time. I have the opposite experience. Quark is easy to pronounce, it is a distinct, hard to confuse name for a specific particle which has in fact a quite interesting etymology. I just love quark. It is actually one of the best examples of naming I could come up with, if asked off the top of my hat.

Re:Preempting the prefix war

[pizzach]

It really didn't start out that complicated, but it's the manufacturers who keeps F*ing it up because they are trying to stretch the numbers. I think it's the consumers who are the victims in this.

Hard drives have the MiB-MB problem because manufacturers wanted to be able to say 60GB instead of 54GB. When you buy a monitor, you have look for viewable size in much smaller print. Then there is the dithering you hear about on modern LCDs. I've also heard that early monitors were measured by their horizontal instead of their diagonal. Of course, the diagonal is longer so...

Meh. Buyer beware.

Re:Preempting the prefix war

[sconeu]

If you define that 1 KB is 1 billion bytes, then they are are really fleecing you

I'd say that i fyou define that 1KB is 1 billion bytes, then you've got bigger problems than the marketing departments of drive manufacturers.

Re:Preempting the prefix war

[VirusEqualsVeryYes]

I am not a professional linguist, but I think I can explain.

In any spoken language, different sounds are loosely associated with different ideas. As a simple example, voiceless sounds, like p, k, t, f, and s, are well suited for pointed use, as in pejoratives; and r, especially the alveolar trill variety, is associated with intimidation or primality. These associations are made either because it sounds like something else ("rrrr" sounds like an animal's growl or roar -- notice the Rs in "growl" and "roar"?) or because the sound serves a purpose (hard, clipped sounds serve well as punctuation -- notice all the hard sounds in "punctuate"?). In the latter case, combinations of sounds can invoke a wide array of ideas or feelings. Utilization of these things is key to a good punchline or to controlling semiconscious undertones of speech. I admire Dr. Seuss in particular for his mastery of sound combinations in making up suitable words to balance sing-song silliness with gravity and purpose.

Now, returning to "tebibyte" and all the other -bibytes, soft, voiced consonants like B are associated with childishness (a baby might make these sounds), silliness, bounciness, or informality. Two Bs in a row are especially so: bib, baboon, babble, bob, boob. The reason tebibyte sounds "stupid" is that it describes a technical idea using unsuitable sounds.

This being said, if you were used to using the word, you wouldn't think twice about it. You would probably complain about "flop" and "watt" in the same way if the words were new, but established use overrides the weak sound associations. The President could be instead called the Biggyloppalo and few would care, as long as the term were already established in the common vocabulary. I'd think people would move on even if a video game console were named something as ridiculous as "Wii" ... but that's just a wild guess.

As for my opinion on the matter, I'm in favor of it. The SI prefixes are already assumed to be powers of 10 in all other fields except the computer and information sciences. Tebibyte will maybe sound silly for awhile, but the problem will go away given time. And I, for one, look forward to buying futuristic data storage without feeling a little cheated.

Re:Preempting the prefix war

[Dachannien]

Okay, you win :)

Re:Preempting the prefix war

[Profane MuthaFucka]

Ignore the lame and contrived kibi and tebi nonsense. Just call it a disk-Meg, or a disk-Gig, or a disk-Tera. You don't even have to type extra characters to say what you mean. Disk-gig is 8 characters, and so is gibibyte.

Re:Preempting the prefix war

[sik0fewl]

The real problem is that "terabyte" already has the definition of 2^40 bytes. Yes, we clearly fucked up when we decided 1024 bytes was a "kilobyte", but it's way too late to change that now. We used an SI prefix, but when any SI prefix prefixes "-byte" it's no longer an SI prefix, I'm afraid. "Kilo-" means *1000 in SI, but not in bytes.

Now we've gone and created *another* definition for 2^40 bytes, called a "tebibyte". How does this remove the confusion? It doesn't. We still have the ambiguous term "terabyte" (thank you, hard drive manufacturers!).

Perhaps tebibyte should have been defined to mean 10^12 bytes and hard drive manufacturers could use that term when they want to exaggerate.

Re:Preempting the prefix war

[DeathCarrot]

Yeah, but now the problem is that KB is now ambiguous Nonsense! I think you'll find the Kelvinbyte is a strictly defined unit.

Re:Preempting the prefix war

[martyros]

Maybe a linguist can pitch in to explain why tebibyte sounds so awful?

I don't care how it sounds, it's just a pain in the neck to pronounce the two 'b' sounds in a row. In England they're to lazy to pronounce the "chest" in Leichester, and pronounce it "Lie-ster". Here in Cambridge, they're too lazy to prounounce Mag-da-lene in Magdalene College, and pronounce it Mawd-lin. Do you really expect English speakers to say "Mebbbibbbyte"? It's not going to happen.

Re:Preempting the prefix war

[swillden]

We used an SI prefix, but when any SI prefix prefixes "-byte" it's no longer an SI prefix, I'm afraid. "Kilo-" means *1000 in SI, but not in bytes.

Except when it doesn't. Like the way hard drives have been, since the time when computer memory sizes were talked about in "words" and "characters", and the SI prefixes used for random access memory were also powers of 10.

And what about when it prefixes "bit"? When we talk about kilobits, megabits, etc., it's usually in the context of data communications, where the prefixes are powers of 10. When we talk about low-level flash and RAM sizes, though, the same prefixes are powers of 2.

It's stupid to overload the prefixes and try to guess their meaning based on context. Tebibyte and gibigyte only sound funny the first hundred times you say them. After that, they're just natural, and precise.

Re:Preempting the prefix war

[NitroWolf]

I wish I had mod points to mod you up. The "new" prefixes are ridiculous. I usually am all for clarity and useful jargon, but the new prefixes suck. They sound terrible and they don't really serve a very useful function for 99.9% of the people who use the words anyway.

Yes, they provide a slight bit of clarity, but in the instances that they do provide it, it's just as easy to say "A binary TB" to differentiate it from a decimal TB. It gets the point across and it doesn't add useless complexity to the jargon.

Re:Preempting the prefix war

[gr8dude]

Check out the kiki-effect; the concept is pretty much the same.

Re:Preempting the prefix war

[kryptkpr]

Are you not aware of the Single, Definitive standard?

Re:Preempting the prefix war

[Grishnakh]

It really didn't start out that complicated, but it's the manufacturers who keeps F*ing it up because they are trying to stretch the numbers. I think it's the consumers who are the victims in this.

Exactly right.

Hard drives have the MiB-MB problem because manufacturers wanted to be able to say 60GB instead of 54GB.

Right. They shouldn't have been allowed to do that. Up until that time, everyone was doing just fine counting KB, MB, etc. in powers of 2, because it makes sense to do that with computers. But the HD makers got greedy and their evil marketing people wanted to muck things up by changing the definitions.

I've also heard that early monitors were measured by their horizontal instead of their diagonal. Of course, the diagonal is longer so...

It's worse than that. The early monitors were measured by their horizontal dimension instead of their diagonal because they could advertise a LARGER number. How's that? Simple: they measured the entire case, not just the screen! So they had a bunch of monitors with excessively large cases, relative to their screens, so they could advertise large screen sizes.

The solution to all this is simple: execute all marketing people, and put the engineers in charge of advertised specifications. Marketing people are leaches and have never done anything good for the world, and this is yet another example of them screwing things up for everyone else.

Re:Preempting the prefix war

[Grishnakh]

Instead of "binary" and "SI", why not just "terabytes" (implying binary) and "marketing terabytes"? That's what this is all about, after all: marketing people want to advertise larger numbers instead of telling the truth.

Re:Preempting the prefix war

[sik0fewl]

It's stupid to overload the prefixes and try to guess their meaning based on context.

I agree, but it's already happened.

Tebibyte and gibigyte only sound funny the first hundred times you say them. After that, they're just natural, and precise.

Apparently they're also difficult to spell :). Unforunately, terabyte and gigabyte will still be ambiguous.

Re:Preempting the prefix war

[swillden]

Tebibyte and gibigyte only sound funny the first hundred times you say them. After that, they're just natural, and precise.

Apparently they're also difficult to spell :). Unforunately, terabyte and gigabyte will still be ambiguous.

Oops :)

Assuming I spell them correctly, they're never ambiguous when I say them. Increasingly, Linux utilities are careful to use the appropriate prefix (e.g. GB vs GiB), and HDD manufacturers are starting to provide both values, with appropriate units.

If enough people begin begin careful to distinguish, in a few years the ambiguity will have disappeared.

Re:Preempting the prefix war

[rts008]

Hey! He does distance conversions for NASA....using Excel, you insensitive clod!

Re:Preempting the prefix war

[zsau]

language is a tool, to facilitate written and oral communication.

Precisely, so it's awkward when people in one field usurp terms with confusing new meanings. Everyone knows a kilometre, kilograms, kilolitres, kilo- this, kilo- that. Why should kilobytes be any different?

Maybe a linguist can pitch in to explain why tebibyte sounds so awful?

It's your own personal opinion, but the repeated /b/ sound is probably what's causing you distress though. Repeated (or similar) syllables are frequently skipped in many languages at different times in a process called "haplology" (which could do with some haplology itself).

For instance in English, most people (I speak to) say words like "particularly", "library", "similarly", without the "ar" syllable (so "particu'ly, libr'y, simi'ly"), ro "probably" with only one b (so "prob'ly"). If you think "tebibyte" sounds bad, you might want to do the same thing to that and have "kibbyte, mebbyte, gibbyte, tebbyte".

In any case, it's important that kilo- and other prefixes have only one meaning. It should be the original meaning i.e. 1000. Usage will sort out any infelicities of pronunciation before long, as long as no-one's too caught up in our spelling system. After all, this is English. Spelling has practically nothing to do with pronunciation.

Wait, what?

[jesdynf]

Did you see the section on timestamps? Nanosecond resolution out to 2514.

Nanoseconds.

We're dealing with a process whose maximum useful precision is "has the green light gone off yet", and we've got nanosecond timestamps.

Re:Wait, what?

[Uncle Focker]

The nanosecond resolution is there for mission critical systems that need a finer resolution than seconds.

Re:Wait, what?

[VeNoM0619]

Seeing how processing becomes faster and faster: in batch file processing (just an example) where tens of files get processed in a single second. It may be useful to know which files processed in what order, in which case the precision could be useful. Think of it more as a feature than a necessity though I suppose.

Re:Wait, what?

[jesdynf]

I'm having trouble with that one. I mean, the statement would've been true without invoking "mission critical" -- of course the ability to get resolution better than seconds is for applications that need resolution better than seconds. I'm not sure why you've invoked the specter of "mission critical" here, and I'm having a damn hard time picturing any utterly important, world-ending task that's going to (a) rely on the *timestamp in the filesystem* and (b) run on Linux and ext4fs.

And the timestamp isn't in milliseconds (an incremental improvement that wouldn't've even raised eyebrows) or microseconds (which would have been future-proofed overkill) but nanoseconds. I know how PC clocks work -- you'll have a hard time convincing me that you can maintain nanosecond timing long enough for the difference between two nanosecond timestamps to be accurate down to the nanosecond.

Re:Wait, what?

[Waffle Iron]

They're probably using a 64-bit number to hold the timestamp. That gives you 1.8e19 discreet time intervals, so you're going to get ridiculous precision, dates ridiculously far into the future, or both. I assume that they went for precision because that arguably has more potential for use in the real world than worrying about files thousands of years into the future.

IIRC, today's PCs have high-resolution timers available that surpass the old 14.318MHz clock chip. If you can't get accurate nanoseconds out of the timers yet, they'll just round the numbers off. No big deal.

BTW, NTFS uses 100ns timestamp granularity, and it was designed when systems were almost 100X slower than today. So it had a similar amount of overkill, but that certainly doesn't seem to have had any negative impact on the acceptance of NTFS.

Re:Wait, what?

[flnca]

Nanoseconds isn't that high a precision, when you think about the fact that everyone can buy a computer with 2.0 GHz nowadays, which would be a clock cycle of half a nanosecond (500 picoseconds).

Even my old Amiga 1000 computer had a timer with 2 nanosecond precision in 1986 with a CIA 8520 chip.

High-precision timestamps are used in cases when time stamps of files have to be compared to detected changes, as in the ubiquitous "make" program, which uses timestamps to detect whether a file has been modified.

The same applies to some scripting situations ...

Anyway, the resolution is also geared at the big irons, on which processors run that exceed the teraflop range (1 THz cycle duration = 1 picosecond).

Re:Wait, what?

[flnca]

BTW, every IA-32 compliant CPU (PI,PII,III,4,etc. or Athlon etc.) has an instruction called RDTSC, which reads the CPU clock tick counter into a 64 bit register. This creates a timing facility that has a precision as high as the CPU clock. So 1 GHz clock cycle = 1 nanosecond precision.

Re:Wait, what?

[Chris Mattern]

Drive access is a red light.

Actually, on my system, it's a yellow one, just under the green power light.

Re:Wait, what?

[flnca]

Yeah, it's not precise enough for some applications. picoseconds, or femtoseconds would've been more reasonable. Also, why not make a time stamp 128 bits instead of just 64 bits? Those 8 extra bytes wouldn't have made much of a difference. So, we'll have to wait for ext5 to support that ... ;-)

Re:Wait, what?

[flnca]

Only, that in some applications, it's not tens of files per second, it's thousands, or hundreds of thousands of files per second, as when you're checking and/or updating lots of small files, like source trees (for programs or web sites).

Re:Wait, what?

[techno-vampire]

you'll have a hard time convincing me that you can maintain nanosecond timing long enough for the difference between two nanosecond timestamps to be accurate down to the nanosecond.

Not now, and not in the near future, sure. However, who's to say that it won't happen, possibly sooner than we think? The developers had the room to store times that accurate, so they probably just put it in to allow for future developments.

Re:Wait, what?

[skulgnome]

However, on a Core 2 chip (and presumably its descendants), a RDTSC is sometimes undone by the speculative execution scheduling unit and can, in some circumstances, report time going backwards. Not to mention that on a dualcore processor these counts are not strictly in sync, and do not tick forward predictably enough to be used as a clock.

Good idea though. Unworkable in practice, but good idea.

Re:Wait, what?

[bfields]

IIRC, today's PCs have high-resolution timers available that surpass the old 14.318MHz clock chip.

Last I checked the actual time source used for file timestamps was actually jiffies, so even though the filesystem inode may in theory have space for lots of precision, in practice the resolution is only hundredths of a second.

That's a problem if you're trying to use the timestamp to decide whether a file has changed or not--if you happen to check the time between two writes that come within (say) a millisecond of each other, then you may not ever see the second write. (E.g., consider the case where you're an nfs client trying to check whether your cached data for a file is still up-to-date.)

Re:Wait, what?

[flnca]

It's possible to be used regardless if you set the thread affinity mask to a particular CPU. Then the RDTSC time stamp is quite usable. In fact, I've used it for over a decade for high-precision timing, especially on Windows with its awkward task scheduling (the performance timers in Windows often aren't sufficient). I've never encountered that time going backwards thing yet, thanks for mentioning it.

Re:Wait, what?

[Sentry21]

BTW, NTFS uses 100ns timestamp granularity, and it was designed when systems were almost 100X slower than today. So it had a similar amount of overkill, but that certainly doesn't seem to have had any negative impact on the acceptance of NTFS. Yeah, amazing but true, NTFS is actively used by as many operating systems as Ext3 is. :p

Re:Wait, what?

[AndrewHowe]

Speculative execution can cause your RDTSC to give you odd results, for sure. It's normal to do a CPUID first to serialize everything. But I'm quite surprised that time would go backwards on a single core. The minimum sequence would be something like:
rdtsc
mov ebx, eax
mov ecx, edx
rdtsc
(sub eax, ebx
  sbb edx, ecx)
I can't quite imagine the second rdtsc being executed before the first, even with out of order execution. But I'd love to know if anyone has ever seen it happen :)

Re:Wait, what?

[Crayon Kid]

...doesn't seem to have had any negative impact on the acceptance of NTFS.

Yeah, like there was a choice. What were people supposed to do if they had issues with NTFS, use something else with Windows? Like FAT?

We're really spoiled for choice under Linux, by the way.

Re:Wait, what?

[dotancohen]

On my Dell laptop, drive access is green like power. Battery can be either green or orange, depending on state.

To all ext3 users...

[c0l0]

...who are on the lookout for a new fs to entrust with keeping their precious data: make sure to check out btrfs ( http://oss.oracle.com/projects/btrfs/ ). It's a really neatly spec'd filesystem (with all the zfsish stuff like data checksumming and so on), developed by Oracle employees under GPLv2, which will feature a converter application for ext3's on-disk-format - so you can migrate from ext3 to the much more feature-packed and modern btrfs without having to mkfs anew.

On a related sidenode: I'm very happy with SGI's xfs right now. ext\d isn't the only player in the field, so please, go out and boldly evaluate available alternatives. You won't be disappointed, I promise.

Re:To all ext3 users...

[DJProtoss]

I agree btrfs looks nice, but its somewhat behind ext4 in terms of implmentation and stability (which is saying something) - theres the small matter of not yet handling E_NOSPACE, for instance

Re:To all ext3 users...

[miscz]

The only thing that keeps many people using Ext3 is availablity of drivers on operating systems on Linux and clear upgrade path. Being supported by third-party partition management tools helps a lot too. I can ready my data on Windows (Ext2 IFS for Windows is awesome), OSX (though the only driver doesn't have write support) and more.

I don't want to entrust my data to a single OS.

Re:To all ext3 users...

[swilver]

There is no way I'm installing anything Oracle on my Linux system ever. I will definitely not entrust my data to them after having witnessed over the past years what a mess their flagship product is.

Re:To all ext3 users...

[Anonymous Coward]

I'm an XFS fan as well. I have been using it for years. I usually have my root/boot partition as ext3 (so grub works) and all data on XFS.

XFS kills ext in terms of not losing data. I have recovered lots of data from failed drives that were XFS formatted. Not so with ext3 which tends to flake out and destroy itself when it gets bad data.

And don't even mention ReiserFS, that has always sucked. I have lost more data to Reiser than any other filesystem (ext is a close second though). Sometimes it would corrupt files just from rebooting the machine. I have never lost data on an XFS partition that wasn't due to hardware failure.

Re:To all ext3 users...

[sconeu]

I'd use JFS, but then I'd have to pay SCO money....

Re:To all ext3 users...

[EvilRyry]

btrfs is a completely new ground up filesystem. I would expect it to take a while longer than ext4 which is just another incremental improvement on ext2. btrfs isn't stabilized at all yet. I would consider the running out of space issue a non-issue at their current stage in development.

Re:To all ext3 users...

[johannesg]

I notice they are happily throwing benchmarks around, which is funny considering that Oracle will not allow benchmarking at all for their flagship product...

Re:To all ext3 users...

[piojo]

I agree. Around the time when SuSE had ReiserFS as the default filesystem, it still had a data-eating bug (at least, it did when I used it on slackware).

I like JFS a lot, because it seems to perform well on many small files, and if one does a lot of compilation or other management of small files, it's pretty nice. XFS seems to win the benchmarks for operations involving large files, but I find that on the occasions that I use huge files, I don't care so much about speed. JFS has made me happiest, among the filesystems I've used.

Re:To all ext3 users...

[Jherek Carnelian]

btrfs -- How fast are deletes?

ext3 is both so slow and so bottlenecked that mythtv had to implement a special "slow delete" mode which gradually truncates files instead of just unlinking them. Without the "slow deletes" mode, you get hiccups in any shows that are being recorded while old shows are deleted.

On my system, deleting a 20GB file can take a minute on ext3 (and the filesystem is completely locked - all other processes are blocked), but on ntfs it is almost instantaneous.

Re:To all ext3 users...

[DJProtoss]

indeed, but I think you will agree, its possibly not quite something for people "who are on the lookout for a new fs to entrust with keeping their precious data" [parent] - well, unless you don't intend to be doing the entrusting until a few years down the line. - it may well end up superior to ext4, but it isn't *currently*

Re:To all ext3 users...

[cecom]

I completely agree.

Re:To all ext3 users...

[glwtta]

On a related sidenode: I'm very happy with SGI's xfs right now.

I seem to be plugging XFS in every fs thread recently, so I'll second that - I'm really surprised it's not more popular.

ext3 may have, more or less, caught up to XFS in IO speed recently, but file operations on large filesystems are still a disaster - just try deleting a 2TB tree with a couple million files in ext3, I dare you.

Re:To all ext3 users...

[manitoulinnerd]

I moved to XFS from Reiser3 and ext3 because of file delete times coming up on 2 years ago. XFS has lightening fast deletes of large files over Reiser3, I was blown away. I only use it on a data partition (/home) and have left the rest of my system on reiser3 or ext3. XFS has a great set of tools and is stable. I see no reason not to use it (at home at least).

Re:To all ext3 users...

[jabuzz]

I has not always done so. Therefore if at some point in the past you found out it did not work, then you would tend to think it still does not. My Debian Etch installation is using lilo for the very reason that it is pure XFS.

Re:To all ext3 users...

[pankkake]

Better solution: have a /boot partition, which is the default on Gentoo. Your /boot partition (under 40 mb) will be ext2, and the rest will be whatever you want.

Step 1

[vga_init]

Step 1: Install Fedora 9

OK, all done!

undelete

[Nimey]

Oh, please. ext2 had "undelete" capability, just as it had filesystem compression capability. Neither were ever implemented.

Re:undelete

[jsm300]

Huh? There are undelete tools for ext2. Undelete is impossible for ext3 since the information needed to do it is gone immediately once a file is removed, whereas it will still be present in a ext2 file system until it gets overwritten as new files are created.

Indulging the prefix war

[JustinOpinion]

Of course, in this case you have to balance the confusion stemming from the Tera in IT context meaning 1024 in some cases. It's worse than that. According to SI prefixes, "Tera" should mean 10^12 (1,000,000,000,000), but in common usage applied to computers it sometimes means 2^40 (1,099,511,627,776). But it also sometimes means "1024 Giga", where the Giga could be using either convention (and, for all you know, the "Mega" implied within could have been computed using either convention). So you can get a gradient of "mixed numbers" that conform to neither standard. You might say that only a non-professional would make such a stupid mistake... but on the other hand, if you see a column of numbers listed in "Gigabytes" and you want to convert them to Terabytes, what conversion factor would you use? How would you know what conversion factor the previous author had used? How could you guarantee that you were doing it right? Would you be able to confidently convert it into an exact number of bytes?

Personally, I think the whole thing is a mess, and computer professionals should be working harder to enforce a consistent scheme. Unfortunately, only a minority of computer professionals seem interested in changing the status quo confusion.

Maybe a linguist can pitch in to explain why tebibyte sounds so awful? I'm no linguist, but I don't think "Tebibyte" sounding silly is the real problem. I admit that I laughed when I first heard the binary prefixes. They sound lame. But who cares? "Quark" was silly when it was first coined. So was "Yahoo" and "Google" and "Linux" and "WYSIWYG" and "SCSI" and "Drupal" and so on... Silly names become second-nature once they are used enough.

I think the real problem is that people, inherently, are loathe to change. They are more apt to come up with rationalizations and justifications for doing things "the old way" rather than put in the work to learn (and code!) a new system. Sorry if this sounds harsh, but I find the people who say the binary prefixes "sound dumb" or say that "the current (inconsistent)* system works fine" are just coming up with excuses to avoid doing the work to use a properly consistent standard/notation.

Maybe you're right, and that if the new prefixes had sounded "cooler", then adoption would have been faster... but I'm not so sure. Even if true, it doesn't absolve any of us for allowing the confusion to persist: cool or not, we (geeks especially!) should have the discipline to use proper standards.

* The current system can be roughly described as: SI prefixes are powers of 10 everywhere except in computer science, when they become powers of 2. But only when referring to memory, and some data structure sizes, but not when referring to transmission rates or disk space (unless it's a flash drive, sometimes), and other kinds of data structures.

Re:Indulging the prefix war

[sootman]

But it also sometimes means "1024 Giga", where the Giga could be using either convention (and, for all you know, the "Mega" implied within could have been computed using either convention). So you can get a gradient of "mixed numbers" that conform to neither standard. You might say that only a non-professional would make such a stupid mistake... but on the other hand, if you see a column of numbers listed in "Gigabytes" and you want to convert them to Terabytes, what conversion factor would you use?

My 7th grade science teacher, when we were learning the metric system (this was in the mid-80s... any day now!) was really picky about how we did conversions. If you were going from millimeters to kilometers, he'd say convert mm to meters, then meters to km. Feet to meters was feet to inches, inches to cm, cm to meters. So same here--just go all the way down to bytes, then all the way back up. :-)

Re:Indulging the prefix war

[complete loony]

Personally, I think the whole thing is a mess, and computer professionals should be working harder to enforce a consistent scheme. Unfortunately, only a minority of computer professionals seem interested in changing the status quo confusion. A minority of computer professionals defining 1KB = 1000B when everyone else is using 1KB = 1024KB only *adds* to the confusion. Maybe we should stop using the word "byte" when counting storage in base 10?

Re:Indulging the prefix war

[dubl-u]

Personally, I think the whole thing is a mess, and computer professionals should be working harder to enforce a consistent scheme. Unfortunately, only a minority of computer professionals seem interested in changing the status quo confusion. That's because it doesn't matter.

The only time I hear a real-world complaint that this would address is the way disk manufacturers measure in one number, but your OS measures in another, so noobs get confused, feeling rooked. But this is only a partial solution anyhow, in that the OS and filesystem will eat up all sorts of bytes for all sorts of things, so adding new units to the mix is not going to help the microcephalics who get their technical advice from a Best Buy salesdolt.

When I, as a technical person, am doing any kind of math with these numbers, it also doesn't matter. Either I'm doing back-of-envelope numbers, in which case the small gain in precision is lost in the handwaving. Or I'm doing math that matters, in which case I will use bytes all the way through, and fuck the prefixes.

So in sum, to me it's a lot of pedantry for very little practical gain.

But does it undelete...

[swilver]

That's all fine and dandy, but will it allow me to somehow undelete/recover when I accidently type rm -Rf /hugedir -- yes I know there are other ways to delete stuff, I just find it ridiculous that all linux file systems with the exception of ext2 make no effort at all to be able to recover from such a common mistake. Of course, rm not giving any indication at all about how many bytes and files it is about to remove doesn't help either.

Re:But does it undelete...

[mlwmohawk]

The whole "undelete" thing is a DOS FAT stupidity. The *only* reason why people think that you *can* undelete is that the DOS FAT file system was designed in such a way that file changes could be recovered *IF* you managed not to change the file system too much. DOS being a mainly single tasker, with the exception of the standard "indos" flag games.

POSIX was not and should not be designed in such a way that "undelete" is reliably possible. That's like saying can I unlight that match. Can I unbreak that egg?

An unreliable system that may, on the odd chance that the file structure has not changed too much, recover files from a disk that have not been over-written yet is no replacement for NOT being an idiot and being careful when you delete something.

Re:But does it undelete...

[Hatta]

That's not the file systems job, that's the tool's job. You'll find on windows that when you use 'del' to delete something, it doesn't end up in the recycle bin.

So if you want some sort of soft delete, don't use rm or del. Use a tool that 'soft deletes' a file by moving it into a trash bin, which you can 'hard delete' when you need more space. This is how Windows and KDE both work.

Personally, I think file systems aren't aggressive enough when it comes to deleting files. When I delete something I want it well and truly gone.

Re:But does it undelete...

[EvanED]

So if you want some sort of soft delete, don't use rm or del. Use a tool that 'soft deletes' a file by moving it into a trash bin, which you can 'hard delete' when you need more space. This is how Windows and KDE both work.
Know of a command line tool that fits this description that comes with most *nix distributions?

Re:But does it undelete...

[blitzkrieg3]

I don't know about you, but I rarely every have to type -f. Usually I only do that after I can't do it with a regular rm, in which case I'm considering not deleting /hugedir in the first place.

Re:But does it undelete...

[Anonymous Coward]

Perhaps you should try prm (pansy rm) or psh (pansy shell).

Re:But does it undelete...

[swilver]

I don't expect it to manage "deleted" files at all. It's that sinking feeling you get when you think you're deleting a directory that should be mostly empty, and rm is taking longer than expected (as your only indication that something is horribly wrong.

It's not unreasonable to expect to be able to undo that action when I immediately press cancel it and make sure the drive is not written to anymore. Ext2 can do this easily. Ext3 goes out of its way to make this impossible. XFS/ZFS/ReiserFS etc.. all make it way too hard or impossible.

I'd alias rm to something with a more sane cmdline interface, but that would break every shell script in existance.

Recycle bin solutions are crap. I want stuff to hang around for 30 minutes at most, not weeks on end taking up valuable free space causing unnecessary disk fragmentation.

Re:But does it undelete...

[swilver]

Fine, assume I'm an idiot then. No expert would ever want such a feature, or expect to be able to recover files in some way after they had made a mistake, even if that takes taking the drive offline immediately and having it perform a full disc scan.

Re:But does it undelete...

[Xtravar]

mv file.txt ~/.Trash/

That's essentially what any shell does. Personally, I find it annoying and do all my deleting from the command line.

Re:But does it undelete...

[trolltalk.com]

So if you want some sort of soft delete, don't use rm or del. Use a tool that 'soft deletes' a file by moving it into a trash bin, which you can 'hard delete' when you need more space. This is how Windows and KDE both work.

Know of a command line tool that fits this description that comes with most *nix distributions?

alias rm='mv $1 /home/$USER/archives/$1'

Re:But does it undelete...

[EvanED]

mv file.txt ~/.Trash/

This suggestion is broken for a few reasons.

First, it's just obnoxious to add the ~/.Trash/ at the end.

Second, it would be really easy to forget ~/.Trash/. If the final thing you are "deleting" is a directory, this would mean that it would move everything into that directory.

Third, it doesn't record the restore location. It looks like the Gnome trash doesn't either; this is very surprising to me, and I consider it broken. Even Windows 95 recorded the location it came from, so you could right click and choose restore and it would go back where it was. Mac OS also behaves this way.

Fourth, it isn't sensitive to file systems. So for instance, deleting a file on a file system other than the one ~/.Trash/ is mounted on will result in a long copy over to that file system, which may also cause problems with available space.

Personally, I find it annoying and do all my deleting from the command line.

To each his own I guess. I do a lot of work from the command line, and I find it annoying that I *don't* have a good way to handle this.

Re:But does it undelete...

[EvanED]

alias rm='mv $1 /home/$USER/archives/$1'

This fixes the first two problems I give here, but the third and fourth remain.

You have to do rather more bookkeeping. I do plan on writing a program that will make this all work "correctly", but I'd like to make sure that I'm not duplicating work.

Re:But does it undelete...

[drinkypoo]

Know of a command line tool that fits this description that comes with most *nix distributions?

Not only does it not exist, but if you run out of disk space, you run out of disk space.

Here's a stupid question for the slashdot populace: Why are all operating systems so bad at handling the process of deleting files?

It seems like it would be bog simple (disclaimer: I am not much of a programmer... salt at the ready) to remap unlinking of a file to relinking it to a trash directory, something like $HOME/.Trash (there's per-volume trashes too, but I always forget where those live. Probably on the volume) :P Then, and this is the important part: 1, when free space is requested, files in the trash are not shown in the quota and 2, when a file operation requires disk space, instead of failing it, we unlink some files for real based on some sort of reasonable algorithm.

Why, why, WHY does no one do this? It makes no sense. It would save everyone from ever having to empty the trash. When files have metadata, or based on metadata which can be generated from files programatically, files can be assigned scores and deleted in order of rank... or on whatever basis the user likes, you can select files to delete and return their inode numbers with a shell script for all I care. Just fall back to some reasonable heuristic if the user space tool fails.

Anyway, if you want to move files to the trash yourself, it seems like a simple shell script would do the job. You could also use find -exec (read the manpage for find.) to do all kinds of fun selective "deletions". There must be some way to instruct nautilus to trash a file with dbus, which would surely be the best way to go about it... but I have no idea what that would be. While looking around I found libtrash (this is about using it actually, not a direct link) which does some of what I want; I also found Gvfs which provides unified trash; it looks like Nautilus (GNOME's file manager) has trash functions in a private library, I don't know if that's built shared but it might not be too hard to use the same functions to trash files.

In order to delete all your potential trash files you have to do the following (GNOME):

1. sudo rm -rf $HOME/.Trash/*
2. sudo rm -rf $HOME/.local/share/Trash/*
3. for each mounted filesystem, do the following:
   if [ -d "${filesystem}/.Trash-${USER}" ]; then sudo rm -rf "${filesystem}/.Trash-${USER}" ; fi

To put trash into the trash properly... that's much more complicated, because files with the same names are removed. I think you can make nautilus empty the trash via dbus but I don't know about anything else, I don't think so.

It seems like you could get this functionality at the command line and in the GUI by installing libtrash, and wrapping a script around the 'rm' command that would set LD_PRELOAD to load libtrash. But I haven't tried it.

Re:But does it undelete...

[EvanED]

This suggestion is broken for a few reasons. ...

Fifth, if you delete two files in different directories with the same name, both can't exist in the .Trash directory at the same time.

Re:But does it undelete...

[r_jensen11]

mv

Re:But does it undelete...

[quanticle]

How hard would it be to write a simple shell script that can do this for you?

In bash, for example:

#! /bin/bash
mv $1 ~/.Trash/

Re:But does it undelete...

[EvanED]

*sigh*

This has already been suggested a number of times...

Anyway, it's rather harder than you think. (Read my third and fourth reasons from that post, and the fifth from my followup. Three reasons why a naively simple script like this doesn't work very well.)

Re:But does it undelete...

[Bob The Cowboy]

Try rm -I. I know that's not exactly what you wanted, but maybe it'd help.

Re:But does it undelete...

[kasperd]

The whole "undelete" thing is a DOS FAT stupidity. The *only* reason why people think that you *can* undelete is that the DOS FAT file system was designed in such a way that file changes could be recovered *IF* you managed not to change the file system too much.

That is actually not the case. The FAT table is used both as a linked list of the contents of a file, and as a free blocks "bitmap" where a special value would indicate a free block. That implies that to delete a file, the list of blocks used by that file would have to be overwritten with this special value. So after a file was deleted all information about where it is located on the disk is lost, except from the first block, which was recorded in the directory entry. What undelete utilities did was to assume the file was not fragmented, and simply take the first free blocks after the first one and pretend they were the file. If the file had been fragmented, it would be corrupted after undeletion. On ext2/ext3 you can undelete a file much more reliably because the free blocks bitmask and the pointers to actual blocks are separated. Though I don't know if the list of the first 12 blocks, which is in the inode, has to be wiped when the file is deleted.

I have successfully undeleted a 1GB file from an ext3 file system a month after it had been deleted on a file system that had writes happening all the time. But maybe I was just lucky.

Re:But does it undelete...

[manitoulinnerd]

Speaking as someone who has done just that it is not fun. But recovery is possible. I was using Reiser3 at the time and the included tools and a Gentoo tutorial made recovery of all but the most recent files no problem. It lost filenames though. I am lazy right now so you will have to find a link on your own. Joel PS - Don't write anything to the drive. The blocks are unallocated and if you write anything your are risking losing the data for good.

Re:But does it undelete...

[steveha]

It seems like it would be bog simple (disclaimer: I am not much of a programmer... salt at the ready) to remap unlinking of a file to relinking it to a trash directory, something like $HOME/.Trash (there's per-volume trashes too, but I always forget where those live. Probably on the volume) :P Then, and this is the important part: 1, when free space is requested, files in the trash are not shown in the quota and 2, when a file operation requires disk space, instead of failing it, we unlink some files for real based on some sort of reasonable algorithm.

Hear, hear. I second this.

You know why I want this so much? Because I used to have this. When I ran Windows 98 with Norton Utilities I had a feature called the "Norton Protected Recycle Bin". It had the following properties:

0) When a file was deleted by any means it would go into the NPRB. If you copied file "foo" onto file "bar", the old "bar" would go into the NPRB. If you went into a DOS shell and ran the "del" command, the files you deleted would go into NPRB.

1) The original name, original location, timestamp of the file at the time of deletion, and time of deletion were all saved and were visible in the list of deleted files.

2) You could specify a do-not-save list, both by filespec (example: do not save "*.tmp", "*.bak", etc.) and by location (example: do not save any files in C:\Windows\Temp, etc.) And sensible defaults were provided (all my examples here were set by default, plus more).

3) Your deleted files would age out. It would automatically purge files that were deleted more than 14 days ago (by default; of course you could change it). I liked to keep mine set for 3 days auto-purge; I pretty much always un-deleted something seconds after deleting it, so the longer period didn't help much. And disks were small in those days...

I don't think it would ever empty the NPRB automatically just because you were out of disk space. But it was trivial to right-click on the Recycle Bin icon on the desktop and choose "Empty Norton Protected Recycle Bin".

To do the above properly in Linux, the file system should have a way to specify whether a given file should be saved or simply deleted. The Ext2 file system has an attribute you can set on a file requesting that it be saved instead of deleted; read the man page for chattr(1) and look for the 'u' attribute. (Then read the "Bugs" section at the end, where it says that the undelete feature has never actually been implemented.)

The file system should handle the deletion or saving of files, and then user-space utilities should be able to get a list of saved files, undelete a saved file, empty the saved files, etc. Of course users should only be able to delete their own files, so ownership and permissions need to be checked too.

With this feature, and today's huge hard disks, a Linux desktop would be great for ordinary users.

P.S. If you have Vista, and you get Norton Utilities, is there still a Protected Recycle Bin feature? Or is that one of the things that Vista loses? For that matter, can you still do this in XP?

steveha

Re:But does it undelete...

[kasperd]

remap unlinking of a file to relinking it to a trash directory, something like $HOME/.Trash

Technically that could be done, but I can come up with a number of reasons why you might not want to implement it.

1. It might not be compliant with the posix standard, but I don't know for sure.
2. It screws up the algorithms to avoid fragmentation. If you want to keep files around until you are absolutely out of space, it means block allocation will have less freedom when choosing locations for new files. Less free blocks to choose from means it is more likely to fragment your file system.
3. Actually figuring out the right per-user trash directory is not something you would want to happen inside the kernel. Sometimes the original location is not even known once the file is deleted. You could have hardlinks from multiple users' home directories, then delete all of them while the file is actually open. Once the file is actually deleted, it doesn't even exist in the file system.
4. Having quota behave differently depending on where a file is located is problematic. It is hard to update efficiently and correctly when directories can be renamed and files can have multiple hardlinks.

Re:But does it undelete...

[Blakey Rat]

Please.

The greatest feature of modern software is "Undo." Everything I can screw up on the computer should have an Undo-- that's what the Recycle Bin (or Trash Can for Mac users) is there for, although it's a bit more awkward than pressing control-Z.

Call it stupidity if you want, but my system files (you know, the ones that file permissions actually protect from malware) are worth approximately zero, and my personal files (the ones that malware can delete no question asked) are worth hundreds of man-hours, if not thousands, and ten times that in dollar value.

Windows Shadow Copy has an exact template on how to implement it, now go implement it.

(And yes, I keep backups, as should everybody. But there's no excuse not to use spare disk space as another layer of defense.)

Re:But does it undelete...

[lucas teh geek]

more importantly, the suggested script stops working correctly as soon as you start using wildcards.

rm-script *.jpg

that script will only delete the first file matching the wildcard because the shell does the wildcard expansion, while arguments $2 onwards are not affected. you could write a more complex script, but it still wont solve the other problems you pointed out

Re:But does it undelete...

[Trogre]

No, no, and a thousand time NO!

Undelete might not be a problem for a lot of people. In fact some security-paranoid people might like it that way, but that's what 'shred' is for, isn't it?

For me however, it's a dealbreaker. No matter how much I educate my users, someone at least once a year manages to delete a file they didn't intend to. So what, you might say, just recover from backup. No good if that file is one they've been working on that day, and backups are done nightly. Of course, RAID is of no benefit here since this is on the filesystem level. It is a reasonable expectation to be able to 'undo' many things these days. Even for us sysadmins - I can happily type rm -Rf /usr/local/oldprogram, but if a finger slips and I get rm -Rf /usr/ local/oldprogram, it's fatal. I guess that makes me an 'idiot' too. Then again I do occasionally need to press the BackSpace key on my keyboard, so that pretty much confirms it. Why on earth should unlinking a file be like breaking an egg?

IMO the ability to undelete files is a basic requirement of any production-level filesystem. Okay, granted that EXT3 recovery isn't anywhere near as easy as I'd like it to be (nowhere as good as EXT2 for example), but it is for the most part possible.

Re:But does it undelete...

[Peter H.S.]

KDE allready has a trashcan or a "Protected Recycle Bin". If you choose so, all files user deleted within the KDE framework goes to the Trashcan instead, there is of course a slight speed penalty for doing this when deleting lots of files. But since it doesn't work at the system level there are no penalties in e.g. speed when the OS performs deletes of temporary files. So basically the ordinary KDE users using a GUI can have all the delete protection the user wants no matter what filessystem is being used.

--
Regards

Re:But does it undelete...

[apt-get moo]

If you get into this situation that often, make rm an alias for 'rm -i'. The interactive flag overrides the force flag, so you'll have to confirm every deletion. Or better yet, be more careful

Re:But does it undelete...

[Pig Hogger]

In order to delete all your potential trash files you have to do the following (GNOME):

1. sudo rm -rf $HOME/.Trash/*
2. sudo rm -rf $HOME/.local/share/Trash/*
3. for each mounted filesystem, do the following:
if [ -d "${filesystem}/.Trash-${USER}" ]; then sudo rm -rf "${filesystem}/.Trash-${USER}" ; fi

Er, no.

Sorry, but no way I'm gonna have a script that contains "sudo rm -rf" on my system...

Re:But does it undelete...

[Saxerman]

You're not asking for something you want in the file system. In order to 'undelete' something, you need to have not actually deleted it in the first place. Once you 'really' delete something, you want that free space to become immediately available. If you want to keep around 'deleted' files for awhile, before you actually remove them from disk, you just want the "trash can" style functions, where you move 'deleted' files off to a trash can directory where they will eventually be 'really deleted' after a specific period of time. Which, if you really want, you could add by creating a simple shell script to sub out your rm command, and a cron entry. Or you could just use whatever similar functionality already exists for your distro.

Re:But does it undelete...

[jimmyharris]

The greatest feature of modern software is "Undo." Everything I can screw up on the computer should have an Undo-- that's what the Recycle Bin (or Trash Can for Mac users) is there for, although it's a bit more awkward than pressing control-Z.

And Linux (at least if you're running GNOME - I don't use KDE but I would guess it's the same) acts identically to OS X. If I delete things using the GUI they end up in the Trash under both GNOME and under OS X. If I delete them using the command line under either system, they're gone for good.

Re:But does it undelete...

[timbo234]

Everything I can screw up on the computer should have an Undo-- that's what the Recycle Bin (or Trash Can for Mac users) is there for, although it's a bit more awkward than pressing control-Z.

Both KDE and GNOME have this built-in. In KDE there isn't even an option in the right-click menus anymore (unless you know to hold down the shift key) to properly delete something - it always goes to the trash can.

Re:But does it undelete...

[trolltalk.com]

Well, you could run a subversion or git server, and alias the rm command so that it saves everything in the filespec, then deletes everything. That gives you the best of both worlds.

Re:But does it undelete...

[Blakey Rat]

Yes, but if a hostile script ran in your user account, it would delete them "using the commandline" (i.e. programmatically) and they'd be gone forever. That was my point.

Windows Shadow Copy helps prevent this by keeping track of last revisions of files regardless of how they were deleted. OS X has "Time Machine" which provides much of the same capability.

Security experts on computers are always so concerned about "the system." Will a virus be able to corrupt your system files? Can a worm screw with the bootloader? Is the system secure? Screw the system. The system is worth maybe two hours, the amount of time it takes to reinstall, the only *really* important files in the computer are the ones the user creates and uses to run their life. Computer permissions are currently designed entirely bass-ackwards for home computers.

Re:But does it undelete...

[Blakey Rat]

Yes, but everything should have it. The CLI too. In DOS, OS X Darwin, Linux, whatever. The option to Undo should *always* be available for *every* action (if technically possible; i.e. there's enough disk space left). Right now, the closest we have is OS X's Time Machine and Vista's Shadow Copy.

Re:But does it undelete...

[mlwmohawk]

Undelete might not be a problem for a lot of people. In fact some security-paranoid people might like it that way, but that's what 'shred' is for, isn't it?

The problem is that the actual definition of "delete" has been co-opted into meaning "remove from sight, and really delete later."

If you want to be able to recover from a mistake, don't use "rm" or at least double check before you press the enter key.

KDE and Gnome have trash cans, use the GUI or maybe someone can write a plugin replacement for "rm" that works with the trashcan metaphor.

Re:But does it undelete...

[mlwmohawk]

The option to Undo should *always* be available for *every* action

I am sick and tired of this "training wheel" society where no one takes the responsibility to double check their actions and everyone wants to be protected from themselves.

I like "power tools" There is no "undo" for a drill or a saw. There is no undo for hammer. I HATE the idea that everything that I try to use doesn't do what its told because someone wants to protect me from myself.

If you are unsure, use Gnome or KDE, they use a trash can metaphor that allows undelete. Otherwise, grow a pair and just be careful.

Re:But does it undelete...

[a.ameri]

Temporary GUI recycle bins or trash cans or whatever you want to call them today completely miss the point. I do not always delete my files through Dolphin or Konqueror, I run scripts, I use the shell, and I also interact with my computer through various non-KDE programmes.

Actually one of the initial goals of freedesktop.org was to harmonise this so that Gnome and KDE would use the same trash. I don't know how successful that was since it's been a while since I last ran Gnome.

The GP is right. I remember Windows 98 and that Norton Protected Recycle Bin, and from a usability point of view, it was awesome. Easy to use, fast with no user-detectable performance hit, and very convenient. I can't think why 10 years on, we don't have the equivalent of that in the open source world even with advanced filesystems such as JFS and XFS.

Re:But does it undelete...

[Blakey Rat]

I like "power tools" There is no "undo" for a drill or a saw. There is no undo for hammer. I HATE the idea that everything that I try to use doesn't do what its told because someone wants to protect me from myself.

If you are unsure, use Gnome or KDE, they use a trash can metaphor that allows undelete. Otherwise, grow a pair and just be careful.

So don't use Undo if you don't want to use it. That doesn't have anything to do with whether it should exist or not.

I mean, if you want a really really hard game set the difficulty to "nightmare." If I don't, I can use "normal" and everybody's happy, right?

Re:But does it undelete...

[petermgreen]

all files user deleted within the KDE framework goes to the Trashcan instead
Sounds very much like the windows recycle bin, files deleted through the shell or through common dialogs etc (and presumablly files deleted through some special API that hardly anyone uses) get sent to the recycle bin. Files deleted by any other means or files overwritten by saving a new version over the top do not.

From the GPs description it sounds like the norton protected recycle bin did a lot more than that.

Re:But does it undelete...

[dubl-u]

The greatest feature of modern software is "Undo." Yes and no.

That's a great feature for user-facing applications, no denying it. I love my undo. But filesystems aren't just for users. I would like my filesystems to focus on storing files.

If they provide hooks so that somebody can implement undo on user-focused systems, that's great. But that undo should be part of a carefully-considered common user interface to undo for system operations, and not something jammed into a single filesystem because people think that'd be cool.

Re:But does it undelete...

[Peter H.S.]

AFAIK the Norton bin only gave protection for files deleted within the Windows environment, not if one used "delete *.*" from the "console", or if one shelled out to DOS. Anyway the FAT filesystem had no notion of "undelete" or any kind of delete protection, though some hacks could be used because the FAT filesystem was incredible badly designed and therefore technically didn't delete the file, but just the entry to it. But the cost of this design was among other things a really, really nasty tendency to have badly fragmented filesystems.

I don't think it makes sense to implement delete protection on the filesystem level, it should be done as KDE does it, and for those few that wants to delete files from the commandline, request a modified "rm" GNU tool, or use 'mv *.zap /tmp instead of 'rm *.zap' instead.

--
Regards

Re:But does it undelete...

[drinkypoo]

Whatever, the actual script is left as an exercise to the reader. You could for example use find, reverse the order of the lines output by find, then for each file test to see if it is a file or a directory, then rm -f the file if it is a file, or rmdir the directory if it is a directory.

The whole point was to illustrate how retarded the current trash situation is, not to write you a shell script. That's not my job.

ext3 tops out at 16GB files?

[XorNand]

Ext3 tops out at 32 tebibyte (TiB) file systems and 2 TiB files, but practical limits may be lower than this depending on your architecture and system settings--perhaps as low as 2 TiB file systems and 16 gibibyte (GiB) files.

Is this really the case? I created a 100GB file on ext3 earlier this week. It contains a virtual machine image that I am currently running under Xen. I haven't yet had a problem. I would guess that >16GB files are pretty commonly used in the world of Xen.

Re:ext3 tops out at 16GB files?

[Uncle Focker]

Did you miss the part where it said "may be" lower. As in, it's in some cases that might be true, but not others.

Re:ext3 tops out at 16GB files?

[Firefalcon]

This may help (from <a href="http://en.wikipedia.org/wiki/Ext3">Wikipedia's ext3 entry</a>):

"Block size     Max file size     Max filesystem size
1KiB         16GiB         2TiB
2KiB         256GiB         8TiB
4KiB         2TiB         16TiB
8KiB         2TiB         32TiB

It should be noted that the 8 KiB block size is only available on architectures which allow 8 KiB pages (such as Alpha)."

Teletebi

[davidwr]

1024^4 television shows and not one of them worth watching.

Re:Better option:

[jdinkel]

Buy a Mac, then won't even be tempted with having a choice of something better.

Getting off topic, but...

[teh_commodore]

Might I recommend passing the -I option? I have rm aliased to 'rm -I' on my work machine.

Re:Getting off topic, but...

[swilver]

Yes, I have that too, and it's useless.
Frankly, the whole command is poorly designed. I don't want to confirm each and every directory, I want an overview of what is getting deleted, and then I want it to ask me "are you sure?". It doesn't even offer an option "yes to all" when using interactive mode when you finally tire of pressing "y" 50.000 times.
I've even went looking for replacements that did just that, but that turned up nothing.

Re:Getting off topic, but...

[teh_commodore]

Have you tried the good ol' 'quick and dirty bash script'? I haven't written it up, because I haven't run into a significant problem yet, but it doesn't seem like it'd be anything hard.

Wrong arguments

[The Mighty Buzzard]

I think you're looking for rm -rI /hugedir then. Adding the -f option is you specificly stating that you know exactly what you're doing and do not want to be asked if you're sure you want to remove /home and all it's subdirectories.

Re:Wrong arguments

[swilver]

Capital I not being an option on my rm, I assume you are talking about the "interactive" mode. The problem is that rm's behaviour is ridiculous in this mode. Watch this:

[root@MyServer 0 ~]# rm -Ri MPdeletethis
rm: descend into directory `MPdeletethis'? y
rm: descend into directory `MPdeletethis/Gui'? y
rm: descend into directory `MPdeletethis/Gui/wm'? y
rm: remove regular file `MPdeletethis/Gui/wm/ws.c'? y
rm: remove regular file `MPdeletethis/Gui/wm/ws.h'? y
rm: remove regular file `MPdeletethis/Gui/wm/wskeys.h'? y
rm: remove regular file `MPdeletethis/Gui/wm/wsmkeys.h'? y
rm: remove regular file `MPdeletethis/Gui/wm/wsxdnd.c'? y
rm: remove regular file `MPdeletethis/Gui/wm/wsxdnd.h'? y
rm: remove directory `MPdeletethis/Gui/wm'? y
rm: descend into directory `MPdeletethis/Gui/skin'? y
rm: remove regular file `MPdeletethis/Gui/skin/cut.c'? y
rm: remove regular file `MPdeletethis/Gui/skin/cut.h'? y
rm: remove regular file `MPdeletethis/Gui/skin/font.c'? y
rm: remove regular file `MPdeletethis/Gui/skin/font.h'? y
rm: remove regular file `MPdeletethis/Gui/skin/skin.c'? y
rm: remove regular file `MPdeletethis/Gui/skin/skin.h'? y
rm: remove directory `MPdeletethis/Gui/skin'? y
rm: remove regular file `MPdeletethis/Gui/interface.c'? y
rm: remove regular file `MPdeletethis/Gui/interface.h'? y
rm: remove regular file `MPdeletethis/Gui/Makefile'? y
rm: remove regular file `MPdeletethis/Gui/bitmap.c'? y
rm: remove regular file `MPdeletethis/Gui/bitmap.h'? y
rm: remove regular file `MPdeletethis/Gui/app.c'? y
rm: remove regular file `MPdeletethis/Gui/app.h'? y
rm: remove regular file `MPdeletethis/Gui/cfg.c'? y
rm: remove regular file `MPdeletethis/Gui/cfg.h'? y
rm: descend into directory `MPdeletethis/Gui/win32'? y
rm: remove regular file `MPdeletethis/Gui/win32/interface.c'? y
... few 1000 lines more removed ...

You get the idea. Is no wonder I type -f when I want to remove a directory.

Re:Wrong arguments

[Sentry21]

I default rm (as root) to rm -vi. Why? Not so that I can confirm every file deletion. Rather, so that when I delete something as root without using -rf (which I have made a habit), it asks me if I'm sure. I can then say 'Oops, that's not what I wanted to do!' or 'Oops, I need to use -rf'.

The more PEBKAC I can prevent, the better.

Why bother?

[jabuzz]

ext4 is the biggest waste of time and effort in Linux. There are already good extent based filesystems for Linux. Why anyone would consider using what is an experimental filesystem for a multi TB production filesystem is beyond me.

What ever they do XFS and JFS will have way more testing and use than ext4 will ever have. I just don't get the point of ext4. It would be far more useful to fix the one remaining issue with XFS, the inability to shrink the filesystem none destructively, than to flog the dead horse which is ext2/3 even more with ext4, which is not one disk compatible anyway.

Re:Why bother?

[skulgnome]

It has value as an experiment, even if it ultimately doesn't turn into much. These people have ideas, and they want to implement them. They aren't maintenance programmers and should not be shoehorned into that task even at the level of J. Random Person On Slashdot's thought.

Remember how reiserfs was the first filesystem to have journaling in Linux, and how some people were ready to state that there is no need to do an ext3 any more?

Re:Why bother?

[trolltalk.com]

ext4 is the biggest waste of time and effort in Linux. There are already good extent based filesystems for Linux. Why anyone would consider using what is an experimental filesystem for a multi TB production filesystem is beyond me.

TFA points out that it's NOT for production use yet.

As for why we want more choice - look what happened with reiserFS. The development environment now only runs in a chroot jail ...

Re:Why bother?

[clampolo]

As for why we want more choice - look what happened with reiserFS. The development environment now only runs in a chroot jail ...

Is Reiser4 really a dead project? (no pun intended) Seemed like it was just breaking some kernel coding standards but was working.

Would be a shame for the project to get scrapped. The performance numbers were great (faster and files were smaller.) ext4 is a LOT less interesting.

Re:Why bother?

[ChrisMaple]

From what I've read, Reiser4 is very slow deleting files.

Re:Why bother?

[dbIII]

It had a very small team working on it and a rabid hero worshipping fanbase that would not accept criticism. We've seen XFree86 reworked as Xorg without the social issues dragging it down so it may continue to the point where it is viable (eg. filesystem recovery that works), and across the board performance instead of good performance at specific file sizes.

Re:Why bother?

[JayAEU]

Interestingly enough, the same was being said about ext3. Back then, everybody was wondering why anybody would want to use ext3, when ReiserFS provided journalling and a btree-based on-disk format.

The rest is history.

Re:Why bother?

[jabuzz]

Yes but at the time ReiserFS was brand new. However now in 2008, XFS has been available on Linux for at least six years, and has a much longer pedigree on Irix. The situation is not comparable at all. I would also say that ext3 is an dismal filesystem for a large one. Nobody in their right mind would push ext3 to it's limits.

Re:Why bother?

[zsau]

(no pun intended)

You noticed the pun. At the time you pressed "Submit" you intended the pun. Why deny it? If you think its tasteless, edit what came before. If you like it, admit it; we won't think any less of you (I mean seriously, he's a murderer according to a jury of his peers).

Saying "no pun intended" is fine, because you can't unsay something. Writing "no pun intended" on a computer is silly and makes the language meaningless. Meaningless language is bad because it defeats the purpose.

Re:Why bother?

[ioshhdflwuegfh]

(no pun intended)

You noticed the pun. At the time you pressed "Submit" you intended the pun. Why deny it? If you think its tasteless, edit what came before. If you like it, admit it; we won't think any less of you (I mean seriously, he's a murderer according to a jury of his peers). But still, there is no guarantee that the reader will have noticed the pun. That's the pun of writing "no pun intended". Then "no pun intended" may defeat the intentionality of the writer (who might have intended to write: yes, I know there is a pun here, but the text has meaning beside the pun, I'd like you to focus on that).

Saying "no pun intended" is fine, because you can't unsay something. Writing "no pun intended" on a computer is silly and makes the language meaningless. Meaningless language is bad because it defeats the purpose. The pun of written "no pun intended" thus forks the meaning by diverting the purpose, intention, etc...

This is going to sound crazy but...

[pizzach]

Wouldn't it be easier to make manufacturers use the old MB=1024 type standard than to get the common people to understand a new prefix that they just won't remember?

Comment removed

[account_deleted]

Comment removed based on user account deletion

What about comparison to other filesystems?

[Khopesh]

Those features may be new to ext3, but not to the real competitors. I see nothing that might grant an edge over JFS or XFS. The real justifications will come from performance tests.

This reminds me of the recent NTFS article here, which actually suggested that since Hans Reiser is in jail and reiser4 is dead, we should consider NTFS. WTF? The ludicrousness of using NTFS as the primary filesystem is further justified in this article by its similar performance to ZFS, but both run in user-space (and are thus horrible in performance), so neither is really an option. What the heck is wrong with JFS and XFS?

Here are some real comparisons: First, Wikipedia's Comparison of file systems gets you started with a nice mapping of features. Second, a benchmarking of filesystems from 2006 which is still quite applicable (though it doesn't yet cover ext4). What we need is a comparison of EXT4 to XFS and JFS (et al), with EXT2/3 in there for reference.

Recall that the biggest reason for using ext3 is that it is supported best of all the filesystems. If all hell breaks loose, even Tomsrtbt (an ancient rescue floppy pre-dating knoppix) can fix it. Ext4 breaks this backwards-compatibility to ext2. Therefore, I see no reason to use it. One might as well use something more stable and proven, especially while we lack numbers suggesting it performs as well or better.

Re:What about comparison to other filesystems?

[Sentry21]

If your recovery procedures involve using pre-knoppix floppy recovery tools, you shouldn't be administering any systems with important data on them.

Aside from the fact that no non-obsolete machine I've seen in the last few years has a disk drive, 'backwards compatibility with ext2' is a pretty lousy minimum requirement for a filesystem.

Heck, I can do recovery on Ext2/3, ReiserFS, JFS, XFS, and more using only a few-dozen-meg Debian netinstall image. I don't even want to know what an Ubuntu or Knoppix LiveCD could recover from.

Re:What about comparison to other filesystems?

[oddfox]

Here's an Ext4/XFS/ZFS benchmark

I would like to see a more recent benchmark that did include JFS/Reiser/etc, though.

Re:Better option:

[EvilRyry]

To quote Linus... "Their file system is complete and utter crap, which is scary."

Based on real life experience with HFS+, I'm inclined to agree.

What abbout Compresion.

[Forge]

My favorite file-system reportedly performed faster on certain workloads if it was compressed. However the primary maintainer will be incarcerated for a very long time.

Dose EXT4 (or EXT3 for that matter) support running as a compressed file-system? If so, what is the speed like?

Especially on that workload most likely to benefit substantially from running as a compressed file system. Mid to High Traffic Email server.

Why to bother...

[tempest69]

ok, I'll agree ext4 blows.. XFS and Jfs are dogs in the small file arena. sure they both address it, but neither one does it well. nor do they have undelete, which is pretty nice. and one second file timing is a bit on the loose side, especially if your running a batch that checks that sort of thing. And none of these are tuned for a SSD at all. So yes we need upgrades, big time, ext4 isnt it..

Converting from Reiser...

[fialar]

Anyone know an easy way to convert from Reiser? I set up a Reiser partition a while back. It's quite large. I think I'd need to do it peacemeal, like, size reiser partition down, create ext3.. move files.. size it down again, increase size for ext3, etc.. until the reiserfs has no space on it?

I don't have the space to unload all the reiser files someplace else.

Any ideas? Does gparted resize reiserfs without destroying the fs?

Re:Converting from Reiser...

[Spoke]

You're pretty much screwed. Generally, the only way to extend a filesystem is to add space to the end of the partition. Extend the partition, then tell the filesystem to expand to use the newly allocated partition space.

Now, you could fake this by using LVM and multiple partitions, but the end result would not be pretty.

Re:Converting from Reiser...

[apoc.famine]

My best idea would be to move your data to your normal backup medium, then resize/reformat your drive as needed.

Of course, if you don't back up, your file system of choice is moot. Recently, I doubled my drive space for less than half what the initial drive space cost. I formatted the new drive as needed, and then copied things over, and mirrored the old drive. Everything fails, it's a matter of when, not if.

Re:Converting from Reiser...

[petermgreen]

I'm pretty sure gparted can nondistructively resize reiserfs partitions. So your method should be possible.

Unfortunately during each resize operation you will have to move the start of the ext3 partition. This is slow and relatively risky since afaict gparted moves the start of a partition by moving all the data on the partition, you can imagine what state that would leave the filesystem in if there was a power failure partway through.

Personally I don't think it is worth the hassle and risk.

Ext4 still far from stable

[Anonymous Coward]

The article doesn't say this explicitly enough: if you like to keep your data, DON'T USE EXT4 YET! It is still in highly experimental stage. Try it out, fine... but not as the only copy of your data.

What about corruption resistance?

[rrohbeck]

I've been bitten by the ext3 journal corruption bug due to out-of-order writes of the journal (write-heavy operation on a large RAID with write-back caching and a power failure), so... how well is journaling and recovery being tested under such conditions?

Re:What about corruption resistance?

[Anonymous Coward]

Ext4 now checksums all the journal data. I can't comment on testing, but they have done work on improving detection of bad journal data (aka checksumming).

Re:What about corruption resistance?

[mikelieman]

how well is journaling and recovery being tested under such conditions? Since it's Open Source software, the 'stock' answer will do:

"How well are you planning and testing the journaling and recovery?"

More precisely...

[Ecuador]

...flushes the free space left on its array.

Re:But does it undelete... - correction

[EvanED]

There are three problems with such a simple solution; I talk about them above. (Ignore the first two; they don't apply. The third/fifth is in a reply to that post.)

OMFG. NTFS?

[rrohbeck]

Anybody who advocates NTFS has never worked with directories with a large number (>100,000) of files under one directory tree. Just deleting one file can make the disk seek for several seconds, during which the filesystem is completely frozen. I guess it's reshuffling its entire B-tree.

Re:OMFG. NTFS?

[NitroWolf]

Anybody who advocates NTFS has never worked with directories with a large number (>100,000) of files under one directory tree. Just deleting one file can make the disk seek for several seconds, during which the filesystem is completely frozen. I guess it's reshuffling its entire B-tree.

>100,000? What kind of system are you running? If I have more than a few thousand files in a directory and I want to do anything with the directory, it bogs down to a crawl. If it have 30k - 40k, I have to be very careful what I delete and select a lot of files at once to delete. Deleting them individually is suicide.

Re:OMFG. NTFS?

[marcosdumay]

I guess it's reshuffling its entire B-tree.

Well, one doesn't reshufle a B-tree... But I wouldn't be too surprised if I discover Windows is doing that.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:Consistency and integrity are important

[ZachPruckowski]

ZFS does full checksumming, but for licensing reasons, it's not available for Linux (outside of via FUSE). It runs on BSD and Solaris.

B sub 2?

[bill_mcgonigle]

Yeah, but now the problem is that KB is now ambiguous - it could either be 1000 bytes, or it could be 1024. Before anyone mentions HDD manufacturers, it isn't ambiguous there, either. 1 KB is 1000 bytes, yeah that's because they're fleecing you, it sucks but oh well.

What if we CS geeks agree to write 56Kb₂? If it's conversational, no need.

e2fsprogs still not ready,

[gzipped_tar]

and that is the problem.

OT: Your sig

[swillden]

Whenever I hear the word activist, I reach for my revolver.

Better watch that around these activists. They're a peaceful bunch, but don't even think about drawing on them, especially if you only have six shots.

SSD improvements?

[Larsrc]

What I was hoping for was some improvements in making use of the characteristics of SSDs. But maybe that'd have to be an entirely new filesystem that is not designed with seek time as a generic assumption. Any takers? -Lars

The Linux RAID implementation is the problem

[Colin Smith]

The ext3 file system is pretty much OK, ext4 is only tweaking functionality.

RAID on Linux handled by the MD subsystem, which is largely cluster unaware. This means it's unsafe for example to stripe across multiple iSCSI devices then run a cluster file system like GFS on top, which also therefore means that you have to have hardware RAID on a per device basis.

Re:The Linux RAID implementation is the problem

[Tsagadai]

You are talking out your arse. The other part of mdadm is multipath. Multipath is designed and programed for that specific role of joining many iSCSI and FC attached storage devices. Please take your ignorance and fud and go back to sleep.

Here's my suggestion:

[warrax_666]

Have the "trash-instead-of-rm" script md5sum the "deleted" file and rename it to ~/.Trash/.. If you get clashes it's very likely to be an identical file anyway and you avoid the various issues that might crop up if you try to recreate directory structures in ~/.Trash.

This might need a fair bit of cpu on very large files (anything below a few MB is trivial on modern CPUs), but then you probably don't want to keep very large files lying around in ~/.Trash anyway (prevents good block allocation by the file system), so you could just opt to delete those immediately.

Don't be too concerned

[marcosdumay]

By then, the onboard battery will already have failed, and data will be restored to January 1st 1970.

Re:Better option:

[yourlord]

I wish I had mod points because you would get a +1 insightful.

Microsoft is a serious enemy to freedom and choice, but even they pale compared to the draconian control freaks at apple..

Comment removed

[account_deleted]

Comment removed based on user account deletion