Making ZFS and DTrace Work On Ubuntu Linux

New submitter Liberum Vir writes "Many of the people that I talk with who use Solaris-like systems mention ZFS and DTrace as the reasons they simply cannot move to Linux. So, I set out to discover how to make these two technologies work on the latest LTS release of Ubuntu. It turned out to be much easier than I expected. The ports of these technologies have come a long way. If you or someone you know is addicted to a Solaris-like system because of ZFS and DTrace, please, inquire within."

Comments

ZFS on Linux

[dnaumov]

So what am I supposed to do about all the kernel panics and absurdly slow IO and transfer speeds?

Re:ZFS on Linux

[Anonymous Coward]

Run them on Solaris instead (or, more likely, illumos).

Re:ZFS on Linux

[SolitaryMan]

I used to have very high expectation of OpenSolaris after Ian Murdock became the head of the project... But then Oracle came and destroyed all my hopes.

BTW, is ZFS SSD-aware?

Re:ZFS on Linux

[stox]

DTrace and ZFS are quite mature running under FreeBSD.

Re:ZFS on Linux

[Liberum Vir]

I haven't done any performance testing so far. My objective with this was just as a proof of concept, if you will. I'm sure, if you are having kernel panics and absurdly slow IO/transfer speeds, the developers would welcome your input to make it better. Personally, I prefer LVM and ext4 for most uses. Again, this was more just to prove that it could be done.

Re:ZFS on Linux

[Anonymous Coward]

ZFS has some pretty heavy hardware requirements. I have yet to see it copy faster than 4MB/sec on my system with 768MB of RAM 500mhz.

Re:ZFS on Linux

[Anonymous Coward]

....

BTW, is ZFS SSD-aware?

ZFS has had SSD support for the ZIL and L2ARC for years.

Re:ZFS on Linux

[frodo from middle ea]

+1, After having used linux daily for more than 14 years, I have recently ventured in to the BSD land. And I like it a lot.

If you've been playing in Linux land, and never bothered with any of the *BSD, do yourself a favor and install one of the BSDs in a VM. You'll not be disappointed.

Re:ZFS on Linux

[greg1104]

Yes, no one would suggest ZFS as the filesystem for your phone. You might have to do things like disable checksums if you have an older or otherwise underpowered CPU, and it's tuned by default to use memory quite heavily. That anecdote isn't very relevant for today's desktop or server environments though.

Re:ZFS on Linux

[Anonymous Coward]

So what am I supposed to do about all the kernel panics and absurdly slow IO and transfer speeds?

Learn to use and configure linux

Not quite that simple. A number of presumptions that zfs makes about the underlying OS are in conflict with the linux kernel approaches (zfs was written for Solaris after all).

Re:ZFS on Linux

[catmistake]

I used to have very high expectation of OpenSolaris after Ian Murdock became the head of the project... But then Oracle came and destroyed all my hopes.

Good news! Your high expectations and hopes are alive and well at the [open] crossroads of America . They're also welcome at freenode on #openindiana.

Re:ZFS on Linux

[Anonymous Coward]

If you think a SSD is the way to go you don't need zfs because you already show that you don't care about your data. I use a ssd for speed in applications where I don't care if the data is lost. How does your ssd drive with fat32 ensure the integrity of the files themselves. I'm not just talking about file system metadata. Hopefully people won't pay heed to ignorant hipsters like yourself. Then again if the goal is data integrity running the Linux port of zfs isn't the answer either.

One of the jewels of zfs some of that uncool to you crcs or checksums on data. What you should be saying is why are SSDs so crappy that the can only take 5k writes?

Re:ZFS on Linux

[Anonymous Coward]

Using a small SSD for L2ARC and ZIL can bring near SSD performance from spinning disk... (near I said, not identical - but damned close)

Re:ZFS on Linux

[Anonymous Coward]

zfsonlinux is quite solid; I've been using it on all my systems including this laptop for over a year.

Re:ZFS on Linux

[MightyYar]

I have to second this... Debian was always my preference but I tried FreeBSD to get ZFS. For dependencies, ports does some things... differently than APT, but they are similar enough that it won't completely shock your system.

And just like Debian, it is easy to start with an extremely minimal system and only add what you need, so stability and boot speeds are excellent.

I think that Debian is still faster at certain things, though that is subjective.

Re:ZFS on Linux

[Anonymous Coward]

Anything Solaris lost what was left of this mojo after the Oracle acquisition.

It still exists in some form or another but has left mind share than any *BSD derivative.

Re:ZFS on Linux

[Anonymous Coward]

ZFS stands for Zombie FS, an outdated disk management bloat of last century "enhancements" for spinning drives. ZFS fancy features only contribute to more unnecessary disk writes that are killing my modern SSD. A current SSD drive has it's own hardware level management, there is no need to keep some nasty write, log, index, trace, journal -- the write-happy file systems around, actually the plain old FAT32 is kinda rock solid compared to for example NTFS, the current #1 SSD assassin.

Wahh! WAHHH! I couldn't get it to work so it must be broken!

Re:ZFS on Linux

[hawguy]

So what am I supposed to do about all the kernel panics and absurdly slow IO and transfer speeds?

I thought ZFS ran in userland on Linux - how does it cause kernel panics?

In any case, I've been running zfs (raid-z) on a home Ubuntu based fileserver for over 2 years without a single kernel panic (record uptime was 9 months before I rebooted to apply updates).

This fileserver is used to stream movies, as well as act as a DVR for 3 home security cameras, and is the backup target for several Windows computers so it gets a fair bit of use.

You're right about slow I/O though.. it's not nearly as fast as hardware RAID would be, during heavy I/O the CPU power of the Atom processor is a limiting factor.

Re:ZFS on Linux

[MightyYar]

I don't know about "heavy"... I have it running on a 6-year-old HP xw4400. Sure it's a nice old machine with a 64-bit processor (required for stability) and ECC RAM (required to get the real data integrity benefits of ZFS), but you can get a HP Microserver new for under $400 that has both of those things.

Re:ZFS on Linux

[MightyYar]

This article uses the kernel module, not the userland FUSE stuff.

Re:ZFS on Linux

[philip.paradis]

You're an idiot. I suggest performing some Google queries for ZFS and SSDs. Then again, judging by your post, you're probably coming from a world where NTFS is common. I would say it's understandable that a Windows-centric sysadmin would have trouble understanding these things, but I happen to know several Windows admins who do understand them. Thus, I'm pretty sure you're an idiot.

Re:ZFS on Linux

[hairyfeet]

Dude, you're doing all that...on an Atom? Doesn't it drag ass? if it were me I'd replace that sucky Atom with a cheap Phenom X4e, those support ECC and can be had for $62. Figure in $30 for an AM2+ board and $20 for a 2Gb RAM stick and for less than $115 you'd have a machine that would be a HELL of a lot faster than an Atom at multitasking.

Re:ZFS on Linux

[jd]

Fairy nuff. ZFS has a lot of benefits over the standard Linux filesystems for certain things (just as all the Linux filesystems have their own niches in which they are the supreme overlords). It's rare for me to create a system in which I use fewer than 4 filesystems and if I were to try for a fully-optimized system that would probably go to 5 or 6. ZFS running reliably under Linux would pretty much guarantee me moving to such a model.

Re:ZFS on Linux

[Anonymous Coward]

But the last I checked there are at least 30 filesystems available (buildable and bundled) in a standard linux kernel. Some offer backward compatibility to very old systems, some offer unique network support (plan 9 is a different kind of beast, but not bad, just very interesting), and some run under fuse (like zfs) which makes them 'out of kernel', and thus, rather slow. BTRFS is (apparently) coming along. Someone will surely point this out on this forum. Last I tried it, it was still a work in progress. There was no automatic pruning, poor ssd support, and a lot of other features missing (like fsck repair tools). That was 2 operating systems ago, and reiser, although dated, still works very well. When btrfs matches it, I switch.

Re:ZFS on Linux

[Damouze]

I am terribly disappointed with ext4. If you want a good and stable combination of volume management and a filesystem, I'd recommend LVM (which of all the software solutions I've worked with is by far the best), combined with XFS, which is a stable and reliable journalling filesystem. Where ext4 fails just as terribly as its predecessor, ext3, as far as disk performance goes (kjournald, big kernel lock and the likes), XFS just ploughs on and on.

As for Solaris and ZFS. At the company I work for we still use UFS for everything. ZFS has its niceties, but it does one thing many applications, especially DBMSes, won't like: it does not support direct I/O (http://blogs.sybase.com/database/2009/07/directio-file-system-devices/). Even if you do the zpool, zvol-as-raw-disk-device option you'll end up terribly disappointed, because although it claims to support direct I/O, you will not bypass the primary cache. Also, using ZFS on a SAN LUN does not really make any sense. It wants to do things with your disk devices that you don't want it to do with them. So, while as a concept ZFS is marvelous, quite brilliant actually, its practical application is not well-suited for a high-performance, high-load datawarehouse. Why? Because like another software company whose name I will not mention, Sun designed ZFS to be a little too smart for its own good.

Re:ZFS on Linux

[Damouze]

I actually run an OpenIndiana installation on a system with an Atom D510 and two 320GB 2.5" drives. The performance is actually not all that bad. Even so, I didn't really build it to perform very well, I built it to see if I could set up something that eats less than 100 Watts while still offering something remotely resembling a performance. With regard to that, the system is working perfectly ;-).

Re:ZFS on Linux

[unixisc]

One funny thing about OpenIndiana - it's fully supported on x86/x64, but not on Sun's own SPARC servers. So ironically, if one wants ZFS and DTrace support on SPARC, one is probably better off going w/ FreeBSD 9, rather than OpenIndiana.

Re:ZFS on Linux

[unixisc]

It will be interesting once Debian's kFreeBSD is out to see whether it supports ZFS & DTrace or not.

One thing I'm wondering - b/w OpenIndiana and FreeBSD, the main difference is that the former is based on SVR4 and the latter on 4.4BSD. Are there differences b/w them that would cause anybody who needs ZFS to prefer either one over the other?

Re:ZFS on Linux

[TheRaven64]

If you want to try FreeBSD with ZFS, I recommend that you use the PC-BSD installer. This can set up a complete FreeBSD environment (with or without the extra PC-BSD stuff - I think the 'server' install is vanilla FreeBSD) on a ZFS root. Doing the same with the current version of the FreeBSD installer requires some manual intervention, which is not really fun for people who aren't experienced with FreeBSD. Or for anyone else, for that matter.

Re:ZFS on Linux

[TheRaven64]

Yes, no one would suggest ZFS as the filesystem for your phone

Actually, some guys at Samsung would. They trimmed a lot of the optional features (made them optional at compile time, rather than run time), reduced the size of the ARC to 4MB, and got ZFS running reasonably well on a machine with 16MB of RAM. We're investigating bringing some of their changes into FreeBSD for ARM systems, but it's starting to look as if it would be one of those things where by the time it's done the hardware has improved to the point that it's no longer worthwhile.

Re:ZFS on Linux

[Zero__Kelvin]

Don't write any kernel code yourself, and you won't see any kernel panics.

Re:ZFS on Linux

[bheading]

I agree that ZFS on a SAN doesn't make sense, but that seems to be to have been the intention; ZFS wasn't aiming to work with your SAN, it is aiming to replace it, and I'm sure had the guys at Sun remained in control SAN features would have been added to it. That's why NetApp brought them to court.

Re:ZFS on Linux

[bheading]

ZFS is intended to use cheap, recent hardware to deliver high performance. The idea is that you'll build a box with gobs of RAM, SSDs and CPUs and save significant amounts of money compared to buying a similarly specced storage box from one of the major enterprise storage vendors.

Re:ZFS on Linux

[Guspaz]

Speak for yourself, I've been using it for months without a single kernel panic, and the performance has actually been better than it was on OpenSolaris.

Re:ZFS on Linux

[MightyYar]

I use Solaris at work, but as a workstation and not as an admin so while I have a "feel" for it, I can't directly compare them. I would say they are very similar but do have semantic differences... de-facto file locations, behavior of temp directories, etc. I feel that they are more similar than Linux is to either FreeBSD or Solaris.

I looked into both (at the time) OpenSolaris and FreeBSD and decided that FreeBSD's better hardware support was worth the tradeoff in lagging ZFS versions. And not that I anticipated it, but it turns out that the FreeBSD Linux emulation environment was also useful (for running a CrashPlan server).

Re:ZFS on Linux

[Lennie]

I do know kFreeBSD atleast supports ZFS, pf, pfsync and carp from OpenBSD which are all part of FreeBSD kernel.

Not sure about DTrace, haven't tried that.

Re:ZFS on Linux

[Lennie]

Interresting enough btrfs works really well on a phone.

Re:ZFS on Linux

[Anonymous Coward]

That possibly makes it ok for you to use it for your particular situation. (If you are willing to put your neck on the line if there is a problem).

(And it is good occasionally to see someone have the balls to take responsibility for something personally).

I have managed to break ZFS on FreeBSD before though not doing anything I wasn't supposed to do.

(I used it on my laptop pretty foolishly when I quit using Opensolaris thinking it would be at least as reliable). .

It is always more fun working in a job knowing that you might be tested.

(I think running it in production (on OS's other than Solaris) without knowing how to use zdb properly is not worth it though).

And how to fix it if you force it to import on a Solaris box.

Re:ZFS on Linux

[hawguy]

Dude, you're doing all that...on an Atom? Doesn't it drag ass? if it were me I'd replace that sucky Atom with a cheap Phenom X4e, those support ECC and can be had for $62. Figure in $30 for an AM2+ board and $20 for a 2Gb RAM stick and for less than $115 you'd have a machine that would be a HELL of a lot faster than an Atom at multitasking.

It runs surprisingly well, I get around 15MB/second write speeds (and over 30MB/second read) which is more than I need for what I use it for. About the only time I notice it being slow is after I've ripped a movie from DVD and am copying it over to the fileserver. Most of the time I access it via Wifi so the disk is faster than the network. It's used only as a headless fileserver, no windowing system is installed so I don't need to worry about interactive performance.

I thought adding the webcams and zoneminder would push it over the edge, but even with doing motion detection on the 3 cams, the CPU hovers around 30% utilization, so I really have no complaints with the performance.

Not too bad for 35 watts of power (including the UPS). The TDP of the Phenom is 95W, and the motherboard is probably not all that power efficient either, so I'd probably at least double my power consumption if I went with a faster CPU. The Atom costs me around $50/year in electricity to run,so if I doubled the power consumption, it'd cost me around $100/year.

Re:ZFS on Linux

[Ben Hutchings]

The current Debian stable release (6.0, squeeze) has kFreeBSD ports for i386 and amd64.

Re:ZFS on Linux

[hairyfeet]

The link is for the low power e series, so around 55w at full load and idle is less than 30. If all I cared about was low power I think I'd have went for an E350 they max at 18w and idle less than 9w but since they are an out of order CPU and dual cores its got a lot better performance than Atom. But the nice thing about that X4 is you wouldn't need to have some other machine do the ripping or transcoding, slap in a cheap DVD or BD player and let it do the work for you.

I've built a couple of servers using the E series and its actually quite nice for that job, pretty easy to passively cool yet it has enough power that it can do the heavy lifting when needed. But if all it has to do is serve files hell anything can do that, I use an 8 year old Sempron for that job in the shop, quiet as a church mouse but it can still rip a DVD if I needed it to.

Re:ZFS on Linux

[Anonymous Coward]

Also, the thing with DTrace on linux is that it is not stable and supports a tiny subset of the probes available on FreeBSD, Solaris or OS X.

Re:ZFS on Linux

[Pinky's Brain]

IMO L2ARC is just a little too primitive for subset caching for permanent storage ... what is needed is a mix between MFU and Superfetch.

Re:ZFS on Linux

[BitZtream]

ports is not like apt, pkg is.

pkg_add -r zsh

and

apt-get install zsh

are the same

cd /user/ports/shells/zsh && make install clean

is entirely different.

Ports is to build from source, if you need something none standard.

Re:ZFS on Linux

[BitZtream]

Sigh ... yes it does, learn how to use zdevs properly.

also

zfs set sync=always zfs/filesystem

But thats retarded if you have any idea how the ZIL works.

I have a commodity hardware machine with 4 best buy disks, 2 of the cheapest SSDs I could find, a decent mobo. I use it as a SAN over dual gigabit links to a VMware ESXi server. One link will easly move 130-140MB/s and last time I looked it was averaging about 12k IOPs/second. We've done 4 times that in testing. Thats over NFS, with deduplication turned on. Basic commodity hardware.

I'm fairly certain you have no idea what a properly configured ZFS setup is capable of. You're probably one of those guys who tried to run multiple terabytes of data on a ZFS machine with a couple gigs of ram and no SSD or RAM cache and gzip-9 compression, sha256 check summing and deduplication.

Re:ZFS on Linux

[MightyYar]

Yeah, sorry I wasn't more clear. pkg_add still fits into the ports framework, though - so really you can use the two simultaneously and things generally work... most of the time. I've taken to just using ports unless I'm in a huge hurry. You can technically use apt-get to bulid from source as well (-b flag).

Re:ZFS on Linux

[jd]

ReiserFS works extremely well on small files, so I like using it for /tmp. I would use it for /etc, /usr/share and /usr/man as well, but I've had problems where it has corrupted the FS. I can afford to lose /tmp, but /etc is another matter. If I felt safer with it, I'd consider ResierFS to be near-obligatory for any directory in which short files were the norm.

(More partitions is no big deal with gparted-style partition tables. More than 6 partitions was more of a bother under the MSDOS-style partition tables. The overheads are a pain, but in some case the space efficiency of specialist FS' in their niche will cover this. In other cases, the speed improvement is worth the space penalty, especially on modern drives.)

Btrfs is promising but the repair tools have been promised for a very long time and I'm beginning to think that Oracle might try to recover the cost of development by charging for "professional-quality tools" in the same way as is done for MOSIX. (I can sympathize with the MOSIX stance as anyone building a large cluster probably has spare cash.) Oracle has done something along these lines with MySQL and I don't trust them to not do the same anywhere they can.

ZFS has its own niche - it essentially folds hardware RAID, LVM and a logging filesystem into one unit, with tight integration and resulting efficiency gains, but being run in userspace does harm most/all of those gains. With work, though, it could be made into an excellent filesystem where you need that kind of dynamic - such as for database work, or software development. You can't predict in such cases what the final requirements will be and migration is a major headache.

The P9000 filesystem is fascinating and may well be great in certain clustered environs. There are other clustered filesystems but I'm beginning to think Lustre has lost its way, POHLFS has recently been reinvented and can't be regarded as stable yet, Gluster hasn't been heard of for a while and the "top" commercial cluster FS for Linux (Polyserve Filesystem) died when HP bought the company owning it, with suspicions of murder. The field is a bit of a mess.

Re:ZFS on Linux

[FreekyGeek]

Well, you better get used to ZFS. In Solaris 11 not only is it the default, but booting from anything but ZFS is not supported. UFS is supported as a legacy filesystem so I guess you can keep using it for a while, but Oracle is definitely pushing 'ZFS everywhere.'

ZFS works fine on a SAN LUN. It's main drawback in my opinion is that it isn't a clustered filesystem, so sometimes Veritas is still required.

Issue is not implementation

[Anonymous Coward]

The issue with ZFS and Linux has always been more about copyright than implementation.

Re:Issue is not implementation

[Liberum Vir]

The whole GPL/CDDL issue is still around, however, since the CDDL code is not added to the Linux Kernel, but instead a loadable kernel module distributed separately, it is possible to satisfy both the GPL of the Linux Kernel and the CDDL of ZFS and DTrace. Because of the incompatibility of CDDL with the GPL, you could not distribute a complete system using of Linux, ZFS, and DTrace. You can, however, distribute packages to allow people to build it themselves. This is what the authors of these projects have done.

Re:Issue is not implementation

[unixisc]

Which is what makes me wonder - why are people trying this? If ZFS is what is needed, why not go w/ a perfectly good and compatible FOSS unix, such as either OpenIndiana or FreeBSD? The former has the same license as ZFS, while the latter's license is not incompatible w/ CDDL.

If the objective was to get some of the nice things from the Linux world, that might be possible w/ Debian kFreeBSD, which has some support for ZFS. Might be a better option than Linux.

Re:Issue is not implementation

[Anonymous Coward]

Linux has better HW support.

Re:Issue is not implementation

[Zero__Kelvin]

"Which is what makes me wonder - why are people trying this? If ZFS is what is needed, why not go w/ a perfectly good and compatible FOSS unix, ..."

Because a filesystem isn't an operating system. You are asking why people don't throw the baby out with the bathwater.

Re:Issue is not implementation

[Thumper_SVX]

Because a filesystem isn't an operating system. You are asking why people don't throw the baby out with the bathwater.

Ah, but the implementation under the various operating systems can have a lot of effects that are relatively unknown until you hit them. Yes, the ZFS Kernel Module programmers are excellent about reponsiveness, but that doesn't help much when you've got a ZPOOL that's suddenly dropped offline and because the other ZPOOLS are still running you can't take an outage to reboot.

I've played with ZFS under Ubuntu 10.04, BSD and OpenSolaris (and recently OpenIndiana) and there are "quirks" to each implementation that you either end up abandoning one implementation or working around them. None of them are perfect, but my personal preference these days is the OpenIndiana implementation. The kernel is pretty much written from the ground up to support ZFS instead of having it being a plug-in. Now there are plenty of things I don't like about OI but for my uses it works just fine.

Re:Issue is not implementation

[Anonymous Coward]

This is debatable. Linux has better support for video cards. For networking devices or disk controllers, it's a wash.

OK Howto article, but missing key points

[laffer1]

I just looked at this article as my employer uses Debian and Ubuntu heavily and I've been pushing for ZFS on our file servers. There is no mention of ZFS version, the feature set available, or even a link to the source material.

There isn't much mention of how to use ZFS. I happen to know most commands, but I think this article would be difficult for a beginner even though it seems to be targeted at that demographic.

Re:OK Howto article, but missing key points

[Darik]

I just looked at this article as my employer uses Debian and Ubuntu heavily and I've been pushing for ZFS on our file servers. There is no mention of ZFS version, the feature set available, or even a link to the source material.

ZoL is based on ZFS version 28 from the last open Solaris release, and currently integrating Illumos as its upstream.

There isn't much mention of how to use ZFS. I happen to know most commands, but I think this article would be difficult for a beginner even though it seems to be targeted at that demographic.

It looks like the Slashdot editors are doing this blogger a favor by linking to a mostly empty article.

At a minimum, this article should link to the ZoL home page, the ZoL Launchpad page for packages, and maybe the ZFS introduction or another tutorial.

Re:OK Howto article, but missing key points

[Liberum Vir]

The zfsonlinux.org site is what I used to set this up. The pool version is reported as 28 and filesystem version as 5. My apologies there. My objective with this article was to create a simple to follow tutorial on how to get it set up. My objective never was on how to use ZFS or DTrace. I'm intermediately familiar with ZFS, and very much a beginner when it comes to DTrace. Again, a "how-to use" tutorial was not my objective. There are far better sources for documentation on use. My article is just to get people an environment set up in which to test the platform. I had been toying around with the idea of ZFS and DTrace on Linux for a while, however, there was nothing like my article to help people in setting up a testbed system. So, that's what I wrote once I figured out how to make it all work together.

Re:OK Howto article, but missing key points

[Liberum Vir]

Thanks for the recommendation on adding in those links. I've added them. You'll find them at the end under "Further Reading and Sources". As well as a shout out to you. Cheers!

Re:OK Howto article, but missing key points

[Anonymous Coward]

If a debian shop, kfreebsd might be the way to go.

Debian is kernel agnostic, several choices including freebsd (kfreebsd should be ready for when wheezy goes stable).

You get a fully baked distro with everything you expect from Debian (which is ummm... pretty much everything you could ask for already vetted and pre-packaged-- with ease of management fully thought out), and the stability of zfs on a freebsd kernel (which lately has become pretty solid).

I work in a Debian shop too, and really prefer Debian to any other distro out there (though a soft spot still for slack), but I would probably just go with freebsd at work, if production ZFS was desired-- larger userbase (probably) than kfreebsd debian will ever garner.

Re:OK Howto article, but missing key points

[Anonymous Coward]

Next time don't be such a lazy bastard. You get paid while your readers do the legwork for free. That's fucked, try earning your keep by performing basic due diligence like this. No one expects you to be perfect but at least TRY.

Too bad you're replying to the person that submitted (and wrote) the original article, and not the Slashdot editor that posted the story, dumbass.

Re:OK Howto article, but missing key points

[unixisc]

Debian seems to argue this as well - they point out that in Linux, support for ZFS is unlikely to be there in the mainline kernel.

Why bother?

[Anonymous Coward]

Just use solaris? ZFS, dtrace, SMF, and about 100 other things you dont have on Linux (I've been a Solaris and Linux user since about 1994).

Re:Why bother?

[Anonymous Coward]

Why bother with Solaris?

ps.: long time Solaris user here. don't see the appeal anymore. the illumos/openindiana project suffer form the same short slightness that caused opensolaris to fail (couldn't expect anything different if the same sun people are major forces behind it).

Re:Why bother?

[jd]

There's things Solaris has that Linux doesn't, there's things Linux has that Solaris doesn't, there's things Inferno has that neither Solaris nor Linux have, there's things Hao Ya tea has that no OS will ever contain. Interoperability is best when essentially external components are portable, tea is best when hot and not Earl Grey.

Re:Why bother?

[Anonymous Coward]

Could you elaborate? OpenIndiana is entirely community based and has none of the original OpenSolaris people behind it.

Huh?

[Score Whore]

So there's a list of 10 steps to install zfs and that's it? Didn't do anything? zfs/zpool upgrade -v? zvols? zfs send/receive? snapshots? rollback? Scrub? Performance tests? Compression? Encryption? Can I export my pool from my Solaris 11 SPARC system and import it into linux, make some changes and then move it? L2ARC support? Separate ZIL support? Case sensitivity?

I know this isn't exactly a great comment, but is it at all possible that someone make a judgement as to the value and truth of a submission before putting it up?

Re:Huh?

[ZorinLynx]

Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. It performs fairly well in my testing so far. Yes. Yes. Yes, if the pool version is below the currently supported Linux port's version (28). Yes. Yes.

Granted, we haven't been using it long, but so far it's been fairly stable and capable.

http://zfsonlinux.org/

Re:Huh?

[Anonymous Coward]

Encryption? So you're beyond v28? Or are you talking about encryption at some other level than zfs?

Re:Huh?

[Zero__Kelvin]

The submission was titled "Making ZFS and DTrace Work On Ubuntu Linux". You're thinking of the article you are going to write called "Now that I got ZFS and DTrace working on Ubuntu Linux Thanks to Some Other Guy Being Nice Enough To Help My Lazy Ass, Here's What I Did With It".

Re:Huh?

[bill_mcgonigle]

Mostly all yes though performance is spotty in some use cases. The code is evolving rapidly - it's always been safe, but sometimes new releases have regressions. The developers usually fix these quickly.

I'm using a dm-crypt volume under ZFS where I need encryption as that hasn't made it out yet.

I've been using it in test since August, for in-hosue production since January, but I'm not yet recommending it to clients because of the rough bits.

Now that it's been Oracled...

[Just Some Guy]

I've been running ZFS on FreeBSD for a few years and it's lived up to its promises, but I think I'll be migrating off of it. The problem is that I trusted Sun. They did some goofy things, but you knew where you stood with them. They release ZFS under an Open Source license? You could take them at face value and know that you were allowed to use it. But now that Oracle holds the reins, I have no desire to depend on any Sun-borne projects anymore. Yes, ZFS is Open Source. So was Java, and Google just spent roughly a bazillion dollars defending themselves for using something that looked like it. I can't afford to take on a case like that.

Other than the Oracle-owned btrfs, what ZFS alternatives are available and ready for use today?

Re:Now that it's been Oracled...

[Liberum Vir]

Well, I know that it may not be as widely known as ZFS, but I would recommend checking out NILFS2. It certainly seems interesting and seems like something that could solve many of the same problems ZFS sought to solve.

Re:Now that it's been Oracled...

[bmo]

So was Java, and Google just spent roughly a bazillion dollars defending themselves for using something that looked like it. I can't afford to take on a case like that.

So you take the Oracle vs. Google case as Oracle eventually going after individual users of legitimately licensed code?

Nonsense.

As much as I think Larry Ellison is a douchebag, he is motivated by profit. The results of this last case were less than optimum for him, going away from the case with bupkis and a bunch of fees from BS&F. Alsup also established he fact that independent implementation of APIs are not copyright violations, ever, under current law, which had not been proven until now, which is a big win for everyone including Google, and a stupendous loss for Oracle.

Larry Ellison learned an expensive (David Boies doesn't come cheap) lesson here, that even his bluster and hubris doesn't win court cases.

Google was not the loser here.

ZFS and btrfs have free licenses and it's tough to put the worms back in the can once something is under a free license. Forks happen. Look at what happened to OpenOffice and Libre Office. Sure, Oracle can close off future code, but Very Useful Stuff like this gets forked by the community. There are enough smart people poking around in the guts of ZFS and btrfs that *do not* work for Oracle and the projects will continue on in the community even if only to give Oracle the finger.

Your fears are overblown.

--
BMO

Re:Now that it's been Oracled...

[Anonymous Coward]

Check out Dragonfly BSD's Hammer FS. It has offline dedup that doesn't require much mem at all (unlike ZFS's online dedup), some other features to ensure data integrity. It is not the life the universe and everything approach of zfs and btrfs, but it has much of what would be traditionally thought of as the fs part of the functionality.

Re:Now that it's been Oracled...

[Anonymous Coward]

everything looks good btrfs!

Re:Now that it's been Oracled...

[Anonymous Coward]

I don't trust Oracle at all, but I do trust the CDDL. The CDDL allows others to use the copyrighted code, and provides patent protection.

That's huge.

Oracle would never have chosen the CDDL. While they stopped releasing new versions, they can't put the toothpaste back in the tube: projects can keep using the previous CDDL'd code.

This will not be another Oracle vs Google, provided people use the CDDL'd code to get both copyright and patent protection.

You can't say the same about SystemTap and btrfs. Do either infringe on the many patents Sun filed on DTrace and ZFS, which Oracle now owns? If I'm going to adopt btrfs (say, in a product), will Oracle release it under the CDDL to afford the same patent protection that ZFS provides?

Re:Now that it's been Oracled...

[subreality]

Suing users is completely plausible for Oracle. The case would be meritless, but I don't have the time and money to go up against Oracle in court.

Re:Now that it's been Oracled...

[bmo]

1. You are not that important
2. More serious answer: Going after end users is a waste of resources and Larry isn't as dumb as Darl McBride.

--
BMO

Re:Now that it's been Oracled...

[subreality]

1. I'm a disposable cog, but my employeer may make waves...
2. ... and if those waves compete with Oracle's, they may start an IP pissing match.

Re:Now that it's been Oracled...

[puppetluva]

David Boies has had a frustrating decade in tech work. He won the Microsoft antitrust case only to have it overturned. He lost representing SCO va unix, he lost representing napster and now he lost it for Oracle. He's a legal superman, but he should stay away from tech for a while. . .

Re:Now that it's been Oracled...

[bmo]

See, here's the thing...

If you, or your company, files lawsuits repeatedly for meritless reasons, you can rapidly find yourself being classified as a "vexatious litigant" where you need a court's approval before even filing a lawsuit.

Sure, Oracle can sue for anything. If they make an ass of themselves in the system, the system can slap them back. It happens. It happens with these "Sue 5,000 John Does" that various extortion-firms have filed against copyright infringers, like what happened to Righthaven, which no longer exists as a result.

If you or your company are using code as licensed, I don't see how you are under any risk. If it was as bad as you think it is, companies would simply have assassination squads instead of lawyers.

--
BMO

Re:Now that it's been Oracled...

[Anonymous Coward]

Not so fast.

While the btrfs engineers have not pursued patents, it is a ZFS knock-off, and Sun patented the hell out of ZFS (all those $2000 patent bonuses for the ZFS engineering team), and Oracle now owns those patents. btrfs is GPLv2, which does not cover patents. ZFS is CDDL, which does.

Re:Now that it's been Oracled...

[jd]

My preference would be for the wide range of open source u*ix-style OS' to get together, hammer out a cross-os VFS layer and thus reduce the discussions of FS' to technical points on the FS itself, eliminating the OS from the equation. There is nothing inherent about mapping/remapping/versioning/distributing physical data in logical files to blocks of data that is the least-bit OS-specific.

Re:Now that it's been Oracled...

[Guy Smiley]

The Oracle/Solaris version of ZFS is dead to the open-source world. Almost all of the original ZFS developers left Oracle and are working at other companies developing OpenSolaris (now called Illumos). That means that the open source version of ZFS is getting better all the time, and the Linux ZFS code is pulling fixes and features from Illumos, so it isn't just sitting still either.

Oracle doesn't "hold the reigns" on ZFS anymore, though they may like to think that. That is the benefit of open source software - once it is free, nobody can stop you from using it. In this regard, CDDL is even better than GPLv2, since CDDL explicitly grants a patent license for all patents embodied in the ZFS code.

As for stability, there are full time developers working on deploying ZoL on a 55PB Lustre+ZFS+Linux parallel filesystem running 1TB/s for one of the largest computers in the world, so you can bet it will be stable enough for your desktop.

Re:Now that it's been Oracled...

[bmo]

David Boies signed the Contract From Hell vis-a-vis SCO.

The contract had a cap on fees, and BS&F was to represent SCO vs IBM until the heat-death of the Universe, because they stood to gain a portion of the FIVE BILLION DOLLARS IBM was supposed to cough up.

--
BMO

Re:Now that it's been Oracled...

[bertok]

Other than the Oracle-owned btrfs, what ZFS alternatives are available and ready for use today?

The only serious filesystem with similar features (B-Trees everywhere, hashing for integrity, etc...) that I know of is Microsoft ReFS. It's still beta, and will be missing key features of even NTFS when released, so it won't exactly match up to a mature filesytem like ZFS.

Re:Now that it's been Oracled...

[Anonymous Coward]

Why just tech? He lost it for Algore as well

Re:Now that it's been Oracled...

[unixisc]

That's a great idea - HAMMER FS. Comes under the BSD license, and is developed by DragonFlyBSD to start w/.

Question - is Veritas' file system under any open source license, or is it only closed source?

Re:Now that it's been Oracled...

[rev0lt]

Oracle has no way of "taking back" the already available source. The only problem is that any new ZFS features implemented by the FreeBSD team will probably be incompatible with the Solaris version. But, looking at v28, you may have enough funcionality for a decade (there is encryption missing, and I think dump still not works as expected, but that's about it).

If you want a BSD-licensed alternative, try DragonFly BSD. It comes with the HAMMER filesystem, that provides some (most?) of ZFS funcionality.

Re:Now that it's been Oracled...

[jez9999]

Other than the Oracle-owned btrfs, what ZFS alternatives are available and ready for use today?

ext4?

Re:Now that it's been Oracled...

[multipartmixed]

Wow, it's almost like the dude got 419d!

Re:Now that it's been Oracled...

[unixisc]

One more alternative - AdvFS. That was the file system of DEC's OSF/1 a.k.a. Digital Unix a.k.a. Tru64 Unix. When HP decided that they had nothing to gain by moving it to the Itanium, they made this file system GPLv2 in 2008. It is available on Sourceforge.

I have no idea about how it compares to Veritas, ZFS, XFS, HAMMER, BTRFS, ext4, or anything else. That's one more thing you could go to under Linux, or even BSD, since OSF/1 was originally a BSD based unix.

Re:Now that it's been Oracled...

[bmo]

There is no scam without a mark.

--
BMO

Liicensing?

[ducomputergeek]

I always thought the hold up on ZFS and DTrace on linux was the fact the CDDL and GPL didn't play nicely with each other. It was never a technical reason.

I've been running both on FreeBSD for a couple years now. Still don't have any production machines with ZFS yet, but I've found DTrace to be a life saver on more than a few occations.

Used both on Linux: ZFS is great, Dtrace unstable

[GrumpyOldMan]

I use ZFS on Ubuntu 11.10 in "production" for my main workstation and fileserver with a 3x3TB raidz pool with an L2 ARC. I/O is blindingly fast, and it has been rock solid. It serves about 10 machines, and feels an order of magnitude faster than the md/lvm based xfs array it replaced.

I write 10GbE drivers for Linux, MacOSX, FreeBSD and Solaris. I make heavy use of Dtrace for both debugging and performance analysis. I feel naked without Dtrace, and I've used the linux dtrace a few times for debugging. Unfortunately, I've never had dtrace run on linux for more than a few minutes without crashing a machine. This is not necessarily bad, and often just a few seconds is all I need. But I would never run linux Dtrace on any production machine, whereas I use it all the time under Solaris / FreeBSD and MacOSX and often have customers run Dtrace probes on those OSes to diagnose issues.

Low bar for entry

[outZider]

So an article lacking knowledge of the technologies, any sort of testing, anything beyond "make install" or "apt-get install", will make it to the Slashdot homepage? This person openly admits that they didn't test ZFS beyond creating a zpool, and they don't know enough about DTrace to try... anything.

As an aside, why was Linux capitalized, but Solaris was not?

I don't see Solaris users migrating

[mysidia]

3) Partition the new drives.

)9 .... “sudo zpool create zfs-blog raidz /dev/sdb1 /dev/sdc1”

Ha ha ha. You know part of the magic of ZFS is management of the entire disk drive. No partitioning

Look: The ZFS on Linux project is a noble effort, and I am sure many Linux users will eventually benefit, and it will maybe be good enough for them to not switch to Solaris.

But none of this stuff is production quality yet on Linux, and the performance VS Solaris is questionable. Linux doesn't have the components to implement all the integrations and beneficial "layering violations" ZFS has on Solaris; and they won't for a long time, Sun spent over 10 years and tens of millions on development of Solaris and their filesystems, I don't think it's reasonable to expect to see the same kind of polish on Linux for ZFS or Dtrace at this point, and we don't; lots of work and funding could change the situation, but for now the Linux implementation doesn't hold a candle to the Solaris implementation.

You can go ahead and add: COMSTAR, SMF, FMD, and an excellent native NFS server implementation, to the list of things Solaris has but Linux doesn't.

The Linux implementation of even ZFS is less mature and sheds benefits of ZFS. Including ease of management. 10 commands just to get setup? Geez.

With Solaris, you have ZFS out of the box and you just do "zpool create tank mirror c1t0d0 c1t1d0 mirror c1t2d0 c1t3d0

No "partitioning" ZFS manages the disks, including disk cache, and fault management.

You can be pretty darn sure zfsonlinux doesn't have the same level of FMA reporting / fault management capabilities.

Re:I don't see Solaris users migrating

[Ash-Fox]

Ha ha ha. You know part of the magic of ZFS is management of the entire disk drive. No partitioning

You don't need to partition, you can use /dev/sdb and /dev/sdc instead. My question to you is, is Linux doing something that Solaris doesn't support? ie: Capable of running ZFS in a partition instead of using whole disk?

But none of this stuff is production quality yet on Linux

Define what will make it 'production quality' please.

You can go ahead and add: COMSTAR, SMF, FMD, and an excellent native NFS server implementation, to the list of things Solaris has but Linux doesn't.

I definitely agree NFS on Linux still needs work. Hell, it's still possible to lock up a server if you don't treat NFS just right. But, to be honest, I'd rather see the CIFS support extended more to be honest, mostly better security models offered on it (not only with authentication, but ACLs and file locking that doesn't suck and I couldn't give a crap about iSCSI anymore).

performance VS Solaris is questionable

Do you have any benchmarks regarding this specific setup. I'm genuinely curious. The stuff I find is regarding other implementations involving FUSE.

Linux doesn't have the components to implement all the integrations and beneficial "layering violations" ZFS has on Solaris

Seriously, /dev/sdb, /dev/sdc, access directly to the hard drives without any 'layering violation' issues, been in Linux since the beginning.

I don't think it's reasonable to expect to see the same kind of polish on Linux for ZFS or Dtrace at this point

Is this the same kind of polish that things like cron get on Solaris - Where it still doesn't support attributes like @reboot, but it's all good because the code is old and so it's matured just right or something? (Seriously, I've been told this is why the cron in Solaris is superior - note that I did not say Open Solaris).

With Solaris, you have ZFS out of the box and you just do "zpool create tank mirror c1t0d0 c1t1d0 mirror c1t2d0 c1t3d0

Actually, you can do the same with this implementation, only difference is you need to give the specific device paths like /dev/sdb instead of c1t0d0.

Re:I don't see Solaris users migrating

[Lennie]

> > Ha ha ha. You know part of the magic of ZFS is management of the entire disk drive. No partitioning
>
> You don't need to partition, you can use /dev/sdb and /dev/sdc instead.

Actually, i tried that, it give me an error.

Re:I don't see Solaris users migrating

[mysidia]

You don't need to partition, you can use /dev/sdb and /dev/sdc instead. My question to you is, is Linux doing something that Solaris doesn't support? ie: Capable of running ZFS in a partition instead of using whole disk?

Solaris is capable of running ZFS on a disk slice, and it's something I will rarely use, but I will use it for the boot media, when running the ZFS boot volume off a CF card, I will mirror it to a small slice on a pair of hard drives, and have another slice on those two drives mirrored for providing the "System log" / scratch disk partition.

What you must never do is utilize a SSD slice for the ZIL. It "works", but can cause problems and has serious consequences regarding data integrity in case of a system crash. Using disk slices is not supported for L2ARC or ZIL, even though nothing technically prevents it, you don't do that.

You can use a disk slice as a VDEV member for a ZFS pool on Solaris, but you lose a number of the benefits of ZFS by doing so, for example: ZFS will no longer be able to turn on writeback buffering for the disk and manage the write cache for the device intelligently, because that means you could have non-ZFS partitions on another slice.

It only makes sense to do that on cryogenically cold partitions, such as system logging/scratch disks, never on mission critical production data datastores.

Define what will make it 'production quality' please.

Production quality means I can safely load mission critical applications on the datastore, in well-understood hardware configurations, eg disk JBODs attached to a SAS controller, where the application requires 99.9% or better uptime, without risk of getting fired because of that decision, and there are no serious doubts about the integrity of the data, availability, and performance, that are not understood about the filesystem implementation, or integration with the OS and hardware support regarding the OS combination of the filesystem and certified hardware, all the availability features you expect are there and reliable, such as fault management, IP Multipathing, Storage multipathing, etc; the Solaris ZFS implementation has been used in production for well over 7 years in this case, the ZFSOnLinux implementation is brand new.

In the case of Solaris, the ZFS implementation has proven stability, performance, and track record for critical applications. I can also implement ZFS in an HA cluster on Solaris with shared storage, with proper design, there are no serious questions about integrity of my data and availability, that don't have good answers.

ZFS on Solaris might not work right with all choices of hardware, but basically, my data is safe, unless I use reliable components, or I experience hardware faults that exceed the amount of redundancy in the ZFS Pool (e.g. 2 drives fail in a 2-way mirror, 3 drives fail in a raidz2 pool).

I definitely agree NFS on Linux still needs work. Hell, it's still possible to lock up a server if you don't treat NFS just right. But, to be honest, I'd rather see the CIFS support extended more to be honest, mostly better security models offered on it

Solaris + ZFS has a better CIFS implementation than is available on Linux, , but honestly when clients are running Unix, NFS is the better choice. When CIFS is needed, for Windows clients, the only real game in town is Windows 2008 server, for most applications. It's fine for simple file sharing, as long as your requirements are simple -- but when you start running third party database applications, support will blindly blame the CIFS server for its woes and refuse to support the application until its moved, "It's not supported, the share needs to be on a Windows server"

CIFS leaves much to be desired in the security department, whereas NFSv4 can utilize kerberos authentication, and RPSEC-GSS is potentially available; CIFS traffic cannot be encrypted, without tunnelling or IPsec

Re:I don't see Solaris users migrating

[obrith]

What you must never do is utilize a SSD slice for the ZIL.

I've been looking for a valid reference for this statement for a while. I've had a couple people tell me this and simply insist with no valid reasoning.

I understand that ZFS can't send a cache flush to a slice, but if I'm using a SSD with a supercap (say Intel 320) I have never heard any valid argument or reference. I have a LOT to gain from not allowing ZFS to 'use' the whole disk; My 300GB SSD can write something in the neighborhood of 600-800TB if I hand the whole thing to ZFS before I run out the media wear indicator. The same disk vastly under-provisioned (giving ZFS a 15GB slice) my media wear indicator will last about 4.2PB (yes, 4200TB). Under my VMware all-sync NFS load this drive might last me 6 months if I give the whole thing to ZFS where I should get years slicing it. On top of this, the drive performs approximately twice as fast vastly under-provisioned.

Can you give me a valid reasoning I shouldn't do this... or maybe some references?

Re:I don't see Solaris users migrating

[mysidia]

I hand the whole thing to ZFS before I run out the media wear indicator. The same disk vastly under-provisioned (giving ZFS a 15GB slice) my media wear indicator will last about 4.2PB (yes, 4200TB).

There is no point in getting a 300GB SSD to use as ZIL, because the maximum amount of disk space that ZFS will use on the ZIL is one half of the system's RAM; you would literally need 600GB of RAM, before a 300GB could possibly be utilized as ZIL by ZFS.. It would not be necessary to "slice" or "partition" your oversized SSD to limit the amount of space ZFS would use.

L2ARC is different, but I would recommend you spend the money on the RAM first, get at least 128gb of RAM, before thinking about buying a 100gb L2ARC, and make sure you scale your CPU power with the amount of RAM (because Solaris will utilize more CPU under certain circumstances as a result of having more RAM). The problem for now is that the L2ARC is not persistent, the L2ARC starts at a blank slate after every system boot, so it has many of the same disadvantages as RAM, as far as caches go, that cache takes a lot longer than the ARC to warm up, and it's not that performant.

I understand that ZFS can't send a cache flush to a slice, but if I'm using a SSD with a supercap (say Intel 320) I have never heard any valid argument or reference.

If ZFS cannot send a cache flush for your ZIL device, you must ensure the SSD's write cache is disabled, which is absolutely terrible for both your ZIL performance and for the longevity of the SSD. For that reason and others, the developers on zfs-discuss have in the past stated that using a log device that is not a dedicated device is a not supported configuration; in response to people who showed such a configuration and indicated they were having some performance problem.

The Solaris documentation for Log and Cache device supports states that L2ARC and ZIL dedicated devices may be utilized. ZIL devices may be mirrored, ZIL devices cannot be RaidZ; mirroring by ZFS alone is the only supported ZIL redundancy. L2ARC devices cannot be mirrored by ZFS. It has never been stated by the documentation that a partition or slice is a supported configuration for ZIL devices, the documentation says dedicated Hard Drive or SSD, which means dedicated hard drive or SSD, not "whatever you like". The ZFS default is that the ZIL resides on the main pool; your main pool can have slices as VDEVs, this is safe on ordinary well-behaved hard drives, because ZFS does not enable any write cache, but administration is complicated by that configuration; SSDs are "special" because many of them enable write cache without permission which is unsafe; it's unsafe to utilize SSD on a disk slice for anything in ZFS, root pool, data pool, etc, without ensuring cache disablement, it's not just use as ZIL that is unsafe with a SSD slice.

It is unsafe to utilize even dedicated SSD devices and other storage devices that do not obey CACHE FLUSH commands properly.

The impact of SSD cache disablement on device longevity and performance is so much so that the value of having a dedicated SSD ZIL goes away; At that point, you would be better off dedicating a mirrored pair of 600gb10kRPM SAS drives for the ZIL and not slicing them up at all, which is less expensive than buying a pair of 300gb Enterprise SSDs, that's for sure.

The use of a SSD with a supercap is strongly recommended for the ZIL as it will reduce wear, but you need to have a configuration where the caching is safe. If you fail to disable the write cache for the SSD in such a scenario where it's unsafe, what will happen, is that in case of power failure, there is a chance that you will have some data flushed from the previous transaction group and some data flushed from the current one, in other words, your ZIL will be in a state that will cause data loss, and that ZFS cannot recover from automatically, you will be potentially unable to import your pool when you boot back up, and the recovery procedure will be painful, due to the duration of downtime.

Strange article

[Anonymous Coward]

No attribution to the original authors of the tools, and the dtrace is a one-line change to the original dtrace-for-linux port.

“sudo make all”?!

[Anonymous Coward]

Why “sudo make all” when “make all” will do?

Wrong about license

[tlambert]

The DTrace integration is via a kernel module, so the license on DTrace is irrelevant..

There are a couple of interfaces in Linux that should be externalized for getting stack tracebacks into user space in a standard manner without caring about binary architecture (they are currently static). I've personally used a modified Linux with DTrace mods and these functions externalized, and it's rather stable and usable. Specultive tracing is also a lot better for finding the origin of some random errno in the kernel, or who in user space is calling gettimeofday() a bazillion times in order to time stamp X events.

Obligatory disclosure: I was on the team that did the DTrace port to Mac OS X.

-- Terry

Re:Wrong about license

[bheading]

Are you sure that is correct ? I'd have thought the license is very relevant. The question of whether or not something is a kernel module, or dynamically linked or whatever, is extraneous to the important question which is whether or not a derived work is being created and therefore whether or not the license can be satisfied.

By distributing a kernel module, you are distributing a derived work in a way that cannot simultaneously satisfy the requirements of both the CDDL and the GPL. That's why you'll never see these ZFS or dtrace modules included as part of a distribution. Which is a major hamper on the development of this work, since it is the major distribution vendors and other major firms (HP, IBM and so on) who fund the lion's share of the effort

The guys who are distributing binary packages of these components are running the very real risk that Oracle will sue them. Sun licensed these components under the CDDL rather than the GPL for a damn good reason, they did not want to allow Linux to cherry pick the best bits and then use them to defeat Solaris in the marketplace.

Re:Wrong about license

[Zero__Kelvin]

I sure hope that Sun doesn't get defeated in the marketplace!

Re:Wrong about license

[Guspaz]

They're not distributing binary packages. The Ubuntu packages have the source code and the DKMS config to build the thing, and have the package pre-requisites such that everything "just works".

Re:Wrong about license

[Lennie]

Sun doesn't exist anymore, it's all Oracle now.

Re:Wrong about license

[Zero__Kelvin]

Reread what I wrote with the knowledge that I already knew that and you'll figure out what that whooshing sound was right before you replied ;-)

Just one complaint

[Guspaz]

I've been using zfsonlinux for a few months now, ever since I migrated my file server from OpenSolaris to Ubuntu Server, and I've generally been pretty happy. It's been stable and fast (faster than osol was, anyhow). My only complaint is that mounting filesystems on boot seems eternally broken.

In previous options, there was a config file option for a workaround, and the filesystems usually (but not always) got mounted on boot. Then that solution was removed in favour of an updated mountall package; unfortunately, this new solution never works. I'll boot the system, no filesystems mounted, but running mountall from the command prompt gets everything mounted OK... Sigh.

Re:Just one complaint

[bill_mcgonigle]

My only complaint is that mounting filesystems on boot seems eternally broken.

yeah, there's udev work to be done yet. I think there's an open bug for that.

I currently use a shell script to interrogate and export/import my zvols to get everything up at boot. Messy.

Re:Just one complaint

[Guspaz]

It seems like I should probably be able to add mountall to a startup script, but I shouldn't have to.

Re:Just one complaint

[Taigitsune]

Try exporting your zpool and importing it using the following: zpool import -d /dev/disk/by-id This assumes you only have a handful of disks and don't mind identifying them by model and serial number. Also make sure the mount point doesn't exist after export as ZFS creates the mount point when it imports the zpool.

Dtrace is amazing

[Anonymous Coward]

It's a must have for a sysadmin.
How many times have you had application folks or devs come by your desk and complain "things are running slow".
Good to be able to find the root cause and sometimes propose a workaround under an hour.

Now that we are making the switch to Linux, people complain that we're not quite as fast as with Solaris when it comes to finding out why stuff is running slow..

So yeah in the perfect case Linux might have a slight edge over Solaris in performance, but that doesnt matter when the tools to diagnose shitty in-house apps are missing.

Re:Used both on Linux: ZFS is great, Dtrace unstab

[jovius]

I've been using ZFS on Linux also a while on my Ubuntu based backup/media box. No problems so far, and the average transfer rate of a 100 GB disk image has been 50 MB/s from internal drive A to internal drive B (non RAIDed, Asus E35M1-I DELUXE Mini ITX with 8GB of mem and 2*Western Digital Caviar Green 2TB SATAIII 64MB,). The CPU usage hits maximum while transferring, and ZFS also uses most of the RAM quite efficiently.

not a serious OS

[rubycodez]

OpenIndiana has only made three "development releases" since 2010, it is not a production grade system. Just a hobbyist system.

So what about performance ?

[Lennie]

I did a quick test with 2 identical VMs on my desktop with Intel SSD, I installed the ubuntu-zfs as from the article and I installed btrfs-tools.

The VMs have 4 CPUs and 4GB of memory, 3 virtual disks.

The btrfs has RAID1 data and meta data, the ZFS setup used RAIDZ as in the article:

mkfs.btrfs -m raid1 -d raid1 /dev/vdb1 /dev/vdc1

(I needed to create the partitions, for some reason the ZFS version didn't want to work without it)

My quick stupid test, create a large file:

ZFS:
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 16.8489 s, 31.1 MB/s

real 0m16.853s
user 0m0.000s
sys 0m0.480s

btrfs:
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 15.232 s, 34.4 MB/s

real 0m15.234s
user 0m0.000s
sys 0m0.640s

Re:So what about performance ?

[Lennie]

I should add that ZFS uses more CPU.

Good Article

[Thumper_SVX]

Congrats... this is a good summary on getting these working under Ubuntu. I did the ZFS install "naked" (without a summary as good as this) with a 10.04 box about a year ago and it has run great guns. Now, having said that it's good for what I use it for which is a temporary location to dump my SQL backups to from a large email archive using dedupe prior to running it off to tape... and another zpool mounted as an archive VMFS volume through NFS to our VMware farm so we can archive decommissioned virtual machines for 30 days prior to deletion per our policy. I am not 100% convinced I would use it for anything production though; supportability is still an issue with this and as such I remain a little dubious whereas with most of our system I can call a vendor and have them fix it. As the storage admin I find this a great way to keep up with the demands for storage while having a relatively transparent way (for my admins) to put stuff into a place where it doesn't take up so much space.

Now having said that there are some caveats; as the zpool gets really large the ability to delete files becomes slower and slower when it's deduped. This is because a lot of database transactions take place to remove the files particularly when there's a lot of deduplicated blocks... and this problem is a lot worse under Ubuntu than it was under OpenSolaris (which is where I first played with ZFS). There are times also that when reading the SQL backups to dump them to tape it can make both storage pools unresponsive enough that VMware drops the NFS datastore and I have to manually remount them. Far less than perfect... but good enough for what we use it for.

I have recently taken a decommissioned physical server (a DL380 G5 with two processors and 16GB of RAM) and put OpenIndiana on it to play with ZFS some more and it is working fantastically well. In my tests though it still has the slowdown issues, high utilization in one pool won't cause the other pool to grind to a halt when both are deduped. Also, it's been nice to (at least in test) create a ZVOL on my ZFS and present it through fiber-channel to my VMware hosts as a potential replacement for the NFS volume on Linux (I have only Emulex cards, and I have yet to see a properly working Emulex target mode under Linux). So far my testing has gone marvelously and I have found dedupe rates to be about the same as the NFS mounted volume... though slightly lower. I suspect that's probably because the data isn't really block aligned all that well but it still saves me a bunch of storage when we have 30 almost identical virtual machines being archived! On the bright side there I have not yet seen utilization get so high on the OI box that it causes any significant issues or dropouts that cause VMware to complain; so far it's been rock solid. I may migrate my ZFS stuff to the OI box and get it off my Ubuntu box... but at the moment they're both working great and I have no complaints.

Re:Good Article

[Thumper_SVX]

Oh yeah, and for bonus points if you have an OCD admin who is allergic to the command line, you can easily put Napp-It on your OpenIndiana server to allow them some visibility into the stats or even create their own filesystems. Simplifies my job as I can give them a zpool to play with and let them build ZFS shares or ZVOLs to their heart's content. Now whether they actually understand the statistics... well that's a different matter but at least you can tell your boss you're giving them the tools they need :)

The big kernel lock is gone

[Anonymous Coward]

The big kernel lock has been gone for a few years.

Not for RAM limited boxen.

[Anonymous Coward]

ZFS likes 2GB of RAM, 1GB is the min.

I'm looking for a solution that runs on 512MB RAM and a 512MB CF disk and a Via c7 CPU.

All the "F*NAS" guys have cut off support for people like us.

And Crossbow, Zones.....

[Anonymous Coward]

Everyone forgets Crossbow and Zones ...... A virtual network stack in a separate, partitioned machine.
Run software, Bind, LDAP, AMP Stack, VirtualBox, all in a separate, zfs snapshot compatible system.
I can roll out a new Zone, with a predefined software stack in 5 minutes, fully scripted, fully partitioned, with external network access, plus load a new VM, and start it.
Shut it down, snap shot, and copy it for a new zone.

Rolling out a new vnic is simple and quick
                  dladm create-vnic -l ${pays_nic} ${new_nic}
add it to a zone
                  zonecfg -z ${zone_name} set iptype=exclusive; add net; set physical=${new_nic}; end
                  zlogin ${zone_name} ipadm create-if ${new_nic}
                  zlogin ${zone_name} ipadm create-addr -T dhcp ${new_nic}/ipv4
Done!

For a VirtualBox just add the ${new_nic} to the machine
Need the vnic mac address to make this work
          dladm show-vnic | grep ${new_nic}
get the mac address
          zlogin ${zone_name} VBoxManage modifyvm ${vm_name} --nic(1-4) bridged --bridgeadapter(1-4) ${new_nic} --macaddress{1-4) ${new_mac} --nictype(1-4) 82543GC --nicpromisc(1-4) allow-all

And then start the Machine backup.

I love the whole Solaris Software stack...... Just can't afford Oracle.
OpenIndiana is nice, but I worry about the long term ZFS issue, version 28 might be the final universal version.

OpenIndiana / SmartOS / Nextena / Illumos

[Anonymous Coward]

Why on earth should I want to switch to Linux if I could use a Solaris-based OS for free?

It's not "just" ZFS and Dtrace, its the whole package of components and how they are designed to work together that linux just can't match in server space.

Linux is great on embedded systems, in the need for some esoteric device drivers or when you'll need some specific software (often touted as "posix compliant") which won't compile under solaris (guess who's fault is that...) . But in the later case there is a backport of brand-z (linux zone) available and KVM got integrated into the latest Illumos kernels too...

So I see people moving away from linux to a free solaris-based OS whose different flavors have each their own strenghts which go far beyond what anything else has to offer...

hammer anyone

[Anonymous Coward]

dragonflybsd has hammer, i use it on my fileserver. It has snapshots, deduplication and online realtime backup as simple as hammer read | ssh host | hammer write sort of.
Best of all, it works with as little as 256mb ram. I even ran it with 64mb although some cleanup won't be done then. I never got why linux wouldn't support that. It could be in kernel instead of fuse. Although fuse is not bad (i like plan9!) it's kinda slow. Hammer is neat. Version 2 is on the way btw. Dragonfly is a very interesting bsd.

What about Dtrace replacements

[INowRegretThesePosts]

Have you tried one of the following replacements?
http://sourceware.org/systemtap/wiki/SystemtapDtraceComparison

I don't like *BSD

[INowRegretThesePosts]

I don't like the *BSD. They are very low on features, and, since
cross-platform software need to be built for the lowest common
denominator, it holds back progress.

Re:I don't like *BSD

[BitZtream]

Name one.

I challenge you to do so because this discussion is about trying to bring 2 things FreeBSD already has ... to Linux, not the other way around.

If you mean it doesnt' have drivers for every POS hardware device out there, sure, FreeBSD devs tend to value their time enough to not buy cheapass obscure hardware.

ZFS and ISCSI under Debian along with KVM/PROXMOX2

[Anonymous Coward]

Here is a link to getting these listed technologies working together
Native ZFS
ISCSI
PROXMOX 2.0
KVM
http://blog.wanfuse.com/?p=22 check out other links at blog.wanfuse.com lots of good stuff!