Oracle To Bring Dtrace To Linux

mvar writes "Dtrace co-author Adam Leventhal writes on his blog about Dtrace for Linux: 'Yesterday (October 4, 2011) Oracle made the surprising announcement that they would be porting some key Solaris features, DTrace and Zones, to Oracle Enterprise Linux. As one of the original authors, the news about DTrace was particularly interesting to me, so I started digging. Even among Oracle employees, there's uncertainty about what was announced. Ed Screven gave us just a couple of bullet points in his keynote; Sergio Leunissen, the product manager for OEL, didn't have further details in his OpenWorld talk beyond it being a beta of limited functionality; and the entire Solaris team seemed completely taken by surprise. Leunissen stated that only the kernel components of DTrace are part of the port. It's unclear whether that means just fbt or includes sdt and the related providers. It sounds certain, though, that it won't pass the DTrace test suite which is the deciding criterion between a DTrace port and some sort of work in progress.'"

Comments

Phasing out Solaris?

[eln]

So, are they porting Solaris functionality to OEL as a precursor to phasing out Solaris entirely? It would suck to see Solaris go from a nostalgia point of view, but it never made much sense to me why one company would continue to develop two Unix-like operating systems.

Re:Phasing out Solaris?

[ILongForDarkness]

Phase out SPARC: probably exactly what they are planning. Add the features that are cool to OEL and then discontinue SPARC and Solaris. If you really really want Solaris you can still got for Open Solaris, otherwise go for OEL that would be my guess on their strategy.

Re:Phasing out Solaris?

[armanox]

Hate to tell you this, but they killed OpenSolaris a long time ago.

Re:Phasing out Solaris?

[the linux geek]

OEL for SPARC has already been announced.

Re:Phasing out Solaris?

[ILongForDarkness]

Oh crap so they did. That is what you get when you don't admin systems for a couple years :-) That stinks. Open Solaris was a bit crippled if I recall a lot of the features that were in the latest releases of Solaris weren't in the open version.

Re:Phasing out Solaris?

[ToasterMonkey]

So, are they porting Solaris functionality to OEL as a precursor to phasing out Solaris entirely? It would suck to see Solaris go from a nostalgia point of view, but it never made much sense to me why one company would continue to develop two Unix-like operating systems.

I think it's more that they want to continue to differentiate OEL from RHEL and provide a direct migration path for RedHat customers to a full Oracle system.

Linux just doesn't make sense in my mind for the space Oracle's software competes in. It's not enterprise friendly. No stable driver ABI. No system interface stability standards. Nothing like Projects, iostat doesn't show tape drives, kernel and userland lack cohesion, to name few of my personal nitpicks, but overall... very little progress. A lot of things you can't change merely by running your own distro. I mean has it really changed the way we work over the last ten years? It feels stagnated while Solaris gets things like Projects, ZFS, dtrace, zones, native fibre channel stack, microstate accounting, init replacement, fault management, and I could go on and on and on. Linux picked up a journaled filesystem at some point in the last ten years /facepalm.

Nobody can convince me Linux should be The One Way forward while it still doesn't have userland utilities for managing SCSI and FC devices. sudo su -c "echo - - - > /sys/foo/wthrufingseriousbro"
That right there is so telling of the level of commitment put towards making Linux a decent server OS. Even fast forwarding to the latest upstream bits (which is cheating because you lose any semblance of interface stability RedHat might provide), I'm not convinced.

If Oracle did port all the cools bits of Solaris over to Linux, the result would have as much in common with Linux as FCoE has with Ethernet. They're not just going to give that to RedHat...

Re:Phasing out Solaris?

[ToasterMonkey]

Hate to tell you this, but they killed OpenSolaris a long time ago.

... as far as aspiring 3rd party Solaris contributors were concerned. For everyone else it was just renamed to Solaris 11 Express.

They probably suddenly ended transparency of Solaris development because they intend to phase it out, port everything to Linux and give it away. /sarcasm

Re:Phasing out Solaris?

[turgid]

Fantastic!!! If they port all the major features of Solaris to OEL, then they can have the same unix for both Sparcservers as well as Opterons/Xeons.

Sorry, I'm having a sense-of-humour failure at the moment so I can't tell whether you're being sarcastic... but what about Solaris x86?

Re:Phasing out Solaris?

[MrNthDegree]

It's a complete clone of Red Hat Enterprise Linux, with an optional Oracle-bastardised version of the kernel.

Re:Phasing out Solaris?

[MrNthDegree]

Sarcasm dude.

Solaris on x86 is a sad joke. The name "slowlaris" came about because of that hideous port.

Ever tried to run Solaris x86 on a normal PC? If you think Linux can't handle basic I/O on slow devices (really, try copying from multiple USB sticks at once and try to use your desktop to the same time!), Solaris lags up with the slightest bit of HDD I/O.

Re:Phasing out Solaris?

[MrNthDegree]

Just some ideas that come to mind and show that actually Solaris isn't as innovative as the Linux camp...

Linux kernel innovations which are more flexible than most alternatives:

SELinux - label-based security which allows flexible controls over the privileges of objects and how they may interact with other labelled objects [Trusted Solaris]
TOMOYO - pathname+history-based security which provides flexible controls over the privileges of processes and how they may interact with objects
AppArmor - pathname based security with flexible controls over the privileges of processes and how they may interact with files/network
cgroups - hierarchical scheduling, resource management customisable entirely by the user within software or through writing to files [Zone Resource Controls]
Kernel Namespaces - flexible separation of networking, PID tables, IPC etc. called simply from within clone() [Jails / Zones]
File Capabilities - labelling files with specific capabilities which would normally only be given to root - removing SETUID from a lot of files
User Mode Linux - port of Linux to run in userland atop a Linux kernel, paravirtualization (Linode built their business upon this)
Completely Fair Queuing (CFQ) - a decent method of prioritising I/O fairly, so that heavy I/O does not grind a desktop/server to a halt
Completely Fair Scheduling (CFS) - a decent method of prioritising CPU fairly, so that heavy CPU load does not grind a desktop/server to a halt
Kernel Mode Setting (KMS) - enables faster switching of terminals, simpler 2D acceleration and better security (Xorg no longer needs root) [Windows GDI]
Full NX Support - helps reduce the risk of successful exploitation of buffer overflows [Windows DEP / OpenBSD W^X]
Full ASLR Support / PIC / PIE - reduces risk of certain classes of exploit by randomising memory location of libraries
NILFS2 - infinite snapshotting log-based filesystem, theoretically capable of tracking every change ever made to files

GNU/Linux userland innovations:

systemd - purely Linux-oriented solution for fast, parallelised initialisation of services
AppArmor - simple way to lock down server/desktop processes interactively through a robust parser
libvirt - abstraction layer to allow many different types of virtualisation to be used in a common way
PolicyKit - secure method of enabling root processes to run without the risk of malicious X clients injecting commands
PackageKit - abstraction layer for a common method of updating/installing/removing packages
PulseAudio - low-latency abstraction layer for handling multiple audio systems, networking audio and multiplexing devices, with per-client volume controls
firewalld - applications can make D-BUS requests to open inbound ports and PolicyKit handles allow/deny, along with any uPnP requests (if allowed)
ABRT - automated bug reporting of most userland applications, sent to the distribution and/or the upstream developers [Windows Error Reporting]
kerneloops - automated reporting of kernel oops errors to detect potential common bugs in the kernel without manual user interference [Windows Error Reporting]
GCC SSP (ProPolice) - prevents buffer overflows in the stack through a canary value [Microsoft Visual Studio (C++) GS]

Re:Phasing out Solaris?

[Fished]

Yeah, Solaris has some pretty awesome features, but at the end of the day all that may be irrelevant in the face of Market Pressures. Sun for many years shot themselves in the foot by failing to deliver useful tools for things like patching/updating, mass installation of Solaris servers (yes, there is jumpstart/wanboot, but it is clearly deficient), and failing to deliver a decent native volume manager (ZFS) until Too Late, and then not having it support root filesystems until Way Too Late.

The reality of Solaris is that there are all these features that look awesome in theory, until you actually have to implement them and discover the practical implications. Take Zones. Zone sounds great, in theory. But, ever tried to patch a server with zones? It's a nightmare. And heaven help you if you actually have a server with zones from multiple, different apps and you need to get outage windows from all the different app groups in order to patch. Or LDoms. Again, they sound awesome. That is, until you realize that there are no tools to manage migrations when a server goes down hard (the most common case for which you would want to do a migration!) So, you end up having to write a bunch of scripts to duplicate LDom xml files etc. to do this, because Sun/Oracle didn't really think through how their technology would be used in a real environment. I also use AIX virtualization technology, and it's much better, and VMWare (which is what we use for Linux servers) blows them both out of the water.

Things like this are why a lot of major companies, including the one I work at, are leaving Solaris as fast as they can. The reality is that it takes twice as many SA's per server on Solaris as it does for any other platform, we have lower virtualization densities, and it therefore costs a lot more money to run. For the kind of money we're talking about, we can deal with a few echoes in the interface for SAN's.

Re:Phasing out Solaris?

[cthulhu11]

Yeah, Solaris has some pretty awesome features, but at the end of the day all that may be irrelevant in the face of Market Pressures

Sad but true. I remember the days when most freeware built and ran out of the box on SunOS/Solaris - now Linux-centrism is rampant, and a growing number of packages are difficult or impossible to build elsewhere.

Sun for many years shot themselves in the foot by failing to deliver useful tools for things like patching/updating

ORLY? Live Upgrade. It has its issues, but so far I've seen nothing close to it in the Linux world. When doing a big yum update, one has to cross fingers and hope everything still works afterwards, as there's no going back.

mass installation of Solaris servers (yes, there is jumpstart/wanboot, but it is clearly deficient)

Other than wanboot only working on (newer) SPARC systems and not x86/x64, what's so awful about jumpstart/wanboot? So far I've seen no amazing capability differences between jumpstart and kickstart. Sure, the latter is convenient in that it can use HTTP instead of NFS, but that's not earthshattering. Kickstart, OTOH, has proven to be a pain in that it doesn't install 32-bit libs on 64-bit systems without hackery or enumeration of every single affected package -- at least, on RHEL 6.1.

and failing to deliver a decent native volume manager (ZFS) until Too Late, and then not having it support root filesystems until Way Too Late.

Sure, they did lag there a bit. ODS/SDS/SVM should have been integrated into the OS before Solaris 8/9 (whenever it was...) rather than being layered, but SVM really isn't too bad. It's not ZFS, but then, nothing is -- and the Linux situation hardly puts it to shame. There we have MD and LVM, each with limitations and with some overlap, but in the end one has to either layer the filesystem on top of LVM on top of MD, which is a real mess, especially since there are conflicting ideas on the RAID partition autodetect bit, or submit to HBA RAID, which sucks in that one can't span HBA's within a single volume, or do 3-way mirrors. I counter that Linux systems *still* don't have a decent volume manager.

The reality of Solaris is that there are all these features that look awesome in theory, until you actually have to implement them and discover the practical implications. Take Zones. Zone sounds great, in theory. But, ever tried to patch a server with zones? It's a nightmare

Agreed that Zones are a mess.

The reality is that it takes twice as many SA's per server on Solaris as it does for any other platform, we have lower virtualization densities, and it therefore costs a lot more money to run.

Are you a web hosting provider? That's the only scenario that I can see where virtualization is truly crucial and not just the buzzword-du-jour, like thin clients were a while back. As for admins/server, my experience has been quite the opposite. Solaris by and large just works. One doesn't have to combat madness like system uid's being assigned on the fly when a package gets installed, or random inexplicable bouts of SSH sessions slowing to a crawl. When I encounter a Solaris problem, there's a wealth of relevant information out on the web to be found. My experience with Linux is that most of what's out there regards A) Old versions that are no longer relevant B) Lame home-user systems and/or C) Balkanized distributions other than the one that one's running, which do things completely differently In 2011, shiny new ext4 can only go to 16TB in size, which is a freaking JOKE.

Re:Phasing out Solaris?

[smash]

AS per most of the problems listed: too little, too late. The boat for solaris sailed long ago. I'm keen to see whether or not Opensolaris goes anywhere, but unfortunately the free unix market is probably fairly well covered.

Re:Phasing out Solaris?

[Fished]

ORLY? Live Upgrade. It has its issues, but so far I've seen nothing close to it in the Linux world. When doing a big yum update, one has to cross fingers and hope everything still works afterwards, as there's no going back.

Running LiveUpgrade in a large-scale production environment is kind of like baby-sitting someone else's four-year-old. When everything's going well you say to yourself, "wow, this is pretty nice! Maybe I'll even have one of these of my own someday!" Then the four-year-old has a meltdown, and you try to call his parents and they don't answer the phone and you're left holding the bag.

Same deal with LiveUpgrade. When it works, great. When it doesn't work, heaven help you, because Oracle can't/won't fix it or give you any reasonable support in any reasonable period of time. Instead, they'll tell you to apply the latest LU patch (to your production ABE, mind you, which was supposed to be the very thing you were avoiding with LU.) This will very likely have prerequisite patches (e.g. the cpio patch, which requires the kernel patch) which will require a reboot of the server, causing the very outage you were trying to avoid in the first place. But what can you do? And then, when you get done applying the patch, guess what?

The problem isn't fixed

So Oracle bumps you up to the LU engineering team, who get back to you two-weeks later, who say, "Oh yeah, we knew about that bug, it's scheduled to be fixed in version 'blah blah' to be released in 2015." In disgust, you w4rite your own version of LU using ufsdump and/or ZFS features. Oddly enough, your two days worth of scripting/hacking seem to work better than the supported Oracle product.

Other than wanboot only working on (newer) SPARC systems and not x86/x64, what's so awful about jumpstart/wanboot? So far I've seen no amazing capability differences between jumpstart and kickstart. Sure, the latter is convenient in that it can use HTTP instead of NFS, but that's not earthshattering. Kickstart, OTOH, has proven to be a pain in that it doesn't install 32-bit libs on 64-bit systems without hackery or enumeration of every single affected package -- at least, on RHEL 6.1.

Really? Really? I'm not much familiar with Kickstart (different guy does that) but... the amount of coding and hacking required to make wanboot/jumpstart work for a production environment with a lot of different server types is just ridiculous! Then there's having to maintain different FLARs for ZFS and UFS builds, having to maintain both Jumpstart and Wanboot (Jumpstart to build the FLARs and Wanboot to install the boxes with the FLARs once you've built them), the awkwardness of adding packages to the build after the FLAR is laid down... I could go on, but I won't.

I counter that Linux systems *still* don't have a decent volume manager.

For the price, Solaris really needs to compete with AIX/HP-UX, which both have had a decent volume manager out of the box for a long time. Instead, on Solaris you almost have to buy Veritas.

Are you a web hosting provider? That's the only scenario that I can see where virtualization is truly crucial and not just the buzzword-du-jour, like thin clients were a while back. As for admins/server, my experience has been quite the opposite. Solaris by and large just works.

Virtualization isn't just a buzzword. It's a way to reduce costs, and the company that I work for is pushing for it hard. It looks really bad when the AIX guys get 10:1, the HP guys get 10:1, the VMWare guys get at least 20:1 (depending on loads with ESX, which is an awesome product), and we're doing 4:1 because of all the implementation detail issues with zones/ldoms and the lack of good supporting tools (we're having to write our own in many cases.)

And, yes, Solaris by-and-large just works. Until you have to patch. Or deal with zones. Or deal

Re:Phasing out Solaris?

[cthulhu11]

Running LiveUpgrade in a large-scale production environment is kind of like baby-sitting someone else's four-year-old.

Perhaps. I did say that it has its issues, eg. the expansion of sparse files that I submitted a bug for. I ran an LU on a specific system that had .dir/.pag files, and was startled when the new /var filled. Sun eventually put out an LU patch to address that with a new cpio, and then months later finally followed with x86.

When it doesn't work, heaven help you, because Oracle can't/won't fix it or give you any reasonable support in any reasonable period of time.

That's been an issue with Sun/Oracle for years, but it's been no less an issue with any other vendor, including Red Hat. And don't get me started on HP's tragic hardware support...

Instead, they'll tell you to apply the latest LU patch (to your production ABE, mind you, which was supposed to be the very thing you were avoiding with LU.)

I've admittedly long found that amusing, but it's also been the case that most of the patches have been against early revisions of Sol10, ones that we never deployed, and it pretty much works without additional patches first.

I'm not much familiar with Kickstart (different guy does that) but... the amount of coding and hacking required to make wanboot/jumpstart work for a production environment with a lot of different server types is just ridiculous!

Again, for me this hasn't been any different in Linux land, and I wouldn't call either ridiculous. We wrote a wrapper script to set up hosts for Jumpstart, and we wrote one for Kickstart. We wrote a finish script for Jumpstart to install local packages, set up SSH keys, etc., and we did the same for Kickstart as %post. Getting Jumpstart set up in the first place took a lot of digging, hampered by conflicting information from Sun re how to structure their screwy DHCP, but setting up Kickstart was no less of a hassle. Nowhere was the entire process documented as a whole. I had to piece it together from other people's /tftpboot trees and scattered web pages. Then, after I had it more or less figured out, I came across a page written by someone who'd had an identically frustrating experience: http://www.smtps.net/pxe-kickstart.html So once I figured out the pxelinux.cfg foo for the installation menu, I found that the menu.c32 foo for stuffing options to the kickstart process is a complete mess. On Sun hardware, it works, with the selected menu item highlighted and everything. On other hardware, it shows nothing, so one has to count down arrow presses and hope that the correct entry is selected. Oh, but one can stick in an obliquely-documented "serial 9600" line. Okay. That breaks on some hardware, and on others the display is a complete mess. This has been soooo much easier than Jumpstart!.

Then there's having to maintain different FLARs for ZFS and UFS builds

Who said anything about FLAR? I've never seen a use for it.

For the price, Solaris really needs to compete with AIX/HP-UX, which both have had a decent volume manager out of the box for a long time

My knowledge of those dates to 1988 and 1992, respectively, but at the time both had their own bizarreness. AIX I'll always remember as requiring navigation through seven menus to add each pty to the system, and for silently commenting lines out of config files.

Instead, on Solaris you almost have to buy Veritas.

Are we back in 1998? ODS in the Solaris 2.6 era sucked and VxVM had its appeal, but once they integrated it with the OS and made it more robust, it became tolerable. ZFS of course once it dropped is friggin awesome.

Virtualization isn't just a buzzword. It's a way to reduce costs

In what s

Bring ZFS to linux!

[Anonymous Coward]

Hey Oracle,

Cool now bring ZFS to linux!

Re:Bring ZFS to linux!

[ZorinLynx]

I doubt they would. ZFS is one of the last remaining reasons to run Solaris, pay for Solaris maint, etc... If they port ZFS to Linux they will lose quite a bit of that revenue stream.

We all want it; ZFS is a beautiful thing, we run it at our site and practically worship its awesomeness....but it probably won't happen.

Re:Bring ZFS to linux!

[larry bagina]

oracle is working on btrfs which has ifs features without replacing the entire fs/lvm/raid stack.

Re:Bring ZFS to linux!

[hedwards]

Well, whether or not that has any bearing on ZFS, unless they change the license on ZFS, they wouldn't have the ability to do it anyways. Or at least not without violating the terms of at least one of the involved licenses.

Re:Bring ZFS to linux!

[mysidia]

But replacing the entire fs/lvm/raid stack is a good thing!

This is quite true... because frankly, the entire fs/lvm/raid stack sucks with big disks.

ZFS solution is a really good answer to a lot of problems -- scalability, manageability, reliability.

It would be a very good thing if Oracle would execute a port of ZFS to Linux (under the GPL, of course), and while they are at it... port AVS and Open HA/Cluster, as a superior alternative to DRBD, port the SMF as a replacement for init, the fault manager, configuration management CLI tools, and other similar tools that could really fill in some serious gaps in Linux.

Re:Bring ZFS to linux!

[hjf]

Developing something like ZFS is pretty simple. Just get some of the world's best programmers, have them work in a project, with good management and leadership, and test and fix bugs. And tell them they're free to do it ANY way they want, there's no "right" or "wrong". There's no obligation in using md, LVM and an FS on top. In fact, md and LVM were created to overcome the filesystem's limitations. If you're creating a fs from the ground up, you might as well skip md and LVM. But some people in mailing lists think rules have to be followed strictly, and you're an idiot if you don't "layer" properly because it's Written In The Book.

Too much layering is not good either... and, you know what? The 7-layer OSI model is just a model. TCP/IP doesn't have 7 layer: it "only" has 4.

Which is why Linux moves so slowly. Features take ages to develop, unless Red Hat or someone else throws in some code and makes stuff work. Otherwise, it's just the brainchild of someone that does it in his free time... and has to take shit from any random idiot.

Re:Bring ZFS to linux!

[hjf]

And why do you want to keep the raid and LVM stack?

If you're creating a filesystem and you can make it aware of its own backing storage (and adjust stuff like block size - cause you know, there are disks with 4K sectors now), and have it manage caching by itself (and thus, be aware of how much memory the has, and how much of it is actually RAM and not virtual), and have it check for redundancy and do online checks and repairs - which you realize it's just awesome if you ever try to do fsck on a 20TB filesystem (and because it knows how much data it's actually used, have it only check what's used, instead of blindly regenerating blank space for an array of disks). And variable strip size, and thin provisioning, CoW and free snapshots and clones, and a lot of other stuff ZFS does because it doesn't need to "respect its elders" LVM and md.

Re:Bring ZFS to linux!

[hjf]

Agreed, and it reminds me of the #1 argument against ZFS: BUT BUT BUT IT BREAKS LAYERS, OH NOES!

And the second one: ZFS doesn't have fsck!! What's one going to do without a proper fsck? ZFS IS A JOKE!

Once you get familiar with ZFS you realize how much sense it makes, even if it violates "rules". For example, someone complained about the fact that zfs does LOTS of checksums, wasting CPU cycles; and doesn't have fsck. Well, it doesn't have fsck BECAUSE it does lots of checksums. Do you have time to wait for that 20TB arrayto finish fscking? I don't (is 20TB even "large" nowadays?) . And my CPU is powerful enough to cope with the "high demand" (sheesh!) of ZFS checksumming.

ZFS is a *reliable* filesystem. Not a lightweight, or "high performance" one.

Luckily with bigger drives people will start realizing that "life's too short" to wait for fsck to finish.

Re:Bring ZFS to linux!

[Luckster7]

It is on Linux: http://zfsonlinux.org/

I've been using Nexenta to get ZFS for about 5 years, however earlier this year I switched my new installs to Linux. ZFS on Linux has been working great for me and my clients. This is not to be confused with fuse ZFS.

Re:Bring ZFS to linux!

[Electricity Likes Me]

ZFS basically does everything I would want in a server file system (hell: any filesystem) and it's a crime that it hasn't been ported to Linux (and that OpenSolaris is sufficiently dead now that you can't run it on newer hardware at all).

We live in an age of 4-core CPUs being commodity items - ANY system can spare the cycles to ensure it's own data integrity since what else are you going to do with them? (alternately: if you need them, then why does that particular machine deal with having a hard disk anyway?)

Re:Bring ZFS to linux!

[icebraining]

(..) it's a crime that it hasn't been ported to Linux

http://zfsonlinux.org/

Re:Bring ZFS to linux!

[Electricity Likes Me]

The POSIX layer isn't stable yet.

ZFS really comes into it's own when you can mount all your home directories as filesystems and use cheap snapshots for upto the minute backups for your user accounts.

That said, I will take another run at it sometime soon I suspect.

Re:Bring ZFS to linux!

[TheRaven64]

The worst thing about the first complaint is that ZFS actually does have very clean layering. At the bottom you have the storage pool allocator at the bottom is basically malloc() for persistent storage (equivalent to the block device layer, but with a more convenient interface), the data management unit sits on top of this and provides transactional I/O to the underlying storage, and the ZFS POSIX layer sits on top of this and provides POSIX filesystem semantics. You could replace the ZPL with something that provides things that look like raw block devices (ZVOLS - used for iSCSI shares, among other things), and you could also fairly easily add something that provided an SQL interface to storage, since the transactional support is all done a layer lower down the stack.

The second complaint is really a matter of semantics. Scrubbing a ZFS partition provides a superset of the functionality of fsck. Not only does it validate and repair the metadata, it also validates and repairs the data on the disks. Most ZFS users don't need to run it, because all it really does is try to read every file on the disk - the underlying filesystem code performs the repairs automatically any time some read data fails the checksums.

I think the real problem with ZFS is the name. Calling it a filesystem makes people mentally class it as something like UFS or ext2fs, when it's actually a complete storage stack.

Re:Bring ZFS to linux!

[TheRaven64]

OpenSolaris may be dead, but FreeBSD now ships with version 28 of ZFS, which includes nice things like deduplication. iXSystems, which sells storage appliances based on FreeBSD, is funding ongoing development work, so ZFS on FreeBSD is going to stay actively maintained irrespective of whether Oracle makes future versions of the code available. FreeBSD also got the ability to boot from ZFS before Solaris and integrates ZFS very nicely into the existing storage stack (it works as a GEOM consumer and provider, so you can easily do things like put a FAT partition in a ZVOL and have transactional I/O and deduplication on the filesystem used by a Windows 95 virtual machine).

Re:Bring ZFS to linux!

[WorBlux]

ZSS is more fun to say out loud as well.

Re:Bring ZFS to linux!

[Anonymous Coward]

ZFS has two major issues missing. You can't scrub a file as in "I have a legal obligation to overwrite the contents of this file", you just can't ever do it on ZFS. The other problem with ZFS is that there is no way to tell it "This file is magic for the boot process, please put it in the first N physical sectors on the physical disk Y"

Re:Bring ZFS to linux!

[makomk]

For example, someone complained about the fact that zfs does LOTS of checksums, wasting CPU cycles; and doesn't have fsck. Well, it doesn't have fsck BECAUSE it does lots of checksums. Do you have time to wait for that 20TB arrayto finish fscking?

Which is a nice idea until something (for example a ZFS bug or a hardware glitch) causes part of the ZFS metadata to become corrupted, and the only way to get your 20 TB array working again is to dump all of the data somewhere else, recreate the array from scratch, and reload all the data.

Re:Bring ZFS to linux!

[Chris Mattern]

I WANT to replace the entire fs/lvm/raid stack! It sucks! It's complicated, limits functionality and is a nightmare to maintain. ZFS is simple, flexible and powerful.

Re:Bring ZFS to linux!

[GrumpyOldMan]

I'm running with this version, and it is working great for me on my first test machine (8T raid-z with an L2-ARC)

Re:Bring ZFS to linux!

[mysidia]

For example, someone complained about the fact that zfs does LOTS of checksums

Yes it does.

However, thanks to a new enhancement in the Xeon 5500 CPUs, called SSE4.2 instruction set, there is actually a CPU instruction for CRC32 accumulation.

Oh wow... we're going to use a couple extra CPU clock cycles per read and write to protect our actual data integrity, to allow us to do "online scrubs" of the filesystem to check for any surface errors instead of some limited arcane "filesystem metadata level consistency" called fsck that requires downtime, and FSCK does not check for or repair data integrity; because without checksums our ext3/ext4 filesystems don't know anything about data bit errors, latent surface errors, bit rot, or "silent data corruption" as it's called.

The checksums using a couple clock cycles per I/O... must be a "lot" slower than having many hours of downtime to check our filesystem, if god forbid our system should go down uncleanly, or we should reboot and the "maximum mount" count or "maximum time" since the last fsck happens to have been found to be exceeded while we are trying to boot back up. feh:)

Re:Bring ZFS to linux!

[mysidia]

ZFS has two major issues missing. You can't scrub a file as in "I have a legal obligation to overwrite the contents of this file"

You're right that there are some extremely unusual requirements that ZFS won't meet. You also can't use SSDs with wear levering or any decently modern hard drive, because modern hard drives have block relocation functions; sometimes a sector is taken out of service and replaced with a "spare block"; if it is found to be failing or predicted to fail. The result is that on modern hard drives you can never be sure you "meet your legal obligation".

The only real reliable way to ensure you destroy the data is to encrypt it from the beginning; dedicate a storage medium to the encryption key, and when your obligation comes to eliminate the data, physically destroy the medium you placed the key on.

The other problem with ZFS is that there is no way to tell it "This file is magic for the boot process, please put it in the first N physical sectors on the physical disk Y"

Put a file on the first N physical sectors. Is a rampant layering violation even bigger than the claimed layering violation of ZFS. If for some reason you actually needed to do that, it would mean your system is broken.. Reasonably modern systems have no hard requirements on what physical portion of the disk the OS resides on.

I would say use a separate pair of very small hardware RAID mirrored bootdisks (or USB stick) and put a 100 megabyte UFS file system on the boot volume for booting in that case.

Re:Bring ZFS to linux!

[mysidia]

Which is a nice idea until something (for example a ZFS bug or a hardware glitch) causes part of the ZFS metadata to become corrupted

The ZFS pool root metadata is protected by having 3 copies of it on the file system, and metadata blocks are checksumed just as the rest of the data blocks.

"dump all of the data somewhere else, recreate the array from scratch, and reload all the data" is really something you should never have to do with ZFS, unless you actually have a storage device failure, and you don't have a redundant device that a clean copy of the data can be read from (e.g. Mirror or RAIDZ).

Re:Bring ZFS to linux!

[hjf]

Yes I forgot about new CPU instructions for checksumming and stuff, but even without them, CPUs today are SO powerful that I doubt you can peg all CPU cores. And if you're running a 20TB+ array I doubt you'll be running it in the same physical box as the DB server. You probably have a DB server with a ZFS backing store, connected by 10gbe or infiniband, or some other means.

If you have a single box, probably even multiple multi-core CPUs. So hitting a 100% CPU usage would be unusual (I mean, you're supposed to size your server to make 100% cpu usage unusual)

Re:Bring ZFS to linux!

[mysidia]

Yes I forgot about new CPU instructions for checksumming and stuff, but even without them, CPUs today are SO powerful that I doubt you can peg all CPU cores.

Well... I can peg all CPU cores easily, if i'm not too careful, and put too many VMs on a compute host..

But part of the deal is storage hosts should not be compute hosts

ZFS is the filesystem you can use to provide storage services to other hosts.

This allows every bit of CPU to be used to serve the targets, and to provide the LZJB compression and dedup.

This allows the expensive mirrored SSD ZIL log devices and SSD read cache devices on one or two ZFS servers to benefit every application.

This allows every bit of those 64gbs of RAM to be used efficiently for filesystem caching.

This allows the total sum of all those TBs of storage on attached JBODs to be available for alloction to any server that requires storage.

Re:Bring ZFS to linux!

[hjf]

That's exactly what I'm talking about.

Some folks get confused because ZFS runs on Solaris (or any other compatible OS), so people think it's a filesystem. So yes, it is, and it can perfectly run as a filesystem for a SMALL database and appserver on the same host. Small as in "test server".

But ZFS is more suited to dedicated storage cabinets rather than a local FS. You couldn't run Oracle on your Netapp storage server, this is the same. One box (or more) for storage, another box (or more) for db, app, etc.

Also I disagree with people who think the cpu should be "mostly idle". Some guys like to size their servers and freak out when the CPU hits 100% usage even during peak hours... that's wrong. CPU should be busy as much as possible (duh, that's why we virtualize nowadays), and if it hits 100% so be it. MOST apps (not all, of course) are not time-critical and can be delayed for some time. Not a "minutes" delay, but if your typical db query takes 100mS. having to wait 200-300mS won't hurt the sales guy hitting the DB server. Sure, he will get impatient but that's his problem.

Re:Bring ZFS to linux!

[badkarmadayaccount]

Until we realize that some network topologies and setups, not to mention connection requirements, well, require a session layer - say, SIP. If you haven't noticed, the IT world has, more or less standardized on a presentation layer protocol - HTTP. Oh, and the application layer is just the semantics the app assigns to the data received by HTTP. So, in short, those who do not know their history, are doomed to repeat it. That aside, I believe that ZFS has layering, just not identical to the classical Unix implementation, which is OK.

FreeBSD?

[liquidhokie]

If you want Dtrace and ZFS, just go with FreeBSD. You get pf and jails thrown in for the effort.

Re:FreeBSD?

[afabbro]

So why is it possible for FreeBSD to have dtrace/zfs and not Linux? I ask out of ignorance...

(I am expecting the answer to be legal rather than technical/no one's gotten around to it)

Re:FreeBSD?

[airencracken]

Because the CDDL (which dtrace and zfs are licensed under) is incompatible with the GPL, but not the BSD license.

Re:FreeBSD?

[hjf]

(I am expecting the answer to be legal rather than technical/no one's gotten around to it)

Stupid licensing issues.
There's always a way around technical stuff: http://zfsonlinux.org/

Re:FreeBSD?

[TheRaven64]

Linux is released under version 2 of the GNU General Public License. This imposes a few restrictions and says that the code may not be distributed linked to any code that imposes more restrictions, nor can any derived works impose any more restrictions than are present in the license.

FreeBSD is released under the 2-clause BSD license, which says, basically, do what you want with this, just don't sue me if it doesn't work and don't claim you wrote it.

OpenSolaris was released under the CDDL, which is generally less restrictive than the GPL (no restrictions on what you can link it to), but adds some anti-patent clauses that are not present in the GPL. Because these restrictions are not present in the GPL, the GPL prevents CDDL code from being linked against it. This means that if ZFS or DTrace were ever ported to Linux by anyone other than the copyright holder they would not be allowed to distribute Linux along with their port.

In FreeBSD, ZFS and DTrace are optional kernel modules, so you can still build a system without them, but they are loadable if you are happy to accept the terms of the CDDL when you distribute FreeBSD. There's no technical reason why either couldn't be ported to any system (well, the Linux storage stack is a mess, so adding ZFS would be a bit harder, but it could be done), but few people are motivated to produce a port when they are not legally allowed to redistribute it.

Re:FreeBSD?

[bill_mcgonigle]

if ZFS or DTrace were ever ported to Linux

This is done (largely). Sun was hired to do the block layer port. The POSIX implementation is independent.

Re:FreeBSD?

[smash]

End user perspective: the GPL, and many of the zealots involved are unwilling to support a port of it. Also, NIH syndrome, which is endemic in linux.

Re:FreeBSD?

[PipsqueakOnAP133]

Sun was hired to do the block layer port.

Really? (seriously curious) Got any more info on that?

Re:FreeBSD?

[bill_mcgonigle]

This mentions the LLNL holding company doing the port:

http://en.wikipedia.org/wiki/ZFS#Native_ZFS_on_Linux

as does the disclaimer on zfsonlinux.org, but I've read before Sun was hired to do the actual work. The purpose was to host Lustre on ZFS, and Sun owned both properties at the time. I can't find a specific citation in the 5 minutes I gave it.

If you happen to see one, please edit Wikipedia.

Re:FreeBSD?

[PipsqueakOnAP133]

Hmm, I was aware of the LLNL port, but didn't know about KQ.
http://www.phoronix.com/scan.php?page=article&item=zfs_linux_coming&num=1

While digging deeper, it seems KQ has been bought out and their work stopped:
http://punetech.com/solid-state-storage-company-stec-acquires-punes-kq-infotech/
http://www.osnews.com/comments/24853?view=threaded&sort=&threshold=0

Okay, so the closest thing I can find is this:
http://www.olcf.ornl.gov/wp-content/events/lug2011/4-13-2011/1130-1200_Brian_Benhlendorf_LUG11_ZFS_on_Linux_for_Lustre.pdf

It actually sounds to me more that Sun worked on porting Lustre to integrate ZFS as a backend, not that Sun worked on porting ZFS to Linux?

Re:FreeBSD?

[bill_mcgonigle]

Thanks - I hadn't seen Brian's slide deck before; there's no more authoritative source.

Great technology story!

[bdgregg]

This is a great technology story - even if only for one version of Linux so far. DTrace will bring tremendous value for troubleshooting and performance analysis, and is a technology I use (almost) every day.

For example, yesterday I had a CPU bound workload with an unexpected level of variation, and used DTrace to measure the effect of CPU thread affinity and interrupt activity on that workload. I used DTrace to pull the runtime along with other details: number of scheduling events for that thread, along with the CPUs that the thread ran on; also, for preemption, the pre-emptor thread (to see why) along with both its user-level and kernel stack traces; also the interrupt thread and device. I fairly quickly showed that the runtime variation was caused by network interface interrupts from an entirely different application. This analysis would take quite a lot longer without DTrace, and may be prohibitively difficult to complete.

Many of my uses of DTrace are much more straightforward than that; including identifying file system latency for applications, application response time, and CPU dispatcher queue latency. I've listed many more examples in the DTrace book (http://www.dtracebook.com). It should be a great resource of ideas for those looking to use DTrace on Linux - since the hardest part for people has been knowing where to start, given the ability to see everything.

Re:Great technology story!

[drinkypoo]

The main question is, is it really feasible? And the other question is, what would dtrace give us that we don't already have with systemtap, besides compatibility with slowlaris? The first question tells us if we should pay attention and the second one tells us if we should care.

Re:Too Late...WE DONT NEED IT...we got SystemTap

[_merlin]

You obviously haven't had to use both in anger. SystemTap is another "me too" project like so many things on Linux, where the only people saying it's as good are the people who haven't used the product it's an imitation of. Oh, and then there's the RMS type who will say it's "better because freedom has value" or something to that effect. Doesn't help you when you're actually trying to tune an application for performance.

Another Linux Today link...

[SwedishChef]

This story appeared yesterday on Linux Today. And it's not even close to the first time this has happened. If we can read about this first on Linux Today then what's the point of coming to Slashdot? Especially 24 hours late.

Re:Another Linux Today link...

[Gavagai80]

Shame on you for clicking TFA link -- clearly you don't belong on slashdot anyway.

Re:Another Linux Today link...

[isopropanol]

Come to slashdot for discussion, not scoops.

How does it compare

[jader3rd]

How does it compare against ETW?

Re:Licensing problems (CDDL is toxic)

[Fri13]

GPL is not toxic.

Pure freedom is even worse situation than pure chaos. You need limitations and rules what can not be done and what should be done.

I'd rather they ported ZFS instead

[melted]

Btrfs seems to have been in development forever, and the developers on the one hand say that it's mostly stable, but on the other there are still some pretty scare bugs. It doesn't make a terrible amount of sense for Oracle to develop two next-gen CoW filesystems.

Re:I'd rather they ported ZFS instead

[impaledsunset]

Porting ZFS would take more effort than completing Btrfs, while Dtrace provides functionality that doesn't exist at all. I wouldn't mind a ZFS port, since after using both, I kinda prefer it to Btrfs, but it's better to first port what's new and doesn't exist at all, and then go to porting things that are already there.

next: the lawsuit

[t2t10]

After bringing DTrace to Linux, they are then likely going to turn around and sue people somehow; kind of like they did with Java.

Don't use anything from Oracle; they are worse than Microsoft.

Re:next: the lawsuit

[WorBlux]

Google ran into problems because they didn't want to touch the GPL'ed sun code. If google had just used that code, modified it for their own use and re-named it they would have been protected by the implicit patent license in the GPL code and wouldn't have been sued. It was Andy Rubin's fear of the GPL that put Google in the position of being sued over Java.

I suspect Oracle is trying for a cash grab

[msobkow]

I suspect Oracle is trying for another cash grab. Port the parts of DTrace that have to be in the kernel and open source them, then sell an add-on package (perhaps only for their Linux) with the rest of the functionality. Let's face it -- Oracle is much more focused and effective at monetizing technology than Sun ever was.

Re:I suspect Oracle is trying for a cash grab

[WorBlux]

I don't think they'll convince anyone at the Linux foundation to maintain their hooks in the kernel, So I only see it as an extra feature you get should you use their long-term kernel release and buy their support.

Re:I suspect Oracle is trying for a cash grab

[drolli]

I guess Oracle wants to sell complete Boxes with Hardware to Database included. If the customer asks why the box does not perform ask predicted, with the total price, there may be a reliable answer required. So if the customer wants to run the box on linux, the mechanisms to answer these questions must be available in linux.

Re:Too Late...WE DONT NEED IT...we got SystemTap

[The Man]

You know what's even more annoying than Linux's "me too" projects? All the stuff they COULD imitate but don't. I have no idea why Linux admins still have to grovel through logs or use stuff like splunk to guess at what's wrong with their hardware, but they do. Even a lousy knockoff is better than pretending the problem doesn't exist and leaving people to cobble together inferior workarounds.

Typical Stupid PHB Decision

[turgid]

Yesterday (October 4, 2011) Oracle made the surprising announcement that they would be porting some key Solaris features, DTrace and Zones, to Oracle Enterprise Linux. As one of the original authors, the news about DTrace was particularly interesting to me, so I started digging. Even among Oracle employees, there's uncertainty about what was announced.

This sounds like a typical PHB decision: make a crazy choice without consulting the engineers as to whether it's a good idea, possible or even wanted, and could potentially threaten the existence of their existing products that have had blood sweated over them for 20 years, makes their continuing relevance to the company (and therefore employment prospects) seem very uncertain and replaces a technically-superior product with a less-able competitor.

Disclaimer: I don't work for Oracle. I know some Solaris people, and although Linux is great (it's what I do), Solaris still beats it in terms of things like high-end scalability. As we move to a massively multi-core, multi-cpu world, the Solaris kernel has a lot of advantages.

Maybe the PHBs don't care any more. Maybe they'd rather spend the effort on Linux. Who knows. It sounds like a bad time to work in the Solaris group at Oracle.

So, again, "good" is being replaced by "good enough." That's the commoditisation of technology. The invisible hand has spoken. That's life.

Re:Licensing problems (CDDL is toxic)

[WorBlux]

It is toxic to an extent. It can in some cases prevent a bit of cooperation that would aide just about everyone involved were it possible. However on the other hand MIT and BSD could be classified as weak in not giving patent licences, and not being copyleft. Every license has it's own costs and advantages. MIT and BSD can be very advantageous if the rate of development can outpace closed efforts or your project has universal interoperability as a goal. Don't get me wrong I think the GPL is a great legal hack, but it isn't the one license to rule them all.

Re:Too Late...WE DONT NEED IT...we got SystemTap

[bill_mcgonigle]

Oh, and then there's the RMS type who will say it's "better because freedom has value" or something to that effect. Doesn't help you when you're actually trying to tune an application for performance.

Yeah, it only gets you the very operating system you're trying to tun your application on.

Re:Good news.

[Morth]

As long as the leave strace in place. Apple replaced ktrace with dtrace and I've been hating it ever since.
It's not that dtrace is bad, it's just that they have different purposes, and dtruss has several problems ktrace/strace does not:

• It's asynchronous. Meaning it won't output write(1, "foo\n", 4); next to the actual output of foo.
• Sometimes stuff gets out of order, not sure why.
• It'll only output pointer addresses, e.g. bind(3, 0x12345678, ...) instead of bind(3, {127.0.0.1, 8080}, ...)

Ok, rant over.

Re:Good news.

[Morth]

Damn, not sure how I forgot the worst part of all. dtruss requires root to be run, so you usually have to do some convoluted double sudo to run a program the same way as without it.

Re:Dtrace for Linux already exists

[GrumpyOldMan]

I've used this ... it has been very helpul for development, but I would not run it on a production machine. In most cases when I've run it, it has managed to eventually crash the machine, usually just after obtaining the information I was after. For driver development, that is fine. But not for production.

s/Linux/Oracle Enterprise Linux/

[gtirloni]

I wouldn't expect this to see the light of kernel mainline ever, or at least not until Oracle stops selling their Enterprise Linux offering.

DTrace on Linux will probably be something like Ksplice where it's available only to paying customer (last I checked, correct me if I'm wrong).

Good thing this opens the doors within Oracle to consider migrating more of the Solaris features to Linux, even if it's only for OEL for the time being. Personally, while being a Solaris sysadmin, I'm not wasting my time on Solaris anymore and I certainly won't be bothering with Solaris 11 unless my employer shows the need. So far our next hardware refresh cycle is up for in 6 months and nobody bothered to ask Oracle for a quote. The word is that it's going to be Linux for everything.

Available since 2008

[sgt scrub]

It has been available for Linux since 2008. 02-Aug-2008 Work in progress port of Sun's DTrace system for Linux. It is actively maintained. http://www.crisp.demon.co.uk/tools.html I don't see anything new to the table outside of keyboard, mouse, and framebuffer recording. I'm not sure a lot of Linux users would find that an attractive addition.

Built-in instruments can track

        User events, such as keyboard keys pressed and mouse moves and clicks with exact time.
        CPU activity of processes and threads.
        Memory allocation and release, garbage collection and memory leaks.
        File reads, writes, locks.
        Network activity and traffic.
        Graphics and inner workings of OpenGL.

Did anyone else read "Oracle To Bring Disgrace..."

[dickens]

Oracle To Bring Disgrace to Linux ? Just saying, is all.

Re:Wanna fix dtrace?

[ahl0003]

Start with the license.

as predicted: http://twitter.com/#!/ahl/status/121257501193809920

Oddly good news here

[swordgeek]

Let's be clear here. The Sun division of Oracle is being run by Mark Hurd, who was last seen gutting HP and screwing his staff member. Oracle will kill off all things Sun, either now or later. Solaris and Java are the only things they seem to care about, and both of those are still rather endangered.

Solaris still has some great advantages over Linux--enough to actually keep a handful of people on it despite Oracle. I assume that they're going to get those necessary features into Linux, and then dump Solaris entirely.

We're spending about $12 million to dump all of our Sun applications, as well as most of our Sun gear and Solaris installs in favour of x86 gear (mostly IBM) and Linux. The scary thing is that $12 million is less than the increase in licensing and maintenance costs from Oracle, vs. what we were paying from Sun.

Bottom line: Oracle doesn't want people running Solaris. The more features we get into Linux before Larry gives up and says "screw all y'all" the better.

Re:Good news.

[ElusiveJoe]

What I wonder is how exactly Oracle will turn this into an immensely dickish move

The license, man. The license.

Re:So, again, good is being replaced by "good enou

[turgid]

I believe that Solaris x86 on x86-64 scales better on large systems than Linux. Large x86 CPU manufacturers like Solaris x86 because it shows off their hardware better than Linux.

Re:Point ?

[smash]

Systemtap is crap, and doesn't have the functionality of DTrace?