Slashdot Mirror
Ryan Gordon Ends FatELF Universal Binary Effort
recoiledsnake writes "A few years after the Con Kolivas fiasco, the FatELF project to implement the 'universal binaries' feature for Linux that allows a single binary file to run on multiple hardware platforms has been grounded. Ryan C. Gordon, who has ported a number of popular games and game servers to Linux, has this to say: 'It looks like the Linux kernel maintainers are frowning on the FatELF patches. Some got the idea and disagreed, some didn't seem to hear what I was saying, and some showed up just to be rude.' The launch of the project was recently discussed here. The FatELF project page and FAQ are still up."
He needs thicker skin if he's going to deal with the LKML crowd. I wouldn't give up just because it's not merged into the official tree.
Cory Doctorow talking about cloud computing makes as much sense as George W Bush talking about electrical engineering.
I like my elves the way I like my tea: thin and exotic.. served while still hot.
and some showed up just to be rude.
I would never have believed that people in the Linux community would show up at an event just to be rude. I've always heard such glowing praise about the Linux community. They're always there to help the new guy, willing to mentor those learning the "So simple a caveman can do it" operating system and break the monopoly of Microsoft once and for all.
His comments can't be correct. Everyone knows what fine, upstanding individuals the Linux community is.~
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
Fat elves
The Navy Motto "IF it ain't broke Fix It" "A day is wasted if you don't learn something new"
The 32-bit vs. 64-bit split is handled pretty well on Linux (well, Debian drug its heels a bit on multiarch handling in packages, but even they seem to be getting with the programme).
Real multi-arch could be useful, but the number of arches on Linux is just too overwhelming. To get somewhat decent coverage for Linux binaries, they'd have to run on x86, ARM, and PPC. Plus possibly MIPS, SPARC, and Itanium. Most of those in 32-bit and 64-bit flavours. Those elves are going to be very fat indeed.
Finally! A year of moderation! Ready for 2019?
I don't get the point in bringing it up.
Things get rejected from the kernel all the time -- because not all things are good, useful, well coded, or solve a problem that needs solving. It's not new in any way.
This in particular seems like a solution in search of a problem to me. Especially since on a 64 bit distro pretty much everything, with very few exceptions is 64 bit. In fact I don't think 64 bit distributions contain any 32 bit software except for closed source that can't be ported, and compatibility libraries for any applications the user would like to install manually. So to me there doesn't seem to be a point to try to solve a problem that exists less and less as the time passes and proprietary vendors make 64 bit versions of their programs.
.
Oh well, so goes it with parts of the Linux culture.
This idea is kind of broken for Linux. On MacOS, with 2 architectures, it makes some sense, since the actual executable code is not huge compared to data. On Linux, withe a couple of dozen architectures, executable code *is* going to start to take relevant amounts of space, and the effort involved in preparing them will be nontrivial. If this system were adopted, virtually no binaries would be made to support all available architectures, meaning that anyone not on x86 (32 bit) would need to check what archs a binary supported before downloading it, which is about as difficult as choosing which one to download would've been.
# cat
Damn, my RAM is full of llamas.
In the entire forked-up mess of the unix tree, there was only one thing that anybody & everybody cared about - source compatibilty. C99, POSIX, SuS v3, so many ways you could ensure that your code would compile everywhere, with whatever compiler was popular that week. For a good part of 4 years, I worked on portable.net, which had a support/ directory full of ifdefs and a configure script full of AC_DEFINEs. It worked nearly everywhere too.
Binary compatibility never took off because there is so little stuff that can be shared between binary platforms. Sure, the same file could run on multiple archs, but in reality that is no different from a zip file with six binaries in them. Indeed, it needed someone to build 'em all in one place to actually end up with one of these. Which is actually more effort than actually letting each distro arch-maintainer do a build whenever they please. OS X build tools ship with the right cross-compilers in XCode and they have more of a monoculture in library versions, looking backwards.
Attempting this in a world where even an x86 binary wouldn't work on all x86-linux-pc boxes (static linking, yeah...yeah), is somehow a solution with no real problem attached. Unless you can make the default build-package workflow do this automatically, this simple step means a hell of a lot of work for the guy doing the build.
And that's just the problems with getting a universal binary. Further problems await as you try to run the created binaries ... I like the idea and the fact that the guy is talking with his patches. But colour me uninterested in this particular problem he's trying to solve. If he manages to convince me that it's a real advantage over 4 binaries that I pick & choose to download, hell ... I'll change my opinion so quickly, it'll leave you spinning.
Quidquid latine dictum sit, altum videtur
My objection is that any such hierarchy of data could be stored as files.
Linux needs tools so that a directory can be manipulated as a file more easily. For instance cp/mv/etc should pretty much act like -r/-a is on all the time, and such recursive operations should be provided by libc and the kernel by default. Then programs are free to treat any point in the hierarchy as a "file". A fat binary would just be a bunch of binaries stuck in the same directory, and you would run it by exec of the directory itself. Also need filesystems designed for huge numbers of very small files and to make such manipulations efficient.
We need the tools to be advanced into the next century. Not use the workarounds of the previous ones as currently practiced on Unix and Windows.
Fatelf was never really a great idea in my opinion. Putting two binaries in a file is not a really good way to solve the problem as there are many more variations of CpU type including all of the x86 variation than one or two. it would be a better idea to do something similar to the AS/400, include, an intermediate form in the file, such as a syntax tree, convert it to native at runtime on the users system, and then store the native code inside the file next to the intermediate code. if the binary is moved to a new system, the native code can be regenerated again from the intermediate code. This does not even requite kernel support, the front of the file put shell code to call the code generator installed on the system, and generate the native code, and then run it. This way, things like various x86 extensions can also be supported and so on.
On Linux, withe a couple of dozen architectures,
Kind of, but not really. No more than there are four architectures (PPC, ARM, X86, X86_64) for OS X. There's two architectures for Linux that actually matter, and they're the same two that run Snow Leopard. X86 and X86_64.
I can see why people are going to get up in arms about this. I've been as big a RISC booster as anyone, I think Apple gave up on PPC too soon, and I'm still bitter about Alpha, but that game's over. 32-bit and 64-bit Intel architectures are what matter, and those are the ones that almost all binaries will work for. I'm not running YDL any more, and neither are you. Game's over, instruction sets lost to marketing. The game's over, the fat lady's sung, picked up her paycheck, and gone home to watch House on her Tivo. Give it up and quit holding up the bus.
This in particular seems like a solution in search of a problem to me. Especially since on a 64 bit distro pretty much everything, with very few exceptions is 64 bit. In fact I don't think 64 bit distributions contain any 32 bit software except for closed source that can't be ported, and compatibility libraries for any applications the user would like to install manually. So to me there doesn't seem to be a point to try to solve a problem that exists less and less as the time passes and proprietary vendors make 64 bit versions of their programs.
EXACTLY! We don't want choice, we want it to just work! Damnit, force people to do things the way they ought to do them, don't give them choice, they'll just screw it up.
Especially when that choice makes things EASY!
Quartz Extreme and Core Image. Are there any other real reasons to spend all that money on generic hardware?
It just shows Ryan isn't used to contributing free software to someone else's project. I once had to wait months before I got my code accepted into a free software project and it wasn't the kernel. If the maintainers add every submission to a project, it will end up in an unstable, unmaintainable mess. Code can last a long time and someone will have to maintain the code even after he's lost interest in it. I am especially leery of code that touches a lot of difference places at the same time, as is undoubtedly the case here.
The problem isn't that its not possible, its that its hard. Your argument is that since its hard now, since the tools aren't ready for it, it shouldn't be done ...
Sounds pretty silly to me.
It would be hard to start from scratch and write a modern OS ... but that is indeed what Linux is.
If you never take the effort to make the hard easier it will remain hard. Changing from single threaded to multithreaded is hard, do you think we should not do that either, because the tools to do it don't make it a cake walk RIGHT NOW?
Seems a silly way to look at things to me. Fortunately other people made multithreading work on other platforms long before x86 could really do it properly, which made it easier to do on x86. Imagine if Linus said 'multithreading in an OS is hard on x86, you have to use timer interrupts and blah blah blah, I'm not doing it' back in the 90s ...
For you, it might not be any different, but you won't know until you give it a try.
For grandma who has a netbook running an ARM processor, and a desktop or laptop running a x86 processor, its probably a little different, don't you think? Do you want to remain in this hole forever, or do you want to get out and catch up to the rest of the world?
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
The problem with fixating on something like ELF for fatness is that it's just the tip of the iceberg.
There's much more to the question of whether or not something will run on an arbitrary copy of Linux than the CPU arch.
Why do you think one of the discriminators in a fat binary can't be a distribution identifier, such that there are fat slices for supporting Debian, RedHat, Ubuntu, etc., all from the dame binary file?
Or that they can't have different slices in the fat binary for Gnome vs. KDE, or desktop vs. Android, and so on?
Also, the arguments about disk space are specious; at least in the Mac OS X world, there is a utility called "lipo" which will pull apart a fat bnary into only the pieces you need/want to install. Typically you only install the actually fat binary on server systems, where the software has to be able to run on multiple client machines, and otherwise you run scripts to reclaim disk space (or in the case of an embedded device, you run them over the install image before you install them).
Same goes for an Apple fat binary really...
Obligatory disclaimer: I am the person who maintains the fat binary loader in the Mac OS X kernel.
-- Terry
Petty fiefdoms and not invented here syndrome will continue to torpedo any chance for a decent Linux on the desktop. Until Linux has a single binary and a universal installation strategy they will continue to be mostly harmless and largely irrelevant to the desktop market at large.
Why bother
Commercial Games. That's who.
maybe its just me but i see 0 advantages for an executable with multiple binaries.
shouldn't this all be handled by the package manager? isn't including all these binaries just jacking up download sizes for no gain?
a boot CD that can run on multiple archs is the only real use i see for this, but i would have to think there is a better way handle that than changing the fundamentals of executables and libraries.
maybe he received a less than warm reception from other devs because his idea provided virtually no benefit to the end user and required more work by the devs.
I want fat binaries for microcontrollers! Give me binaries that can run on PIC16F88, eZ80 and 68HC11!
There's nothing worst than having to replace a 0.50$ chip with another that cost 0.51$!
If I don't care about performance I can already write in Perl, Awk, Python, Tcl, or something. Why do I want to put up with Java?
Well maybe what we could use instead in C byte code, or some other form of byte code, and then have on the JIT low-level compilations.
My first reaction to FatELF was that it is a good idea, since it works on the Mac. After listening to the issues people bring up with the large selection of CPU architectures I can understand this issue. On the other hand, why not allow ELF to support multiple architectures and let the 'market' decide if they want it? In a worst case scenario, it is just minor over-head that doesn't really impact anything.
Jumpstart the tartan drive.
No because a package manager makes it easy to install software for the current arch. Even grandma doesn't benefit from having x86, AMD64 and arm binaries in a single package, much less from some random untrusted binary she downloaded from the internet.
The issue wasn't that there were lots of people saying "That's a stupid idea" or "That's a stupid implementation of an otherwise good idea."
The issue was lots of people saying "You are stupid."
There is a big difference.
I'd weighed in on this, because in the embedded systems I design this actually would have been useful - I have to support different processor types with what is, ideally, the same software load. (Just because MY embedded systems are much larger than some 4-bit microcontroller running 16K of code doesn't make them any less embedded.) People called ME stupid - not "That's a stupid design" or "That's a stupid reason to want FatELF", but "You are stupid."
Yes, developing a thick skin, so that when somebody says "That's a stupid idea" you realize that it is the IDEA, and not YOU, that they are criticizing, is important to any engineer.
But at the same time, saying to somebody "You are stupid" just because you don't like their idea, or don't see how it applies to your needs, is immature and unprofessional.
www.eFax.com are spammers
It has been reported that a law suit was filed against Keebler corporation by a group of fat elves.
Their spokes person was quoted as saying,"It's *mmmm* their fault, these cookies *numnumnum* are just too good!".
I only look human.
My mother is a halfling and my dad is an ogre, so that makes me an Ogreling
I don't understand what the kernel has to do with any of this. Fat binaries can be (almost) completely implemented at the userspace level by extending the dynamic loader (ld-linux.so). The way this would work is that the fat binary would have a boilerplate ELF header that contains just enough information to convince the kernel to load it and launch its interpreter program, which could piggyback on the standard dynamic loader. The fat binary interpretter would locate the correct architecture within the fat binary, map its ELF header into memory, then call out to the regular dynamic loader to finish the job. The only hitch is that a 64-bit kernel will refuse to load a 32-bit ELF, and vice-versa, so you would need an EXTREMELY minor patch to the kernel to allow it to happen. I mean like a one-liner.
But the truth is the Linux movement needs every warm body it can to fight Microsoft.
</quote>
THAT is the problem. Stop trying to FIGHT Microsoft. Start making better software. Innovation, something so tremendously better they start copying YOU.
Vis-a-vis AMD and Intel and x86_64 and VT extensions.
Except software has zero marginal cost, so once you take the lead, it'll take a serious fuckup, and not just money to lose it.
But the lack of universal binaries is not the reason why it's hard to release closed source software on Linux.
I have built cross-distro binary-only applications before.
Some notes on doing so:
Make sure you compile, link against a old version of glibc, this prevents issues of running applications built against newer versions of glibc spitting out "undefined reference errors" on systems with older glibc (much like when you compile a windows program against the winvista platform sdk and try to run that program on XP).
If you must link against the C++ runtime library (libstdc++), then provide a custom version of it with your software. That's about all you need (much like how many Windows applications come with the msvc runtime dlls they were compiled against).
This is legal, no source requirement (outside of providing the source to libstdc++ when requested - which wouldn't reveal anything).
I honestly don't see how this is a substantial difference from Windows, could you explain it better, please?
Change is certain; progress is not obligatory.
You missed half the argument. It's hard, and it's pointless.
The grandma who has a netbook running ARM and a desktop running x86 will install software by going into Add/Remove Programs and picking "Fun Times Photo Album for Grandmas" out of a list. The package manager will figure out what needs to be installed for her, on both her ARM and her x86 computers.
She's not going to go to some random website and download a random installer file and use it on both her computers - her kids have told her over and over again that that's not safe, and she may lose her knitting patterns if she does it.
Seriously, the people who advocate this junk seem to be entirely unaware of the joys of package management. All FatELF does is re-solve a problem that package management has had licked for a couple of years now, and it solves the problem in a less efficient way.
It's hard, yes - but it's not worth doing just because it's hard.
Commercial Games. That's who.
Exactly. Take Blizzard, who ships Windows and Mac versions of their games on the same media. Fat chance of getting an official Linux release in the absence of a universal binary solution. Blizzard tends to ignore platform-specific package formats in favor of their own installers, the better to control and customize the installation experience. By avoiding the standard MSI format on Windows, for instance, they avoid introducing a lot of unrelated dependencies and vastly simplify the post-release patching process.
If you don't mind hacking around on the command line to get a game to work, the current state of affairs probably suits you just fine. But there's no business reason for Blizzard to support Linux users with an official release, if the best they could provide is a different set of command line inputs to type in. This of course assumes they would not develop installers for every Linux distribution on every compatible architecture, along with the necessary documentation and technical support for each. I think that's a fair assumption.
Regarding hard drive and network speed, in today's world of gigabit to the desktop and 10 gigabit backbones and 2 terabyte hard drives I don't know what you're talking about with "10 megabit" and "20 megabyte" cracks. You do realize that the primary expense in a networked workstation environment is administration, not hardware, right? The proper use for local hard drive in a networked workstation environment is for caching, not for software installation. We knew this truth about workstation management twenty years ago, but for some reason it has been forgotten in a world where Microsoft and their deranged horribly expensive and virtually impossible to manage workstation environment seems to be the model for how to do things. How many years of IT experience did you say you had, again? :).
Send mail here if you want to reach me.
Agreed. There is also the question of why the authors of a GNU licensed program are expected to show a passionate interest in a project whose central focus is the distribution of binaries. FatELF makes sense when the source is not available. When the source is available, package management, with a per architecture build, makes more sense.
Here's the log of the full conversation. We was hardly abused, and got persistent help over several hours which was patient and helpful, ultimately culminating in him realizing what he had done wrong and admitting to it.
Clearly, the myth IRC folk are at fault.
http://pastebin.com/m2cfd19dd
Programming languages are already far two high level which incurrs a performance hit. We should all be coding in assembler. Personally, for fast executing binaries I prefer to tap the bits into the hard drive platter with magnetized needle.
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.
FatELF is a stupid implementation of a stupid idea. I.e., even if you want fat binaries, modifying the ELF format is the wrong way of doing it.
Yeah for the Linux kernel developers for keeping this kind of crap out of the kernel.
So, remind me again: why exactly is it not possible to implement all that in a package manager and we need to have a Really Fat ELF?
Because Linux distributions can not agree on a single GUI technology, let alone a package manager.
-- Terry
I've never used a package manager that forced you to upgrade all dependencies to the latest version to install a package. All of them allow not just required packages but required versions of packages, and only force upgrades of dependencies when you don't have a sufficiently recent version.
Some can do patches. I think RPM can. But unless you're using dialup, they're not really that much of advantage. And you also have the problem of having to provide patches from lots of versions to lots of versions. Or you can provide only patches from the last version to the current one, in which case they're useless for anyone who misses an upgrade.
I don't know any package manager that does this. For example, Pacman, the package manager of Arch (my current distro of choice), installs new versions of files with the suffix '.pacnew' if the old version was modified and doesn't clobber.
This is true on pretty much any OS. Multiple versions of the same package will install to the same paths, and your package manager would have to be pretty fucked up to do that. If you'd like to horribly violate widely adopted filesystem organization standards and patch your software a bunch to make it work properly with your new layout, you can do that, but there's no real gain.
They manage packages. Just because they don't implement every feature you'd like doesn't mean that's not true, though evidently they do implement many features you'd like, but you are too busy raging pointlessly to pay attention to the facts. In fact, the only feature they don't implement that you'd like is a major design decision that would require altering pretty much all software on the system for no benefit to the vast, vast majority of users.
Then you get it from elsewhere if the official repos don't provide it. You can even build your own package, something you certainly ought to be capable of if you're applying your own patches to software. You can even set up your own repos!
So basically what you're saying is, "you're not using the package manager except if you are". Gee, really?
Most of your post you've been toeing a fine line between being just wrong and bei