Slashdot Mirror
CDE — Making Linux Portability Easy
ihaque writes "A Stanford researcher, Philip Guo, has developed a tool called CDE to automatically package up a Linux program and all its dependencies (including system-level libraries, fonts, etc!) so that it can be run out of the box on another Linux machine without a lot of complicated work setting up libraries and program versions or dealing with dependency version hell. He's got binaries, source code, and a screencast up. Looks to be really useful for large cluster/cloud deployments as well as program sharing. Says Guo, 'CDE is a tool that automatically packages up the Code, Data, and Environment involved in running any Linux command so that it can execute identically on another computer without any installation or configuration. The only requirement is that the other computer have the same hardware architecture (e.g., x86) and major kernel version (e.g., 2.6.X) as yours. CDE allows you to easily run programs without the dependency hell that inevitably occurs when attempting to install software or libraries. You can use CDE to allow your colleagues to reproduce and build upon your computational experiments, to quickly deploy prototype software to a compute cluster, and to submit executable bug reports.'"
CDE will always mean Common Desktop Environment to me.
Wow, static linking, did anybody for even a second think it is kinda weird to have the same lib on the machine over and over and in every old exploitable version you can find?
Dear god no.
I do not want to execute installshield or any similar crap/wizard for every little thing I install.
I do not want to have a system tray/task manager full of two dozen vendor's update checker processes, each individually bugging me about how I'm running WidgetFoo 1.8.1.20.1.3, and it is critically important that I execute WidgetFoo's custom one-off graphical update wizard with 3 or 4 pages to click through to get to 1.8.1.20.1.4. Then rinse and repeat once per app instead of knocking them out in one shot/dialog/icon/process.
I do not want each application to bundle their ancient ass directx library or ancient library from visual studio or any other similar crap.
Windows installs were not historically 'easy' due to any effort on MS's part (installshield and friends made an entire business out of covering for MS' lack of help, even as MSI matured into a usable solution). Linux (specifically Debian) really got this right first. Apple recognized that model and made it a great success on the iPhone, setting the tone for all of modern mobile devices. Debian did it right first and never gets the credit.
XML is like violence. If it doesn't solve the problem, use more.
I think most people here are not understanding the target audience for this tool (hint: it's not for your typical linux environment). It's not about package management or having a universal installer... it's about being able to run your application in a different environment where you don't have admin rights.
In a lot of university clusters or compute grids researchers have access to a large collection of compute nodes, but they usually don't have any rights to those machines. In fact, most of the time the programs are ran in a sandbox and have a restrictive environment. To run their codes reliably, researchers often have to perform some sort of static linking or package up all of the dependencies with the executable. apt-get or yum are not options in these environments... you may not even be able to ssh into them. Ideally, you could ask the system administrator that controls the cluster to install certain packages, but again, this is not always possible particularly if the researcher requires a niche package used in their domain.
Moreover, the cluster may be composed of heterogenous set of machines with different versions of Linux. Package management does not help you here. The only way to reliably execute your programs on such a heterogenous cluster is to statically link or include your dependencies. If you are wondering who would use such a maddening environment where you have no admin rights... google Condor, OpenScienceGrid and Globus. This is how a lot of research computation is done.
Of course, the hot new thing is virtual machines and clouds... but firing up a VM each time you want to run an application is very heavyweight... especially if your applications has a short run-time.
TL;DR: this isn't for your typical ubuntu or fedora install; it's for scientific research that is done on restrictive computing clusters and grids.
As a side note, I made and use a much cruder tool http://bitbucket.org/pbui/starch/ that packages everything up (executables, libraries, and data) in a self-extracting tarball which can be executed on remote hosts. It's not as slick as CDE, but it's been used with success by various research groups that I collaborate with.
Generally linux distributions follow a fairly standard naming/location convention for files, most of the variations exist in specialised linux distributions (eg android) where there is good reason for the differences. /usr/local.
Most software also allows you to choose where to install it at compile time, although the default will usually be
A linux system is often far less messy than a windows system for instance, where all kinds of files are under the windows and system32 dirs.
Package managers are actually a very good solution to many problems, not only do they handle dependencies but they provide a centralised database of installed software, a file integrity database (both on the system - storing checksums of everything, and off system because the checksums corresponding to a given package versions files are known), clean removal of software, a single place and standardised interface for installing software (thus removing the need to download programs from potentially untrustworthy websites - you only have to trust your os vendor, not hundreds of third parties) and most important of all, a centralised update mechanism for applying important security patches to all of your software...
Other software vendors have chosen different methods to try and resolve the same problems, but most of them are lacking in one way or another, or make different compromises...
The OSX method of program bundles avoids dependency problems, but introduces the inefficiency of reducing code sharing, this has less impact on closed source software where code is rarely shared anyway, but for open source one of the key advantages of the open development model is reduced by this approach. On the other hand, this method does provide clean removal and makes it easy to have multiple versions of something installed.
The Windows method is rather chaotic, individual programs are expected to create their own installation and removal programs as well as handle their own update mechanisms, this has resulted in a whole range of software which behaves in different ways, stores files in different places etc... Update mechanisms and uninstall routines are down to the individual application and may not exist at all, or may not work correctly. This has resulted in lots of very poorly behaved software which assumes you are a privileged user and can write to system locations, and subsequently in order to retain compatibility microsoft have been forced to implement all kinds of dirty kludges to make such applications think they are able to write to system dirs when they can't.
The only potential downside to the linux system, is that application suppliers don't have a fixed list of system libraries which will always be present. Under OSX or Windows you know that a core set of libraries will always be there, and anything else is typically provided by the app (sometimes redundantly), whereas different linux distributions may provide different base libraries.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
Sorry if topic sounds a tad personal, but hey...
> The real problem is that Linux distributions, taken together or individually, presents developers with too many completely unnecessary choices as to where essential library files can be put, and also, there is no standard version naming and locating convention.
Do you need it to boot? Prefix is / /usr /usr/local or /opt /import/x86 etc is a good place
Do you need it after boot? Prefix is
Do you want to install custom stuff that is not handled via the system's default software handling solution? Prefix is
Do you want to install into home dir? Prefix is ~/local or ~/opt
If you are in a heterogenous environment with shared home between lots of architectures etc,
This leads to clean & clear separation of software after a system people poured a lot of thought into. Is it easy to grasp at first sight for someone used to Windows? No. But that is _not_ the priority. Sorry, it's not. People writing code need to learn how the language works. Why shouldn't they learn how to system works?
> Package managers are a complex solution to a problem that need not have existed in the first place, if it was realized that unnecessary choice is deadly dangerous, in the world of large-scale software interoperability.
Yeah, cause grabbing random downloads of .bat, .exe, .msi, .whatnot turned out to be awesome. Especially the integrated updates. Oh, what's that? Everyone is implementing their own system leading to dozens of parallel update mechanisms on a single machine? Now _that_ is efficient! And the programs that don't have an update routine? Simple, just write them bug-free, without holes and a complete feature set in 1.0!
> There does not need to be any choice for where on a file system a given application or a given library should be located.
That is true if you consider every machine to be an island. Unix thrived and continues to thrive cause you can create huge shared environments with almost no work.
> That should be completely determined by the app or library name, version (using a standard versioning scheme), variant (using a standard variant naming scheme), and origin person-or-organization, using a standard organization identifying scheme.
My custom mplayer is in /usr/local/mplayer. My custom git is in /usr/local/git. My custom vim is in /usr/local/vim. I can delete any of those and remove the program, along with all its libraries and whatnot, with one single rm. /usr/local for stuff, again... It's their problem, same as if they did not know how to open() a file.
If devs simply don't know that they should default to
> It goes without saying that there should also be a standard globally unique URI for such libraries and apps (including the unique name, version, variant, origin identification).
No. No. No. This breaks any and all assumptions about being able to install different versions of stuff for different reasons. Use prefixes and use LD_PATH, etc.
> So there should be no choice about where on the internet to get it (except for the choice involved in a standard mirroring URI scheme), and
no choice about where to put it.
Maybe you are too young to have seen this yourself, but after a few years, most URLs are dead. With gittorrent, ideally with a DHT sprinkled on top, this might change in the longer run, but what if the next VCS that whoops git's ass comes along? Static information on the internet is mostly a myth. (Also, git would need to get rid of SHA1 for fully automated code distribution, imo)
> With this discipline, obviously needed in today's universe of code, all such package management, as well as dependency acquisition and installation, could be managed by a single unified and incredibly simple automated package manager; call it the