Slashdot Mirror
CDE — Making Linux Portability Easy
ihaque writes "A Stanford researcher, Philip Guo, has developed a tool called CDE to automatically package up a Linux program and all its dependencies (including system-level libraries, fonts, etc!) so that it can be run out of the box on another Linux machine without a lot of complicated work setting up libraries and program versions or dealing with dependency version hell. He's got binaries, source code, and a screencast up. Looks to be really useful for large cluster/cloud deployments as well as program sharing. Says Guo, 'CDE is a tool that automatically packages up the Code, Data, and Environment involved in running any Linux command so that it can execute identically on another computer without any installation or configuration. The only requirement is that the other computer have the same hardware architecture (e.g., x86) and major kernel version (e.g., 2.6.X) as yours. CDE allows you to easily run programs without the dependency hell that inevitably occurs when attempting to install software or libraries. You can use CDE to allow your colleagues to reproduce and build upon your computational experiments, to quickly deploy prototype software to a compute cluster, and to submit executable bug reports.'"
CDE will always mean Common Desktop Environment to me.
Making Linux easier for the masses is always a good thing.
+1 obvious, but it's as simple as this to make something more popular. The easier it is to use the more people who will use it.
I'm just pointing out a major application - that's not so major anymore - Common Desktop Environment uses this acronym :)
Does sound like a neat tool though!
To more quickly prepare software for easily installation.....
Another non-functioning site was "uncertainty.microsoft.com."
The purpose of that site was not known.
I don't understand why some people insist that dependency hell is inevitable. I have been using Debian for 8 years and Ubuntu for 4 years, using them for a wide variety of purposes, and I never experience anything that I could describe as "dependency hell"... (unless you count trying to cross-compile my linux code for win32)
That being said, I think this is CDE thing is cool, and is a good idea. I just don't think it is a good idea for the same reasons that the original poster specifies.
Wow, portable apps, just like an old Mac. Did anybody, for even a second think it's kinda weird for a program to splatter its parts all over the disk and into every directory it can find?
For justice, we must go to Don Corleone
If those libraries are GPL or LGPL, then when you deliver the binary of the library, you must also deliver the source or an offer to deliver the source, and you must also deliver a copy of the (L)GPL, as part of the CDE. Is this done?
Great, now we can have outdated exploitable libs and every other kind of BS that comes with this. Might as well just statically link everything. Package mangers exist for a reason, use them. Do not bring the errors of Windows to us.
I think this breakthrough may just be the start of the year of linux (on the desktop).
-F
This sounds like an easy way to copy installed proprietary software?
Knowing what files a program will open without running the program is impossible, and since a program can dynamically change what files it opens from run to run, it would be impossible to predict every file that a program would require in all situations. The best that a tool like this could do would be to record the files that were used during a given run of the program and assuming that the program when run later with the same inputs would use the same files, support running the program with exactly those inputs.
I don't think you could reliably package up many programs in this way for general use, but maybe you could package up a program for a specific task like producing a reproducible bug report. But that is of little utility as most distributions that you would run a program on and need to submit a bug report are fairly easily reproduced anyway and the developer probably would have an easier time just reproducing the bug on their own system without fussing with some packaged up binary that the bug submitter sent in.
This honestly seems like a wrapper around "strace | grep ^open". If this is what it takes to be a researcher at Stanford these days then sign me up!
but i can't read which license this software adheres*?
*just in case anyone wants to take a look and change some source code...
Slashdot ya no es que lo era!
Uhm, but using the packaging system already present can do the same with less waste, better support for distribution and with existing tools. File-based dependencies like in RPM may be slightly deficient here, but with package-based dependencies like in .DEB you have full control over what you want.
There were multiple such systems for working against the packaging system already like autopackage, and they turned out to be a disaster. I fail to see how this one is any different.
The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
I wonder how far you can take this? For example, a program that requires certain versions of: Mono, glib, gphoto, and other libraries, would it be able to grab all of those?
Then, if you have something really complicated, you're going to eat up a huge amount of disk space.
Now, I'm going to ask myself, 'this is fucking awesome and wtf am I picking on it so much when I couldn't do any better?'
Time for my anti-dork pills....
RIP America
July 4, 1776 - September 11, 2001
if you write proper packages for your distro. That is after all, the whole point of the package manager...
doesn't this make cross distribution distribution of applications much easier?
meaning, as an example, if Adobe wanted to release Photoshop for Linux, it could without have 50,000 different package formats for every different version of every different distribution?
if so, this could be very awesome.
This is great for application developers that put out new version faster than distros can keep up (Example, Qt-Creator, which i download always as binary from nokia site). The problem is when you want to try an app that is either of something larger (KDE, Gnome), or an actual desktop environment, dependencies make it impossible..
Deja vu: AKA the Loki installer. Still used by CodeWeavers for their distribution independent CrossOver packages. See http://en.wikipedia.org/wiki/Loki_Software and http://icculus.org/loki_setup/ which is the installer's home after Loki Software's closing.
Um, if you are managing a cloud and cannot figure out how to package up your own code and install it as part of a kickstart then you need to be hit and have your sysadmin privileges revoked ASAP, not necessarily in that order.
This is obviously not meant for distributing programs (and the guy doesn't claim it is either; he's using it for repeatable research). It takes a snapshot of the entire filesystem on the system on which it's run. This not only results in huge packages for anything nontrivial, but also simply won't work for things that are system dependent, like, say, OpenGL, or which sound system to use. It also restricts file system access to the CDE package.
Portable, standalone packages can be done the right way by compiling against an older version of GLIBC, including required libraries in the package (GLIBC is backwards compatible, so don't include that, and don't include the entirety of X11, or OpenGL, or sound drivers, etc), and then using either RPATH on the executable or a script that sets LD_LIBRARY_PATH so that the executable finds and uses your included libraries. This is how it's done by many commercial Linux games, it works fine.
Many apps aren't so simple. For example, say you have a GPGPU app that links against a hypothetical library dynamically. That ABI represents a cross-vendor interface so that, say, nVidia or AMD's library could satisfy the link, depending on which hardware is installed. Pulling in that link could be very counter-productive. I don't think GPGPU works like that, but more esoteric things do. The other case is where a dynamic load or file operation is highly conditional. I presume he uses ldd strace and follows the syscalls to pick up the things ldd cannot. If the code doesn't happen to traverse the path, it won't get hit.
When positioned as a 'cloud' tool, lends itself to exacerbate I/O load and memory load. It's a philosophy that undoes a lot of the efforts towards efficiency that dynamic libraries were intended to do. It may be fashionable to scream 'cloud' on every single little thing nowadays, but shared libraries and common infrastructure are a good thing. Even if the apps will be separated by a harder virtualization way degrading efficiency, at least memory dedupe strategies can still dig in and get back some efficiencies if common platforms are used, but if library fragmentation is encouraged, that too will be lost.
Now, in a high performance computing application, interesting way to build 'just enough' OS to run an application. This is a tad different as each server has no more than one real application in flight at a time, and therefore need not worry about memory inefficiencies of running many apps on the same box without the full benefit of shared libraries to mitigate load. Deliver that small package every reboot and you give job submitters ultimate flexibility over their environment without drastic speed penalties to apply them. No need of local disk to host the platform and little to no memory waste as the OS image has only files that would be memory resident anyway.
I very recently published a tool that performs a similar task. dynpk (my tool) bundles programs up with packages from your system, then wraps them with some other stuff to create a bundle that essentially allows you to run a Fedora program on a RHEL machine (and probably Ubuntu or Debian, but this is outside my needs...).
Recompiling loads of libs for RHEL isn't fun or particularly maintainable. Therefore, use the ones from Fedora!
Seriously though this could end up making collaborating on software a lot easier among trusted people.
You can even do it with a Gtk app.
If you're looking for a commercial product that can do just this -- but in a supportable way, and with lots of nifty features that enable building appliances and deploying them to the cloud...take a look at rPath: http://www.rpath.com./ We're using it at our company on thousands of systems and love it!
---- Breakbeats are not just music...they're the soundtrack for my life.
No! I'm not going back! I'M NOT going BACK! MOTIF IS DEAD TO ME!
Maxim: People cannot follow directions.
Increases in truth directly with the length of time spent explaining them
I am far from a guru, but I seem to have no problem running FF 3.6xx pre dailies and also 4.x beta FF on my ubuntu install 10.4 lts. No having to upgrade to do it either, just enable the correct PPAs in the repository list.
A good idea, but there are easier ways to do the same task. The project cpos-recovery on Sourceforge does essentially the same task, but relliably and quickly. There are several ways that this kind of quick copy can be used: 1) back up to an alternate partition as a backup boot partition if you want your system to be bombproof. 2) make an installable tar file that can be quickly applied to many new systems. 3) create another partition of a different type, and copy your system to it (think ext4 to btrfs), with the assurance that the system will boot, and run, properly (given that grub supports that filesystem). 4) create an offsite backup of your working system in case of problems. These are all easily done with cpos-recovery, so check it out.
When I can comfortable run nightly, beta, and stable versions of firefox in linux then I'll know this has been a hit. looking forward to that.
about damned time. I said that this is what was needed back in the 20th century. NOW we can see linux adoption begin to take off. Who amongst the great unwashed users, can begin to figure it all out? NONE. everyone just wants to click the download link and HAVE IT WORK OUT OF THE BOX !!!
this is what I've been asking for... just like OS X
You picked a bad subject, my friend.
apt-get install hearse
Seriously.
This thing is not going to catch things dynamically loaded.
Not all that useful.
It might be amusing to see what:
cde emacs
creates though.
apt-get install foobar
This brings four problems that I can think of:
Wait...this sounds like it's just a couple half-steps shy of an automated, app-specific chrooting system.
Concerns about outdated libs aside, that sounds...awesome.
tasks(723) drafts(105) languages(484) examples(29106)
While I can see this being useful for scientists, it's probably not going to be so useful in terms of an end-user environment (not that it couldn't be used in one). For starters, this is really only good for command-line packages as GUI applications, the one that end-users are most likely to use, would end up including most of the X and UI libraries during the CDE detection, even if the files on the target computer are compatible with the machine.
The other issue with having different projects for different distribution purposes (Klik for applications, CDE for scripts, etc.) is that it induces fragmentation; the very thing these projects are trying to get rid of by no longer relying on the package management system.
Disclaimer: I work on AppTools (http://code.google.com/p/apptools-dist) and therefore I'm biased :)
1. DLL hell - for those who used windows, you know what it is
2. Shared Objects? What's that?
3. xcalc is now 200 megabytes, WTF?
4. Security exploits unpatched galore
5. Support? What support? We can't support every distro that has different file system default file locations!
This is what static linking is for. You can make a binary and selectively link in as many libraries as static as you need to run on older or older machines.... and it doesn't have to be the same kernel version either.
I have static programs that are over 10 years old (big, GUI-intense ones even) that still run 8-10 years later because they are very static.
In any case, for commercial software, static and/or library-including makes sense. It is about the only way you can guarantee your software will run on most or all of the distros, several versions back, without having to make dozens and dozens of separate downloadables of your stuff.
'CDE is a tool that automatically packages up the Code, Data, Environment, and Run-time Security Vulnerabilities involved in running any Linux command so that it can execute identically on another computer without any installation or configuration.
Remember... sometimes system libraries have vulnerabilities and really do to need to be obsoleted or updated, even if the software that uses them doesn't receive updates too often.
That's why there's this whole concept of major and minor version numbers for libraries.
Packaging it up with runtime is no better than static linking, and this therefore problematic.
It's bad enough to have to update system libraries, thankfully package management handles that.
If CDE is bringing its own environment to the table, it is obviously not working with the package management system.
CDE would be much more useful and safe (not a huge security risk like it would be in its current state) if it prepared distro-specific scripts to use the vendor's package management system to correctly install dependencies, which would be managed and updated as needed from there on out.
Unless I'm missing something, this seems like a relatively-useless tool. There are better ways to do everything that it does.
I can see this being useful in two specific cases:
In the former case, CDE is just an extreme form of static linking. You produce a distributable package that contains the full set of libraries, files, etc. that are needed to run the application. That's useful, I suppose, but how does it compare to a regular Linux package? It's heavier, for sure; every CDE package contains a full set of system libraries, binaries, data files, and executables needed to perform an operation. It's also slower, as CDE execution has to re-create the original environment (in-memory or on disk, either way). All to ... what? Cut down on the difficulty of dependency management? Almost every modern Linux distro has an advanced package management system that can easily handle these cases. In some cases it's as easy as declaring dependencies and producing the appropriate source / binary package via a build wrapper. In others, you may have to do a little architecture, but it's still relatively straightforward. Any existing Linux package, by design, seamlessly integrates into the current system and is quite minimal.
I understand the desire for reproducible behavior, but this is easily obtainable with a package-based solution across the same distro. The only practical cases (even including cloud deployment) that involve multiple different distros requiring reproducible behavior are fringe cases, and even then a build-from-source solution is likely adequate.
So now to the other case... I require absolutely perfectly reproducible behavior that I can provide to other people. That sounds like the perfect use-case scenario for a virtual machine. Packaging a VM up and distributing it provides not only the CDE convenience, but additional guarantees as well. For example, a modification in kernel behavior between minor versions could affect the runtime performance of a CDE-distributed application across various platforms; a VM with the exact same setup would be able to guarantee that this is not a problem. In cases where perfect reproducibility really matters, a VM is far more compelling than an execution environment wrapper.
Don't get me wrong; I appreciate the coolness factor. The interior mechanisms that CDE uses to operate are very well-thought-out and interesting, for sure. I am just very skeptical as to the practical usefulness of the tool. It seems too light and incomplete for serious academic reproducibility, and too heavy to solve real-world distribution issues. YMMV, but I don't see this as a real game-changer in any capacity.
From reading the article, I don't think that this is intended as a replacement for deb/rpm or tarballs with configuration scripts. (If it is, it is a very bad idea.)
The tool constructs a reduced dynamic I/O trace of a single program run to determine file dependencies, then packages these. This is great if you're benchmarking deterministic programs on different hardware, and useful for quickly migrating your code onto a bigger machine when a paper deadline is around the corner.
But again, the files it picks are based on a dynamic trace. If you run `cat foo.txt', then `foo.txt' will be part of the trace. If you run it on apache, all the apache modules that apache wound up loading will be part of the trace, but no others. So it's not very general-purpose. Still useful, in the right situation.
I am not a Linux guru or pro or much of an enthusiast for that matter. More of a "Linux on the weekend" type. I've been playing lately with "light" linux distros I want to use to host VirtualBox so I've been doing a lot of installing of VB. The process seems overly complicated and a little different each time I go through it. Even when there's two different distros that are derived from the same parent (like Ubuntu or whatever). So my first thought when i saw this was that it would make installing VirtualBox and its various libraries that much easier. I mean I just want to get it over and capable of being run successfully. Then I can do all the updates to the libraries auto-magic with apt-get or whatever. Just having a single file with included libraries seems like it should be a good thing. Am I missing something? Actually this is how it is on Windows: one VB installer package and you're up and running. No Perl or other runtimes to worry about.
"UNIX is very simple, it just needs a genius to understand its simplicity." -Dennis Ritchie
I think it's a pretty "good thing"tm but would be better if
it looks for identical versions of library files (in other locations than where it wants them) on
the target computer, and just creates symbolic links to those files as long as their permissions
are ok.
Where are we going and why are we in a handbasket?
No, it won't. Unlike Windows, Linux is perfectly happy having multiple versions of any given library installed and available: Fortunately, on Unix-like systems (including Linux) you can have multiple versions of a library loaded at the same time, so while there is some disk space loss, users can still run ``old'' programs needing old libraries.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
The real problem is that Linux distributions, taken together or individually, presents developers with too many completely unnecessary choices as to where essential library files can be put, and also, there is no standard version naming and locating convention.
Package managers are a complex solution to a problem that need not have existed in the first place, if it was realized that unnecessary choice is deadly dangerous, in the world of large-scale software interoperability.
There does not need to be any choice for where on a file system a given application or a given library should be located. That should be completely determined by the app or library name, version (using a standard versioning scheme), variant (using a standard variant naming scheme), and origin person-or-organization, using a standard organization identifying scheme.
It goes without saying that there should also be a standard globally unique URI for such libraries and apps (including the unique name, version, variant, origin identification).
So there should be no choice about where on the internet to get it (except for the choice involved in a standard mirroring URI scheme), and
no choice about where to put it.
With this discipline, obviously needed in today's universe of code, all such package management, as well as dependency acquisition and installation, could be managed by a single unified and incredibly simple automated package manager; call it the TPM (trivial package manager).
I predict that's the direction things will move. It's the direction they should move. ;) evil
All programmers repeat after me:
-Unnecessary choice is (the root of all
-The default shall be good
Yes. It's a religion. But it's good for you. Now eat your vegetables.
Where are we going and why are we in a handbasket?
Dear Professor,
I think I found a bug in my environment. I just tried to copy my robot simulation environment that I run on my Intel PC to the linux that runs on the arm command board I control my robot with.
For some reason I cannot run the program.
Can you help, thanks?
Cool stuff; this allows the automatic creation of chroot environments.
You say this like it's really a problem.
Linux distributions differ for good reason; some distros (often the embedded ones) may have no /etc or /usr/share or other paths because they don't need them and are lighter or faster because of it.
It's not a defect, it IS a feature.
Linux is not Windows, it is meant to embrace ALL environments.
Is it so hard to adapt a *nix program from one *nix to another? No, it's only a matter of knowing the target distro properly.
I don't know the meaning of the word 'don't' - J
Anyone can build any one or more of 1000s of programs from the source, and simply enable static libraries rather than dynamic in the ./configure part of the build, to create programs that can be copied to any Linux distributions that adheres to the Linux Standards Base and it will run just fine. There is a reason why we don't do so, however, and that reason is because we understand security, and know that it would be a phenomenally foolish thing to do. This has been especially true for more than half a decade, since it is easier to click the checkbox next to the program I want in the GUI based installation tool, and let the system just grab it from the repository. If I have 1000 machines to manage, I can automate the whole thing in about 10 lines of Python code.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
this is what i get /usr/local/bin/work# cde ../lame
Fatal error in copy_file [cde.c:190]
I can't tell if that should have been +1 Funny or -1 Fucktarded. I have a suspicion that it's the latter but it is 3am and I'm starting to nod a bit, so I'll leave it to someone else to decide.
Violence is like duct tape. If it doesn't solve the problem, you didn't use enough.
I wrote this script to relink a dynamically-linked program with as many equivalent static libraries as possible, with any unavailable static libraries left dynamic. I use it to ship a static version of libguestfs
Whether static binaries are a good idea, I'll leave that discussion up to others. I would say that dynamic linking should be preferred, but static linking and this script is quite useful in some circumstances.
Rich.
libguestfs - tools for accessing and modifying virtual machine disk images
ihaque's description of CDE is a bit misleading. The point of this tool isn't so that you can install applications easily. The point of this tool is reproducibility. You should only use CDE when you need to exactly reproduce your environment on a different machine.
The target audience of this tool is testing and research, not the general population.
Generally linux distributions follow a fairly standard naming/location convention for files, most of the variations exist in specialised linux distributions (eg android) where there is good reason for the differences. /usr/local.
Most software also allows you to choose where to install it at compile time, although the default will usually be
A linux system is often far less messy than a windows system for instance, where all kinds of files are under the windows and system32 dirs.
Package managers are actually a very good solution to many problems, not only do they handle dependencies but they provide a centralised database of installed software, a file integrity database (both on the system - storing checksums of everything, and off system because the checksums corresponding to a given package versions files are known), clean removal of software, a single place and standardised interface for installing software (thus removing the need to download programs from potentially untrustworthy websites - you only have to trust your os vendor, not hundreds of third parties) and most important of all, a centralised update mechanism for applying important security patches to all of your software...
Other software vendors have chosen different methods to try and resolve the same problems, but most of them are lacking in one way or another, or make different compromises...
The OSX method of program bundles avoids dependency problems, but introduces the inefficiency of reducing code sharing, this has less impact on closed source software where code is rarely shared anyway, but for open source one of the key advantages of the open development model is reduced by this approach. On the other hand, this method does provide clean removal and makes it easy to have multiple versions of something installed.
The Windows method is rather chaotic, individual programs are expected to create their own installation and removal programs as well as handle their own update mechanisms, this has resulted in a whole range of software which behaves in different ways, stores files in different places etc... Update mechanisms and uninstall routines are down to the individual application and may not exist at all, or may not work correctly. This has resulted in lots of very poorly behaved software which assumes you are a privileged user and can write to system locations, and subsequently in order to retain compatibility microsoft have been forced to implement all kinds of dirty kludges to make such applications think they are able to write to system dirs when they can't.
The only potential downside to the linux system, is that application suppliers don't have a fixed list of system libraries which will always be present. Under OSX or Windows you know that a core set of libraries will always be there, and anything else is typically provided by the app (sometimes redundantly), whereas different linux distributions may provide different base libraries.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
Sorry if topic sounds a tad personal, but hey...
> The real problem is that Linux distributions, taken together or individually, presents developers with too many completely unnecessary choices as to where essential library files can be put, and also, there is no standard version naming and locating convention.
Do you need it to boot? Prefix is / /usr /usr/local or /opt /import/x86 etc is a good place
Do you need it after boot? Prefix is
Do you want to install custom stuff that is not handled via the system's default software handling solution? Prefix is
Do you want to install into home dir? Prefix is ~/local or ~/opt
If you are in a heterogenous environment with shared home between lots of architectures etc,
This leads to clean & clear separation of software after a system people poured a lot of thought into. Is it easy to grasp at first sight for someone used to Windows? No. But that is _not_ the priority. Sorry, it's not. People writing code need to learn how the language works. Why shouldn't they learn how to system works?
> Package managers are a complex solution to a problem that need not have existed in the first place, if it was realized that unnecessary choice is deadly dangerous, in the world of large-scale software interoperability.
Yeah, cause grabbing random downloads of .bat, .exe, .msi, .whatnot turned out to be awesome. Especially the integrated updates. Oh, what's that? Everyone is implementing their own system leading to dozens of parallel update mechanisms on a single machine? Now _that_ is efficient! And the programs that don't have an update routine? Simple, just write them bug-free, without holes and a complete feature set in 1.0!
> There does not need to be any choice for where on a file system a given application or a given library should be located.
That is true if you consider every machine to be an island. Unix thrived and continues to thrive cause you can create huge shared environments with almost no work.
> That should be completely determined by the app or library name, version (using a standard versioning scheme), variant (using a standard variant naming scheme), and origin person-or-organization, using a standard organization identifying scheme.
My custom mplayer is in /usr/local/mplayer. My custom git is in /usr/local/git. My custom vim is in /usr/local/vim. I can delete any of those and remove the program, along with all its libraries and whatnot, with one single rm. /usr/local for stuff, again... It's their problem, same as if they did not know how to open() a file.
If devs simply don't know that they should default to
> It goes without saying that there should also be a standard globally unique URI for such libraries and apps (including the unique name, version, variant, origin identification).
No. No. No. This breaks any and all assumptions about being able to install different versions of stuff for different reasons. Use prefixes and use LD_PATH, etc.
> So there should be no choice about where on the internet to get it (except for the choice involved in a standard mirroring URI scheme), and
no choice about where to put it.
Maybe you are too young to have seen this yourself, but after a few years, most URLs are dead. With gittorrent, ideally with a DHT sprinkled on top, this might change in the longer run, but what if the next VCS that whoops git's ass comes along? Static information on the internet is mostly a myth. (Also, git would need to get rid of SHA1 for fully automated code distribution, imo)
> With this discipline, obviously needed in today's universe of code, all such package management, as well as dependency acquisition and installation, could be managed by a single unified and incredibly simple automated package manager; call it the
try to run cde with its full path, e.g.: ./cde ../lame
it did the trick for me
He isn't the first doing this. There was a bachelor project at my university last year that does the same thing. I thought it was too little research for the final work as we already had seminars showing how to find dependencies on solaris, linux and BSD. I wonder how often we develop the same software suites just because we don't see that something is already done because of the massive amount of published papers.
One problem I've had -- many times! is having a system level shared library update break a mission-critical userspace applications. And frankly, the vendor probably can't test updates as well as it tests the original distro-releases -- instead, it has to trust the library-authors. Who make mistakes from time to time.
Another problem I have as an author of environmental modeling software is in distributing user-space applications to a broad user community, across several dozen distributions/versions, to a user community that frequently does not have root access to the servers they're running on. And I don't have access to all those kinds of systems, obviously.
There are two ostensible solutions: to distribute statically linked binaries, which your ideology has made very difficult, or to distribute source and have the user community compile and link. And then they have the trouble of finding the libraries, and we still have 'dependency hell." Unless I make them "compile the whole universe" -- which has become common practice in the environmental community. Thanks to your self-centered ideology, damn it!
"My opinions are my own, and I've got *lots* of them!"
http://www.appzero.com/
Disclaimer: I worked there previously when they were called Trigence and actually worked on something similar to the above.
This is somewhat similar to portions of AppZero's Linux/Solaris product. It has a 'learn mode' where an application's dependencies can be discovered and extracted to a 'capsule' allowing it to be migrated. To do something like this, you just need to monitor system calls (specifically things like 'open()') with something like strace to see which files are touched and copy them somewhere else. Then to run it, you can do things like fudging up an LD_LIBRARY_PATH or LD_PRELOAD to make sure you get the right libraries first. If you pick a good subset of library calls to override with LD_PRELOAD, you can do a pretty good job of sandboxing the output of the app too. e.g., intercept open(), prepend your path, call the real open(), profit!!.
The AppZero product is quite a bit more advanced in that applications can be sandboxed from the OS, migrated between different kernel versions and even given their own network identity. It's a little more heavy in that it requires a kernel module to intercept the system calls directly but it's necessary in order to completely separate the app and migrate between different kernel versions. Makes it easier to do some of the tricky things like network identites and makes it possible to do some things like accounting for subtle differences between kernel versions.
Ultimately, it's a decision between intercepting kernel calls or intercepting library calls. Both have pros and cons and a perfect solution will usually have some combination of both. As an example, there was one crusty old Solaris app that was depending on a changed behaviour of opendir/readdir. Since those have no system call equivalent (getdents is the underlying syscall), the problem had to be solved with an LD_PRELOAD override. It's pretty interesting stuff.
Generally linux distributions follow a fairly standard naming/location convention for files, most of the variations exist in specialised linux distributions (eg android) where there is good reason for the differences. /usr/local.
Most software also allows you to choose where to install it at compile time, although the default will usually be
Usually (even on windows) programs determine paths during runtime instead of compile time to allow more flexibility.
Do you see any reason why setting installation location at compile time is any better?
That would be an excellent point if you could show me where I used the word formal in my post. A peer review doesn't have to be formal to be effective, or haven't you heard of the Linux kernel? Perhaps you have never heard of Linus' Law? (Actually coined by Eric S. Raymond)
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Since the focus of CDE seems to be to share scripts used by researchers that often have unusual dependencies, perhaps CDE should just collect dependency info and URLs for non-mainstream libraries. Then the CDE wrapper script could query for the deps and suggest what is needed and offer a URL for download. Not as end-user friendly, but that would avoid the licensing issues.
"I believe in Karma. That means I can do bad things to people all day long and I assume they deserve it." : Dogbert
But portability is already easy.... a liveCD or USB stick is about as portable as you can get.
For justice, we must go to Don Corleone
How is this any different than the PBI package system PCBSD (pcbsd.org) has been using for years? (That is other than it's for linux.)
If I plug in a liveCD or USB stick, and I don't want to reboot the machine, can't I just chroot to it and then run anything that's on there? X11 won't like it though, will it? I vaguely remember doing the reverse to run lilo once or twice.
For justice, we must go to Don Corleone
Isn't this going against the whole distro idea? Instead of having tree different versions of the same lib, the distro's package manager can actually figure out the components missing for the package you are trying to install. If it the package doesn't exist for your distro, just make one.
The only case where I can see this as useful is to install packages across different distros, but even then, if we adapt this model, we will have incredible bloat in permanent storage *AND* in memory if you run programs that all depend on different version of the same lib.
I have a suspicion that it's the latter
I concur. Though it might be a long and overly complicated joke because of this:
Unnecessary choice is (the root of all ;) evil
"People don't want to learn linux" hasn't been a valid excuse since '03.
Download a statically compiled .tar.gz from the site that has all of it's own dependencies
Chromium does this, and the package manager ends up unable to package security holes that are eventually discovered in the dependencies. That's why Chromium isn't in Fedora.
Download source, './configure' 'make' 'make install' (and before you say 'this is beyond the end users capabilities, they have to go through more shit with installshield
Installing an MSI or InstallShield package doesn't require the use of a keyboard. The command line does, unless you use an on-screen keyboard. What "shit" are you talking about, other than simply presenting a wizard listing things that correspond to all the common options for ./configure? Most "shit" can be bypassed by just pressing Enter to choose the most common configuration. The answer to that, of course, is including a wizard script ./menuconfig that uses Xdialog or something.
But this leaves the problem of not having a compiler and the right -dev packages installed. Should the wizard try to detect whether it's running on Ubuntu or Fedora and run the distribution's counterpart to sudo apt-get install build-essential some-library some-other-library first? And should it come with full forked copies of the source code to these libraries, with all their (as of yet unknown) security holes that the package manager can't patch?
Chromium does this, and the package manager ends up unable to package security holes that are eventually discovered in the dependencies. That's why Chromium isn't in Fedora.
Yes I know this, and this is precisely why it is a bad idea to do it like windows does. But you people would prefer this to having to rely on central repos for packages (well if not I don't see why you'd be bitching about the way repos work)
The command line does, unless you use an on-screen keyboard. What "shit" are you talking about, other than simply presenting a wizard listing things that correspond to all the common options for ./configure?
How many people actually change their configure parameters that are normal users? configure auto-detects what optional packages are installed and compilees accordingly. I have what two constant commands to type in as opposed to several next and tickbox dialogs which can vary depending on program. Consistent constant shorter cli is easier in this regard.
But this leaves the problem of not having a compiler and the right -dev packages installed. Should the wizard try to detect whether it's running on Ubuntu or Fedora and run the distribution's counterpart to sudo apt-get install build-essential some-library some-other-library first? And should it come with full forked copies of the source code to these libraries, with all their (as of yet unknown) security holes that the package manager can't patch?
No it shouldn't try this, because for end users it would simply be easier to deploy the .tar.gz static compiled thing as per earlier. Compiling if it fails should only be a nice option. Also a large problem I find with things like this are distros like ubuntu that only distribute a minimum of packages on a cd iso, a full fedora dvd install has far more packages and you'll almost never run into packages missing. With the install still coming under half the size of what vista is.
for end users it would simply be easier to deploy the .tar.gz static compiled thing as per earlier.
A binary that depends on nothing but Linux syscalls and X11 protocol, with static glibc, libstdc++, Xlib, GTK+, and the like, would be huge. I could see doing this if you are distributing your app on CD but not easily if you are distributing your app online. And because it would have to include its own GTK+ themes for people who happen to use a distro built on Qt or FLTK or something and don't have any GTK+ themes installed, it wouldn't be styled to fit in with the rest of the distribution even if it does use GTK+, and it would stick out like a sore thumb as much as the Windows version of GIMP does.
Also a large problem I find with things like this are distros like ubuntu that only distribute a minimum of packages on a cd iso, a full fedora dvd install has far more packages and you'll almost never run into packages missing.
On an Internet connection with a 5 GB per month transfer allowance, downloading a CD image takes 14% of your monthly allowance, while downloading a DVD image takes almost all of it. Ubuntu comes on a CD instead of a DVD because Canonical wants it to be popular even in countries where home Internet connections are rate-limited in such a manner.
If I wanted to, I could link every single library I'm using statically. Then I'd have no dependencies and be able to run a described without any extra help. I do that now with libraries that are non-standard or changing.
The problem is if I statically linked to every library, I'd have a 200 MB executable. If everyone did that, it would be a problem.
So, does it run under itself? Can you do 'cde cde echo hello'?
-- Ed Avis ed@membled.com
Whoever moderated the parent as flamebait is abusing the moderation system to promote their opinion on a point of valid relevant technical debate. Immature.
Where are we going and why are we in a handbasket?
You do realize I assume that you are calling the python library distribution and installation system (cheeseshop, setup_tools etc.) fucktarded. Because it works on pretty much precisely the principles I advocated, albeit at a slightly smaller scale.
Where are we going and why are we in a handbasket?
"The OSX method of program bundles avoids dependency problems, but introduces the inefficiency of reducing code sharing,"
Uh. No. A method of program bundles which includes the concepts of standard name-and-version-based location of code actually increases the ease (and decreases the complexity) of code sharing. OSX may not support this, but they are only part way along the road I suggest.
Where are we going and why are we in a handbasket?
For large-scale library interoperability and code-sharing without the need for a complex package manager.
Where are we going and why are we in a handbasket?
There's very little about python that I wouldn't call fucktarded given the opportunity. But that it has its own library distribution system that's at odds with OS based package management systems pretty much negates any point you were trying to make with it.
Violence is like duct tape. If it doesn't solve the problem, you didn't use enough.
[glibc, GTK+, and the like] aren't the kind of libaries you'd be static linking, they've been a pretty safe assumption for at least the last five years.
But how can an application publisher be sure that distributions have 1. those libraries or language interpreters and not their alternatives (e.g. a Qt-centric distro not including GTK+) and 2. new enough versions of said libraries or language interpreters (e.g. PHP 5.3 not 5.0), without having the ./install shell script inside the tarball try to figure out what distro you're on and run either apt-get or yum to ensure it?
There is something very elegant about having almost all the software you will ever need in one nice disc, and almost never having to install things for dependencies.
The collection of all packaged software and all language packs for one of the major desktop distros (e.g. Ubuntu, Debian testing, Fedora) would take far more than 4 GB. Therefore a full repository would span more than 1 DVD.
silly 700mb isos
After a bit more research, I just discovered that Ubuntu also comes on DVD according to this page. But a lot of end users have the CD version.
An example would be like comparing writing software for windows 95 vs windows 7
Or more realistic in this case, Windows XP SP2/3 vs. Windows 7. We can assume that SP2/3 is analogous to RHEL/CentOS, with fairly obsolete software that has had security fixes backported, and Windows 7 is like a fairly recent Ubuntu or Fedora.
As for whether compatible libs are installed this is solved by the way shared objects are handled, those with compatible api's and functionality tend to have the same name, those that don't get a new revision
And when the revision that a given app requires is newer than the version that was current when some popular distro froze, that causes trouble. For example, a lot of dedicated servers leased from providers such as Go Daddy will be running 8.04 for quite a while, and depending on PHP 5.3 would exclude 8.04 until this bug is fixed.
go to the main ubuntu page, click 'download ubuntu' do you see any mention of a dvd at all?
Not directly, but if you're interested in CDs, the first official dealer that I looked at also offers a DVD. From ubuntu.com, I click "Try Ubuntu today", scroll down and click "CDs", scroll down to official dealers in the United States, click On Disk, and click Ubuntu 10.10 CD or DVD on the front page.
Any distro that is limited to 700mb install iso is gimped by default, and with my usual setup would require gigabytes of yum/apt-get downloads.
Every user is different and wants to do different things with a PC. These usually need different packages: my GB of extra stuff differs from your GB of extra stuff.
http://nixos.org
For what it's worth (though a little off-topic), the techniques and technologies we use at PortableApps.com are compatible with Wine and most of our apps will run without issue on all the major distros with a recent Wine version installed. We even do specific checks for Wine to work around a couple bugs (which have been reported) in our menu and utilities.
Portable versions of Firefox, GIMP, LibreOffice, etc
hopefully it'll get to the point where installing software on Linux will be as easy as on WIndows and OSX.
I think you might have that the wrong way around. Either that or it was sarcasm and my detector is malfunctioning.
Comment removed based on user account deletion
Comment removed based on user account deletion
Comment removed based on user account deletion
Comment removed based on user account deletion
Comment removed based on user account deletion
Comment removed based on user account deletion