Slashdot Mirror


CDE — Making Linux Portability Easy

ihaque writes "A Stanford researcher, Philip Guo, has developed a tool called CDE to automatically package up a Linux program and all its dependencies (including system-level libraries, fonts, etc!) so that it can be run out of the box on another Linux machine without a lot of complicated work setting up libraries and program versions or dealing with dependency version hell. He's got binaries, source code, and a screencast up. Looks to be really useful for large cluster/cloud deployments as well as program sharing. Says Guo, 'CDE is a tool that automatically packages up the Code, Data, and Environment involved in running any Linux command so that it can execute identically on another computer without any installation or configuration. The only requirement is that the other computer have the same hardware architecture (e.g., x86) and major kernel version (e.g., 2.6.X) as yours. CDE allows you to easily run programs without the dependency hell that inevitably occurs when attempting to install software or libraries. You can use CDE to allow your colleagues to reproduce and build upon your computational experiments, to quickly deploy prototype software to a compute cluster, and to submit executable bug reports.'"

73 of 385 comments (clear)

  1. Isn't that three-letter acronym taken? by Anonymous Coward · · Score: 5, Informative

    CDE will always mean Common Desktop Environment to me.

    1. Re:Isn't that three-letter acronym taken? by Jeremiah+Cornelius · · Score: 2, Interesting

      Me too.

      Common to Sun and HP. :-)

      I guess Ultrix, too.

      Regarding this development - it's really what NeXT and later Mac OSX packages do. In the Windows world they have Thinapp and MS's App-V.

      --
      "Flyin' in just a sweet place,
      Never been known to fail..."
    2. Re:Isn't that three-letter acronym taken? by maestroX · · Score: 2, Insightful

      he could always use tar

    3. Re:Isn't that three-letter acronym taken? by h4rr4r · · Score: 4, Insightful

      That method guarantees security problems. Applications and their dependencies should be managed by a proper package management system.

    4. Re:Isn't that three-letter acronym taken? by Haeleth · · Score: 4, Insightful

      I am still waiting for Gnome or KDE to catch up with the efficiency and usability of these older environments.

      KDE is getting closer now that it's possible for the desktop menu to present a list of applications rather than a handful of useless wallpaper-changing commands, but both major environments seem to be stuck on the stupid Windows 95-derived taskbar paradigm. Give me spatial management of running applications dammit! I want to develop muscle memory, not scan slowly across a list of tiny icons that are never in the same place twice.

    5. Re:Isn't that three-letter acronym taken? by countertrolling · · Score: 4, Insightful

      I prefer to avoid the disagreements over what is a "proper package management system". In fact with each program in its own "sandbox", protected from each other, I see better security.

      --
      For justice, we must go to Don Corleone
    6. Re:Isn't that three-letter acronym taken? by h4rr4r · · Score: 3, Insightful

      It is also sure to piss off users who now have to have another Documents directory for each application.

      Else my bad application could edit a document that another application that relies on an old outdated insecure library uses.

      I am starting to think you are not thinking this through.

    7. Re:Isn't that three-letter acronym taken? by Waffle+Iron · · Score: 5, Funny

      CDE will always mean Common Desktop Environment to me.

      I only used CDE briefly, but I remember that it was like a combination of the sheer visual elegance of Tk's widgets with lush the color scheme of a bordello.

    8. Re:Isn't that three-letter acronym taken? by Greyfox · · Score: 2, Funny

      Yeah, and I was wondering why anyone else would even want to take that acronym and the memories of revulsion it evokes in those of us who were forced to use it for any length of time. CDE is right up there with SCO for how quickly it makes me recoil in horror, evoking memories of clunky motif controls and single-threaded inconsistent desktop environment. If you long for the halcyon days of Windows 3.0, give CDE a try, it's just what you're looking for!

      --

      I'm trying to teach myself to set people on fire with my mind... Is it hot in here?

    9. Re:Isn't that three-letter acronym taken? by h4rr4r · · Score: 2, Insightful

      File extensions mean something?

      Wow, that takes me back. That is another big windows flaw you bring up there.

    10. Re:Isn't that three-letter acronym taken? by countertrolling · · Score: 2, Insightful

      In addition to the AC's very valid point below. I might have an app requires a previous version of an installed library. Remember, the issue is portability. You can't have it any other way. Copying and running a program to any machine should be just as simple as doing the same with any text document.

      --
      For justice, we must go to Don Corleone
    11. Re:Isn't that three-letter acronym taken? by countertrolling · · Score: 2, Insightful

      If a program needs its own own BSD style jail to move from one machine to another, then perfect. I'm not worried about space. I'll probably run the program from the removable drive itself, so I don't even need to copy it over. Hmmm.. kinda like having an old Atari game cartridge.

      Convenience, reliability, and simplicity are the goal here.

      --
      For justice, we must go to Don Corleone
    12. Re:Isn't that three-letter acronym taken? by Mr.+Underbridge · · Score: 4, Funny

      I only used CDE briefly, but I remember that it was like a combination of the sheer visual elegance of Tk's widgets with lush the color scheme of a bordello.

      I'm unfamiliar with this 'CDE' but you're compelling me to try it.

    13. Re:Isn't that three-letter acronym taken? by the_womble · · Score: 3, Insightful

      Suppose a security flaw is found in a commonly used library, do you think you will get more timely security updates by

      1) the packager for that library providing an updates package, or,
      2) every single application that uses it providing an updated package

      The CDE site gives two reasons for doing this:

      1) To solve "dependency hell". This is a rare problem these days - except with Skype!
      2) To provide guaranteed reproducible results for researchers. This is a specialist concern, and not necessaries a good thing - it means that results that are the product of a bug in a particular version of a library will be duly reproduced.

    14. Re:Isn't that three-letter acronym taken? by The+Mighty+Buzzard · · Score: 3, Interesting

      No, solving dependency hell is far worse today. Building from source back in tarball only days you had problems if version W of library X was not installed. Building from source today you have that along with issues if your distro of choice does not have version W of library X in the repository along with actually having version W of library X that you built from source installed but your package manager refusing to install things dependent on it because it refuses to acknowledge anything's existence outside its list of installed packages.

      You also have issues like cpan which is currentish vs your distro's package manager which is usually anything but.

      If it weren't for checkinstall I'd seriously consider LFS over package management in situations where I was constantly having to build things from git/cpan/etc... And I'd probably have a huge dent in my desk from where I constantly banged my head instead of the only moderately sized one I have now.

      --
      Violence is like duct tape. If it doesn't solve the problem, you didn't use enough.
    15. Re:Isn't that three-letter acronym taken? by The+Mighty+Buzzard · · Score: 2, Interesting

      Wait, doesn't source code actually require at least a tiny bit of thought? I thought we were trying to be like Windows now. I mean they're still lightyears ahead of us up in Redmond. They have tons of software that installs without the user even having to be aware while even the package manager distros of Linux still require the user to actually authorize it to get the latest and greatest botnet software installed on their box. ( The preceding was a joke. rm ~/.ass/stick before flaming )

      --
      Violence is like duct tape. If it doesn't solve the problem, you didn't use enough.
    16. Re:Isn't that three-letter acronym taken? by Bert64 · · Score: 2, Informative

      Now it depends on your needs, if your constantly building stuff from git/cpan it sounds like your intentionally living on the bleeding edge and/or doing development...
      For most users, and especially production servers having non cutting edge packages managed centrally is far more desirable. Personally when i want cutting edge i use gentoo, so i can have installed a mix of stable core packages with a select few being cutting edge versions while still being package managed. If i want to install something not covered by the package system (which is quite rare with all the gentoo overlays available) i can build my own package quite easily which makes it easy to replace if a newer package comes out later.

      --
      http://spamdecoy.net - free throwaway anonymous email - avoid spam!
    17. Re:Isn't that three-letter acronym taken? by The+Mighty+Buzzard · · Score: 2, Interesting

      It's really more of a problem when one or two packages on a non-bleeding edge system need to be bleeding edge for some specific reason but actually updating those packages would cause backwards compatibility breakage of some other essential something.

      Trust me, it's far from gone even on arch/gentoo/slackware.

      --
      Violence is like duct tape. If it doesn't solve the problem, you didn't use enough.
    18. Re:Isn't that three-letter acronym taken? by m50d · · Score: 2, Insightful

      It's not a flaw. Unlike mimetypes or resource forks or any of the other various non-solutions around, they are preserved over all transfer protocols. They are under the user's control, without getting in the user's way. They're the best way to record filetypes I've seen, I mean it.

      --
      I am trolling
    19. Re:Isn't that three-letter acronym taken? by Svartalf · · Score: 2, Insightful

      The big problem with this is that he's dragging along a complete sandbox (Incl. X11 for X apps...) for each application.

      FUN.

      It avoids dependency hell, yes- but it's vast overkill considering that I can manage it the other way (i.e Going from the 4 yr old distribution forward...)- and have done it with one indie title that I've ported for the studio, and about to do with another one for a different studio.

      Now, this is not to say that it's not an interesting program, or that I won't have uses for the concept he's come up with there- but to call it "easy" portability or a great answer for Linux on things is to clearly misunderstand the real problems. If it's an answer for all of that, you asked the question wrong.

      --
      I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
    20. Re:Isn't that three-letter acronym taken? by Svartalf · · Score: 2, Insightful

      The thing is...typically, mind...most people don't encounter the fun you're describing. It's when you want something SPECIFIC that it typically gets you the way you're describing it. Seriously.

      --
      I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
    21. Re:Isn't that three-letter acronym taken? by countertrolling · · Score: 2, Insightful

      Maybe I'm misunderstanding things, I don't know, but I see two distinct points of view. For the developer, shared libraries are far more convenient for efficiency's sake, but for the end user, they could care less. The convenience of portability outweighs those concerns.

      The big problem with this is that he's dragging along a complete sandbox (Incl. X11 for X apps...) for each application.

      Time to swap out that 80 meg hard drive :-) And bits don't actually weigh that much, so I don't mind schlepping them around with me. Being able to run my program directly off the USB stick sure is nice. But then, what the hell, I could just put a whole "live" system on it, so maybe portability really isn't an issue.

      --
      For justice, we must go to Don Corleone
  2. CDE 2 by ukpyr · · Score: 2, Informative

    I'm just pointing out a major application - that's not so major anymore - Common Desktop Environment uses this acronym :)
    Does sound like a neat tool though!

  3. Next up, the Flash Transport Packager by Anonymous+Freak · · Score: 3, Funny

    To more quickly prepare software for easily installation.....

    --
    Another non-functioning site was "uncertainty.microsoft.com."
    The purpose of that site was not known.
  4. GPL Compliance warning! by Anonymous Coward · · Score: 2, Insightful

    If those libraries are GPL or LGPL, then when you deliver the binary of the library, you must also deliver the source or an offer to deliver the source, and you must also deliver a copy of the (L)GPL, as part of the CDE. Is this done?

  5. Bad idea or worst idea ever? by h4rr4r · · Score: 4, Insightful

    Great, now we can have outdated exploitable libs and every other kind of BS that comes with this. Might as well just statically link everything. Package mangers exist for a reason, use them. Do not bring the errors of Windows to us.

    1. Re:Bad idea or worst idea ever? by 99BottlesOfBeerInMyF · · Score: 2, Informative

      Great, now we can have outdated exploitable libs and every other kind of BS that comes with this. Might as well just statically link everything. Package mangers exist for a reason, use them. Do not bring the errors of Windows to us.

      Or you could just use OpenStep, get dynamic libraries and portable apps. This is a long solved problem.

    2. Re:Bad idea or worst idea ever? by jd · · Score: 4, Interesting

      One method is to have a tool for interrogating the API version and also testing the API against some set of tests that relate to the application being installed. You'd then apply the following:

      • If the API version is within bounds, do not install the library
      • If the API version is outside of bounds but the tests succeed, do not install the library
      • If the API version is greater than the latest supported and the tests fail and a backwards-compatibility library which IS within bounds of the API provided is within the archive, install the backwards-compatibility library
      • If the API version is greater than the latest supported and the tests fail and no backwards-compatibility library is usable, install the supplied library locally to the application, using the package manager, using an alias so there's no name-clash with primary libraries
      • If the API version is less than the minimum supported and the tests fail and the user authorizes an upgrade, use the package manager to upgrade to the supplied library
      • In all other cases, install the supplied library locally to the application, using the package manager, using an alias so there's no name-clash with primary libraries
      • Where the library is installed locally, all information regarding the supplied API must be removed since it's vital it doesn't clash with anything else - however, there must be a maintenance tool for cleaning out such local libraries when they are no longer required

      This should keep redundancy to a minimum. There will be some, since there's nothing in this to collaborate between apps using this method, but it's a start.

      --
      It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
    3. Re:Bad idea or worst idea ever? by grumbel · · Score: 2, Insightful

      Package mangers exist for a reason, use them.

      Except that distribution specific package manager do *nothing at all* to address the problem of distribution independed binary packaging. On top of that package manage are a really lousy solution to the software packaging problem, as they don't actually solve the underlying problem of duplication and incompatibilities, instead they have a single monolithic repository that they declare as the one and only source for software out there. As long as your software is in that repository, in the version you want, you are of course fine, but if you want a newer version, an older one, two different ones at the same time or a piece of software not yet packaged, you are back at square one as the package manager doesn't deal with those at all. You have to completly bypass the package manager, compile stuff from source and install it somewhere where it doesn't conflict with existing stuff. Its pretty much one big ugly hack.

      Now of course CDE isn't exactly a beautiful solution either, its a hack to solve a problem which the distribution builder have preferred to ignore for the last 15 years.

    4. Re:Bad idea or worst idea ever? by Renevith · · Score: 2, Interesting

      This is not intended to be a package manager replacement. In fact, keeping older possibly-buggy versions of libraries around is a good thing in some of the expected use cases, e.g.:

      • Giving other academic researchers your code that will reproduce your results exactly, even if your code was triggering a bug in a specific library version you have
      • A professor distributing a class assignment in executable form, and not having to worry about how many flavors of linux his students are running

      There are many more use cases illustrated on the website, which you obviously did not read or else you wouldn't have compared CDE to a package manager. Almost all the examples he uses are ad-hoc transfers where the CDE package will only be used on a temporary basis, or where the effort required to bundle a package for each OS would be unwarranted, or where the lack of library upgrades is actually an advantage.

  6. copying proprietary software by tommeke100 · · Score: 2, Insightful

    This sounds like an easy way to copy installed proprietary software?

  7. Re:It's About Time by h4rr4r · · Score: 2

    This does not make anything easier, it just makes it wrong.

    The ubuntu app center makes things easier without this sort of nasty kludge.

  8. Re:Party like it's 1988 by h4rr4r · · Score: 5, Insightful

    Wow, static linking, did anybody for even a second think it is kinda weird to have the same lib on the machine over and over and in every old exploitable version you can find?

  9. Re:It's About Time by couchslug · · Score: 4, Interesting

    Making applications portable is handy for doing things like running them from a USB stick. It also makes backup much more convenient.

    Copy the program and its data in one shot, carry it with you, and use anywhere.

    Windows apps are ahead of the game on this one:

    http://portableapps.com/

    --
    "This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
  10. Re:Party like it's 1988 by countertrolling · · Score: 2, Insightful

    That would all be nice if not for each program replacing shared libraries with its own and breaking the other programs. A program should not disturb the system or mess with its libraries. I prefer it remains as isolated as possible, where it can do the least damage.

    --
    For justice, we must go to Don Corleone
  11. Re:Party like it's 1988 by avaik6 · · Score: 2, Informative

    I would LOVE to have mod points and vote this up... I always loved the fact that the applications use the existing blocks on the system to do their magic, and by sharing the blocks you can have more apps in less space than other platform$. But with this... if you install every f*cking package with this.. you will end up having 12 different glibc versions (hey, maybe 12 times the same glibc version, each for each app that requires it), and 81725 times each library.. I think it's a _GREAT_ tool to perform a fast-and-no-brains-available deployment of some in-development app, or to debug some weird shit. But please, do not pretend to install all my software like this.. I don't want gedit to weight 100mb.

  12. Re:Party like it's 1988 by Haeleth · · Score: 4, Insightful

    That would all be nice if not for each program replacing shared libraries with its own and breaking the other programs.

    Do, please, show me just one widely-used program that does this on a recent UNIX or Unix-like platform.

    A program should not disturb the system or mess with its libraries.

    Right. That's why you should put programs you install under /usr/local, not straight under /usr. Or of course many programs like to be installed in their own self-contained directories under /opt, which is, er, basically exactly what you're asking for and has been common practice for decades.

  13. Re:Limitations by h4rr4r · · Score: 2, Insightful

    You could just use SElinux, which would already let you do that.

    This looks like a solution looking for a problem.

  14. Re:I really like where this is going. by Haeleth · · Score: 2, Interesting

    Installing software on linux is easier than on windows or osx.

    For software that is in the package manager, yes.

    Where something like this "CDE" might be handy is for software that is not in the package manager. Suppose you've written a program that is only of interest to a handful of users. There's no way it's going to find package maintainers for every major distro, and your users might not be happy building from source code.

    There are also classes of software that are not allowed in the main repositories for some major distros like Fedora and Debian. For example, the authors of indie games might want to let Linux users play without making the whole game open source. Even if they open-sourced the engine, some distros will not permit it in the repos if its only use is to access non-free data.

    Basically, "package once, run everywhere" is very appealing to a certain class of software distributor.

    What I don't see is what CDE offers over any of the dozens of existing autonomous packagers. Or why they chose to confuse people by using the same name as the standard UNIX desktop environment.

  15. Re:It's About Time by CannonballHead · · Score: 2, Insightful

    ... so isn't this basically just a way to gather all the appropriate dependencies and put them all into one spot?

    Hm. I guess this is a way to do it without installing. Reading comprehension fail...

    Still, I can see how it could be useful in some situations, just like having certain programs that don't require installation on Windows can be helpful.

  16. My extremely similar tool: dynpk by xgoat · · Score: 2, Interesting

    I very recently published a tool that performs a similar task. dynpk (my tool) bundles programs up with packages from your system, then wraps them with some other stuff to create a bundle that essentially allows you to run a Fedora program on a RHEL machine (and probably Ubuntu or Debian, but this is outside my needs...).

    Recompiling loads of libs for RHEL isn't fun or particularly maintainable. Therefore, use the ones from Fedora!

  17. Re:cross distribution compatibility by icebraining · · Score: 4, Insightful

    Firstly, you don't need 5000, you need 4 or 5 for the most used distros. Ubuntu, Fedora, OpenSuse, Debian and Red Hat. Let the others figure it out from a tar file.
    And if a company like Skype can produce those packages, so can e.g. Adobe.

    Secondly, that already exists.

  18. Re:Party like it's 1988 by Kjella · · Score: 4, Insightful

    For packages provided by the distro, it makes sense to have them all use their complex dependency tree. For installing some other version side by side, this sounds like a great tool. The problem with dependencies is that often a pebble turns into an avalanche by the time you're done. If you want the new version of *one* KDE app, it can drag pretty much the whole of KDE and every library they in turn depend on with it in an upgrade. I've had that happen and ended at 450MB to download and install, and that would pull almost all packages out of LTS support.

    From the user's point of view it's completely illogical to upgrade the whole system just because you want a new feature in amaroK 2.4 while your distro only packages 2.3, you expect one application to install or upgrade independently of any other application. That does not happen with Linux. It is not just about new library versions, via dependencies you pull in unwanted version upgrades. As for security I'd rather have one potentially insecure package on my system than to pull most packages out of support, it probably open ups more vulnerabilities than it prevents.

    I wouldn't want to run a dozen applications like that. But if it's one or two? I got no problem taking the extra overhead of a bit more memory use. And honestly, a lot of software I use isn't in contact with the "outside world" as such. Even if there is an exploit in a library, I'd never open any file crafted to exploit it. Obviously it is good in general to patch stuff, but it's not always that critical...

    --
    Live today, because you never know what tomorrow brings
  19. Re:I really like where this is going. by Junta · · Score: 5, Insightful

    Dear god no.

    I do not want to execute installshield or any similar crap/wizard for every little thing I install.

    I do not want to have a system tray/task manager full of two dozen vendor's update checker processes, each individually bugging me about how I'm running WidgetFoo 1.8.1.20.1.3, and it is critically important that I execute WidgetFoo's custom one-off graphical update wizard with 3 or 4 pages to click through to get to 1.8.1.20.1.4. Then rinse and repeat once per app instead of knocking them out in one shot/dialog/icon/process.

    I do not want each application to bundle their ancient ass directx library or ancient library from visual studio or any other similar crap.

    Windows installs were not historically 'easy' due to any effort on MS's part (installshield and friends made an entire business out of covering for MS' lack of help, even as MSI matured into a usable solution). Linux (specifically Debian) really got this right first. Apple recognized that model and made it a great success on the iPhone, setting the tone for all of modern mobile devices. Debian did it right first and never gets the credit.

    --
    XML is like violence. If it doesn't solve the problem, use more.
  20. Re:Party like it's 1988 by countertrolling · · Score: 3, Insightful

    That's why you should put programs you install under /usr/local, not straight under /usr.

    Not the issue. That's a given. It's when I suddenly find out I don't have some bizarre version of gtk, or ncurses (great name, because that's what I'm doing when I find it missing), and I'm suddenly without internet, it gets a bit tense. I prefer the portability over raw efficiency. It is far and away one of the best things about a Mac. I can take something as bloated as MS Office or Photoshop straight from one machine to the next.

    --
    For justice, we must go to Don Corleone
  21. I almost started screaming by Chibi+Merrow · · Score: 3, Funny

    No! I'm not going back! I'M NOT going BACK! MOTIF IS DEAD TO ME!

    --
    Maxim: People cannot follow directions.
    Increases in truth directly with the length of time spent explaining them
  22. Re:Party like it's 1988 by Bill,+Shooter+of+Bul · · Score: 2, Funny

    You mean like the registry ....

    [ducks outa the way]

    --
    Well.. maybe. Or Maybe not. But Definitely not sort of.
  23. Re:I really like where this is going. by bmo · · Score: 4, Insightful

    Where something like this "CDE" might be handy is for software that is not in the package manager. Suppose you've written a program that is only of interest to a handful of users. There's no way it's going to find package maintainers for every major distro, and your users might not be happy building from source code

    So do the packaging yourself. It's not hard. And when you're done, you have something sitting in the RPM or DEB database with all the others so you can keep track of it.

    There are also classes of software that are not allowed in the main repositories for some major distros like Fedora and Debian. For example, the authors of indie games might want to let Linux users play without making the whole game open source. Even if they open-sourced the engine, some distros will not permit it in the repos if its only use is to access non-free data.

    So set up your own ppa (or rpm equivalent) repository. Your customers can add the repository to their list and then keep track of the package. You seem to be under the impression that repositories are only for "approved" software or that package managers can only handle a small number of entries. I have over 150 entries in /etc/apt/sources.list. Adding another one is no big deal. You also seem to think that licensing issues affect what you can put in a repository. It doesn't matter if you have your own repo. You could put commercial software in there, like Sun/Oracle with their VirtualBox.

    Package management and repositories as they exist in the Linux world are better ways of handling the distribution of software both free and commercial than anything else I've seen on any platform.

    This "CDE" doesn't solve any problems, but introduces its own "dll hell"

    --
    BMO

  24. Re:Party like it's 1988 by tepples · · Score: 2, Interesting

    No the program should just use the libraries the system has.

    Then the package manager will fail to install the program because the program requires a later version of a given library than is available in the long-term-supported distribution that the end user is running. For example, PHP 5.3 made PHP far less of a pain in the ass, but Ubuntu 8.04 LTS (the version found on Go Daddy dedicated servers, for example) still has 5.0.something or other.

  25. Re:It's About Time by neocephas · · Score: 5, Informative

    I think most people here are not understanding the target audience for this tool (hint: it's not for your typical linux environment). It's not about package management or having a universal installer... it's about being able to run your application in a different environment where you don't have admin rights.

    In a lot of university clusters or compute grids researchers have access to a large collection of compute nodes, but they usually don't have any rights to those machines. In fact, most of the time the programs are ran in a sandbox and have a restrictive environment. To run their codes reliably, researchers often have to perform some sort of static linking or package up all of the dependencies with the executable. apt-get or yum are not options in these environments... you may not even be able to ssh into them. Ideally, you could ask the system administrator that controls the cluster to install certain packages, but again, this is not always possible particularly if the researcher requires a niche package used in their domain.

    Moreover, the cluster may be composed of heterogenous set of machines with different versions of Linux. Package management does not help you here. The only way to reliably execute your programs on such a heterogenous cluster is to statically link or include your dependencies. If you are wondering who would use such a maddening environment where you have no admin rights... google Condor, OpenScienceGrid and Globus. This is how a lot of research computation is done.

    Of course, the hot new thing is virtual machines and clouds... but firing up a VM each time you want to run an application is very heavyweight... especially if your applications has a short run-time.

    TL;DR: this isn't for your typical ubuntu or fedora install; it's for scientific research that is done on restrictive computing clusters and grids.

    As a side note, I made and use a much cruder tool http://bitbucket.org/pbui/starch/ that packages everything up (executables, libraries, and data) in a self-extracting tarball which can be executed on remote hosts. It's not as slick as CDE, but it's been used with success by various research groups that I collaborate with.

  26. Four GNU/Linux package mgmt problems by tepples · · Score: 2, Insightful

    apt-get install foobar

    This brings four problems that I can think of:

    • foobar is not available in the predefined repositories because it does not match the political views of the operating system's publisher. Some GNU/Linux distributions such as Fedora package only free software, and not all classes of application are amenable to distribution as free software.
    • foobar is not available in the predefined repositories because nobody has packaged it for your distribution. This could be because the maintainer hasn't yet learned how to package, because the maintainer has never had a chance to travel to get his PGP key signed by more established developers, or because the maintainer uses a different distribution from you.
    • foobar or a library on which it depends is in the "community-maintained" part of the repository, not the "core" part, and a change to an underlying system library broke the application. This is the case with, for example, games that use the Allegro library after Ubuntu started using PulseAudio, which doesn't support the audio sample format that Allegro has used for years.
    • The feature that you seek was not introduced until a later version of foobar than the version included in the predefined repositories. The program hasn't changed since the package freeze over a year ago except for backported fixes to security defects.
  27. Re:It's About Time by visualight · · Score: 2, Insightful

    I've spent time on work like this myself, even used code like this in production...it's really really hard to be sure you've got everything and don't have unnecessary deps, particularly when you've got scripts (that you didn't write) that call out scripts that call out scripts.

    I know a few others that have invested time in this also, when you spend much time on a cluster that you don't own eventually you at least think about doing this.

    --
    Samsung took back my unlocked bootloader because Google wants me to rent movies. They're both evil.
  28. Good for scientists; not for end-users. by Hach-Que · · Score: 2, Insightful

    While I can see this being useful for scientists, it's probably not going to be so useful in terms of an end-user environment (not that it couldn't be used in one). For starters, this is really only good for command-line packages as GUI applications, the one that end-users are most likely to use, would end up including most of the X and UI libraries during the CDE detection, even if the files on the target computer are compatible with the machine.

    The other issue with having different projects for different distribution purposes (Klik for applications, CDE for scripts, etc.) is that it induces fragmentation; the very thing these projects are trying to get rid of by no longer relying on the package management system.

    Disclaimer: I work on AppTools (http://code.google.com/p/apptools-dist) and therefore I'm biased :)

  29. Re:It's About Time by ozmanjusri · · Score: 4, Insightful
    I'm not sure why you're getting "Troll" mods for this.
    1. It does allow you to have outdated and unsecure software where it is likely to be used.
    2. Windows does lack proper package management.
    3. There are better ways to achieve the goal.
    4. It isn't a good idea to turn Linux into Windows. In fact, most mainstream OSs are switching to package managers.

    Guo comes from a Windows background (He interned at Microsoft last year), so it's understandable why he might have a Window perspective. That doesn't make it good for Linux to adopt that mindset.

    --
    "I've got more toys than Teruhisa Kitahara."
  30. Re:It's About Time by ischorr · · Score: 2, Insightful

    Why is this a kludge?

    App portability and dependency problems has been one of the Achilles Heels of Linux since, well, forever. We laughed at Windows for DLL-hell, and if anything package managers seem like the ugly kludge way to resolve it to me. I wonder how many tens or hundreds of thousands of man-hours have been lost dealing with these sorts of issues. It's by far the #1 thing that's prevented me from using Linux for those purposes, and I'd REALLY like to use Linux (though there are others). Hell, it's the main thing that's kept me from taking Linux seriously outside the server room. Particularly since people really don't seem to get why this is a problem.

    - I should be able to install an application QUICKLY and easily. There's no reason why "installation" should be more complicated than "copying/extracting the binaries to wherever I want them to go".
    - I should not be dependent on some third party to get around to porting each version of software to my flavor of Linux. When a new version of Wireshark or VLC or whatever software comes out, I should be able to install it *quickly and simply* without waiting on package maintainers to get around to it (even if some are very responsive)
    - Along with the above, I shouldn't be in a state where I can no longer easily install applications because I'm using a somewhat older distribution (and packages are no longer being maintained for that version)
    - Although the option should be there, it should never, ever, ever, ever, ever, ever, ever, EVER be considered "acceptable" to expect users to compile an application to install it (and all the potential headaches of getting that to work, including hacking make files, dealing with dependencies, patching the software, actually doing C++ debugging, etc).

    Balloon up my applications with static libraries. Please. The trade-off in system administration headache would pay for itself 100s of times over.

  31. Re:It's About Time by ischorr · · Score: 4, Interesting

    That's probably another use, but I really don't think that's the main place where it'd be useful. I DREAM of being able to just download an application archive, extract it *anywhere I want*, and just run it. Just use it, without having to worry. Any application - not the apps (and versions) that some distribution maintainer has gotten around to porting to my flavor.

  32. Linux is not Windows by Zero__Kelvin · · Score: 2, Informative

    "Then the package manager will fail to install the program because the program requires a later version of a given library than is available in the long-term-supported distribution that the end user is running."

    No, it won't. Unlike Windows, Linux is perfectly happy having multiple versions of any given library installed and available: Fortunately, on Unix-like systems (including Linux) you can have multiple versions of a library loaded at the same time, so while there is some disk space loss, users can still run ``old'' programs needing old libraries.

    --
    Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
  33. Re:Party like it's 1988 by Zero__Kelvin · · Score: 2, Interesting

    "I can take something as bloated as MS Office or Photoshop straight from one machine to the next."

    I can pirate software too, but I prefer to use superior free software.

    --
    Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
  34. Re:I really like where this is going. by Pentium100 · · Score: 2, Insightful

    I do not want to compile from source every time I want to install some app on an older or uncommon system.

    I do not want to have a system tray/task manager full of two dozen vendor's update checker processes, each individually bugging me about how I'm running WidgetFoo 1.8.1.20.1.3

    WidgetFoo could just as well check for updates itself and not have a system tray app runnin for that. Example - Firefox - when it is running it checks for updates, but does not leave a system tray icom to do that when it's not running.

    I do not want each application to bundle their ancient ass directx library or ancient library from visual studio or any other similar crap.

    Then maybe you should ask the out-of-business (or not) game or app developer to update its product to make it run on a Windows version that was released 8 years after they released the product?

    Linux (specifically Debian) really got this right first.

    Yes, until an app somehow clashes with the political view of Debian creators.

  35. Re:I really like where this is going. by bmo · · Score: 3, Informative

    You're not even replying to anything I said, except to post an ill-informed rant.

    There are *three* package types. And what these do is far more than anything seen in the Windows universe.

    Windows doesn't even *have* a packaging scheme. Sure, there are installers, but that's all they do. There is no dependency resolution. There isn't any updating except manually or if the individual program checks by itself. Things really haven't come very far from the self-extracting .zip file method of installing. What we've gotten over the years is a graphical front-end to an archive extraction and a script to tweak the registry, and a script to uninstall. That's it. It's arcane. It's stone knives and bear skins.

    At least Microsoft dispensed with the bogus "add and remove software" and renamed it "remove software" in the Control Panel. In XP if you came from Debian, you'd expect the "add and remove software" would be where you'd find the Windows equivalent of Synaptic. Sorry, chuckles, it's not. Now in Windows 7 it's finally fixed. It's a small amount of honesty, but it's still better than before.

    And besides, do you know how long it takes to make a package for Linux after compiling? Literally less than 5 minutes, and it can be automated. Doing packaging for 3 package formats (.deb, .rpm, tar.gz) is not a lot of work at all.

    What is available for software take packaging and distribution for Linux software is light years ahead of anything seen in the Windows universe. The Apple "app store" is the only thing that comes close and even that is clunky compared to a fully functioning Debian or RPM repository.

    Your rant is typical of the Windows user who thinks he knows anything about Linux but doesn't really. I suggest you acquaint yourself with what you're talking about before you open your mouth here again and sound even more silly.

    --
    BMO

  36. Re:I really like where this is going. by ischorr · · Score: 2, Informative

    Why in the world would you need an installer for something like this? I don't understand why 99% of applications are ever distributed that way, except for poor OS design or just bad developer habits.

    OS X (and OpenStep) did it right earlier in the sense that the app is a self-contained unit, well-laid out and hierarchical but a single unit to the user. "Installation" typically involves a single move command or drag-and-drop of a single "file" (bundle), and portability across systems cannot be beat (providing the developer the ability to distribute a single bundle that transparently functions across as many platforms as exist for the OS) . It's a beautifully well-designed piece of work. The choice of usability/ease of distribution versus some amount of bloat is left to the developer, and I think it's a great trade-off.

  37. Re:It's About Time by martin-boundary · · Score: 2, Insightful

    I DREAM of being able to just download an application archive, extract it *anywhere I want*, and just run it.

    Why? No offence, but you dream small. This seems barely one step up from what we've had in the past. Why SHOULD you have to do any of this? A real improvement shouldn't involve any of this mundane stuff. You shouldn't have to go download anything yourself, and you shouldn't have to think about managing where you want to extract things.

    The true goal should be to automate all this and make it transparent. If you want to run a program, that's all that should be needed from you, by the interface. The computer should figure everything out for you and do it, including putting things in your preferred locations if that's what you like. We're half way there with package management systems already. Take a look at Debian/Ubuntu. You choose a package, and you run the program. No manual downloads, extractions, configurations, hunting for dependencies etc. If the community can figure out how to make the packaging literally trivial for developers, then we'll be that much closer to the goal. I'd like to see Linux distributions able to figure out all the details for creating their own packages, just by reading a single URL that the project developer would publish. And I'd like this to be so ROUTINE, that every developer does it.

  38. Re:It's About Time by SanityInAnarchy · · Score: 2, Interesting

    apt-get or yum are not options in these environments...

    Really? Why not?

    Put another way: Rubygems can install to a home directory, and only requires Ruby itself to be in the path. Are you saying that the sandbox environment doesn't allow you to have reliable filesystem access or modify your environment variables? Because that's all it takes.

    I realize apt-get or yum may require system-level access in their current configuration, but if they can be configured per-user, that's a limitation in apt-get or yum, not in the idea of a package manager.

    firing up a VM each time you want to run an application is very heavyweight...

    Not necessarily, especially when VMs can be cloned with COW memory.

    the cluster may be composed of heterogenous set of machines with different versions of Linux.

    Doesn't seem like an insoluble problem, either. If you're just going to statically link anyway, bundling system libraries doesn't seem like that big a stretch. A chroot jail would be a better solution (and I think BSD has a secure version of these), but you can fake that without root anyway.

    The advantage of doing things this way is that you still get the advantages of dynamic linking (lower memory and disk usage for multiple programs installed, easier updates, etc). The only component missing is admin rights, and there's nothing special about admin rights that relates to any of these.

    So, TD;DR: This looks like just another case where the correct solution was staring you in the face, but you went with the easy solution instead. That's not necessarily wrong -- you could actually make a much better case that it's too hard to write a proper package manager for this environment, and there's too small of a target audience to justify the effort, compared to a simpler solution which just Gets It Done.

    But then, why is CDE a big enough deal to be Slashdotted? Presumably it took a significant amount of work...

    --
    Don't thank God, thank a doctor!
  39. Re:Party like it's 1988 by SanityInAnarchy · · Score: 2, Informative

    OS X is routinely first to lose in the "pwn to own" challenge, among others.

    --
    Don't thank God, thank a doctor!
  40. Re:I really like where this is going. by grumbel · · Score: 2, Interesting

    AppBundles are beautiful and easy, but also not flexible enough to handle many cases, which is why quite a few of software on MacOSX comes as .pkg with an installer, instead of as AppBundle. What MacOSX however did right is making the install application a standard thing of the OS, not something that has to be reinvented by every software developer like on Windows (.msi should probably fix that one day).

  41. Re:Use a package manager by The+Mighty+Buzzard · · Score: 2, Insightful

    I can't tell if that should have been +1 Funny or -1 Fucktarded. I have a suspicion that it's the latter but it is 3am and I'm starting to nod a bit, so I'll leave it to someone else to decide.

    --
    Violence is like duct tape. If it doesn't solve the problem, you didn't use enough.
  42. Re:Use a package manager by Bert64 · · Score: 5, Insightful

    Generally linux distributions follow a fairly standard naming/location convention for files, most of the variations exist in specialised linux distributions (eg android) where there is good reason for the differences.
    Most software also allows you to choose where to install it at compile time, although the default will usually be /usr/local.

    A linux system is often far less messy than a windows system for instance, where all kinds of files are under the windows and system32 dirs.

    Package managers are actually a very good solution to many problems, not only do they handle dependencies but they provide a centralised database of installed software, a file integrity database (both on the system - storing checksums of everything, and off system because the checksums corresponding to a given package versions files are known), clean removal of software, a single place and standardised interface for installing software (thus removing the need to download programs from potentially untrustworthy websites - you only have to trust your os vendor, not hundreds of third parties) and most important of all, a centralised update mechanism for applying important security patches to all of your software...

    Other software vendors have chosen different methods to try and resolve the same problems, but most of them are lacking in one way or another, or make different compromises...

    The OSX method of program bundles avoids dependency problems, but introduces the inefficiency of reducing code sharing, this has less impact on closed source software where code is rarely shared anyway, but for open source one of the key advantages of the open development model is reduced by this approach. On the other hand, this method does provide clean removal and makes it easy to have multiple versions of something installed.

    The Windows method is rather chaotic, individual programs are expected to create their own installation and removal programs as well as handle their own update mechanisms, this has resulted in a whole range of software which behaves in different ways, stores files in different places etc... Update mechanisms and uninstall routines are down to the individual application and may not exist at all, or may not work correctly. This has resulted in lots of very poorly behaved software which assumes you are a privileged user and can write to system locations, and subsequently in order to retain compatibility microsoft have been forced to implement all kinds of dirty kludges to make such applications think they are able to write to system dirs when they can't.

    The only potential downside to the linux system, is that application suppliers don't have a fixed list of system libraries which will always be present. Under OSX or Windows you know that a core set of libraries will always be there, and anything else is typically provided by the app (sometimes redundantly), whereas different linux distributions may provide different base libraries.

    --
    http://spamdecoy.net - free throwaway anonymous email - avoid spam!
  43. Complete and utter bullshit! by RichiH · · Score: 5, Interesting

    Sorry if topic sounds a tad personal, but hey...

    > The real problem is that Linux distributions, taken together or individually, presents developers with too many completely unnecessary choices as to where essential library files can be put, and also, there is no standard version naming and locating convention.

    Do you need it to boot? Prefix is /
    Do you need it after boot? Prefix is /usr
    Do you want to install custom stuff that is not handled via the system's default software handling solution? Prefix is /usr/local or /opt
    Do you want to install into home dir? Prefix is ~/local or ~/opt
    If you are in a heterogenous environment with shared home between lots of architectures etc, /import/x86 etc is a good place

    This leads to clean & clear separation of software after a system people poured a lot of thought into. Is it easy to grasp at first sight for someone used to Windows? No. But that is _not_ the priority. Sorry, it's not. People writing code need to learn how the language works. Why shouldn't they learn how to system works?

    > Package managers are a complex solution to a problem that need not have existed in the first place, if it was realized that unnecessary choice is deadly dangerous, in the world of large-scale software interoperability.

    Yeah, cause grabbing random downloads of .bat, .exe, .msi, .whatnot turned out to be awesome. Especially the integrated updates. Oh, what's that? Everyone is implementing their own system leading to dozens of parallel update mechanisms on a single machine? Now _that_ is efficient! And the programs that don't have an update routine? Simple, just write them bug-free, without holes and a complete feature set in 1.0!

    > There does not need to be any choice for where on a file system a given application or a given library should be located.

    That is true if you consider every machine to be an island. Unix thrived and continues to thrive cause you can create huge shared environments with almost no work.

    > That should be completely determined by the app or library name, version (using a standard versioning scheme), variant (using a standard variant naming scheme), and origin person-or-organization, using a standard organization identifying scheme.

    My custom mplayer is in /usr/local/mplayer. My custom git is in /usr/local/git. My custom vim is in /usr/local/vim. I can delete any of those and remove the program, along with all its libraries and whatnot, with one single rm.
    If devs simply don't know that they should default to /usr/local for stuff, again... It's their problem, same as if they did not know how to open() a file.

    > It goes without saying that there should also be a standard globally unique URI for such libraries and apps (including the unique name, version, variant, origin identification).

    No. No. No. This breaks any and all assumptions about being able to install different versions of stuff for different reasons. Use prefixes and use LD_PATH, etc.

    > So there should be no choice about where on the internet to get it (except for the choice involved in a standard mirroring URI scheme), and
    no choice about where to put it.

    Maybe you are too young to have seen this yourself, but after a few years, most URLs are dead. With gittorrent, ideally with a DHT sprinkled on top, this might change in the longer run, but what if the next VCS that whoops git's ass comes along? Static information on the internet is mostly a myth. (Also, git would need to get rid of SHA1 for fully automated code distribution, imo)

    > With this discipline, obviously needed in today's universe of code, all such package management, as well as dependency acquisition and installation, could be managed by a single unified and incredibly simple automated package manager; call it the

  44. Re:It's About Time by Lennie · · Score: 2, Informative

    You can also run different profiles at the same time with -no-remote -P . You can run different versions or the same.

    --
    New things are always on the horizon
  45. Re:It's About Time by multipartmixed · · Score: 2, Informative

    Why dream?

    Buy a Mac.

    Software is most frequently distributed in .dmg (disk image) files. You download the file, maybe gunzip it, then you click on it. That mounts the volume. Then you click on the application. That runs it.

    If you want to turf the .dmg and put the application in your application folder, you just drag it from the dmg to the application folder (or fan) and let go.

    The biggest piece of the puzzle to making this work from a systems POV is the Mach-o linker. The linker understands executable-relative linking. The next biggest piece is the uniformity of the install environment, the common directory structure that goes with applications (like, stuff with the app goes with the app instead of parts in /bin, /etc, /usr/lib, and so on), and the understanding of the directory structure in the file manager (finder).

    Put all these pieces together and you have a really nice cohesive environment for day-to-day use. The best part is, it's still a UNIX box, so you can you pop open a terminal window and do whatever the hell you want.

    It will be a long time since Linux sees this type of paradigm, as it requires deep cooperation the KDE/Gnome folks, development toolchains, systems linker, and a re-invention of the directory layout in the LSB.

    --

    Do daemons dream of electric sleep()?
  46. Re:Party like it's 1988 by Rich0 · · Score: 2, Informative

    From the user's point of view it's completely illogical to upgrade the whole system just because you want a new feature in amaroK 2.4 while your distro only packages 2.3, you expect one application to install or upgrade independently of any other application. That does not happen with Linux.

    Sure it does. It just doesn't happen with Ubuntu, or Debian, or RHEL/Fedora/SUSE/etc.

    If you use a source-based distro then unless the newer version depends on some API exposed by a newer library then it will build just fine and run against the old one.

    It used to be that the preferred upsteam way of distributing packages (outside of a package manager) was as source for exactly this reason. The same tarball worked just fine on sparc/ sgi/ hpux/ linux/ freebsd/ etc.

    I do see the utility of something like this tool, but I'd hate to see it used widely. Besides security there is also wasted memory - if 10 programs are linked against the same library it is only loaded in RAM once. If 10 programs are linked against 10 versions of the library, or even the same version built 10 times (and they load those different versions) then the system can't share their memory.

  47. Re:Party like it's 1988 by Rich0 · · Score: 2, Interesting

    LTS is good for production environments where you have lots of machines to support.

    Suppose you run Ubuntu on 1000 workstations. Each of those runs a variety of programs, not all the same, and some aren't from the PM (they might not be open source, or widely used - they could even be homegrown). Every time there is an upgrade one of these programs could break.

    The idea of LTS is that for the most part everything stays put but you still get security updates. For something like a corporate desktop that is exactly what you want. Why do you think that so many computers are still running XP? Simple, it works and as long as it works nobody wants to pay the small fortune it would take to figure out if anything will break and if so fix it when you upgrade 20,000 PCs running it.

  48. Re:I really like where this is going. by Junta · · Score: 2, Informative

    4) It needs to be open-source for this model to work. Some software isn't. =)

    Not true. Adobe publishes their closed flash plugin via their own proprietary apt and yum repositories. The distros allow arbitrary vendors to hook into the single update management infrastructure with nothing more than the end user's permission.

    --
    XML is like violence. If it doesn't solve the problem, use more.