Ubuntu Developing Its Own Package Format, Installer

An anonymous reader writes "While complementing Debian APT/DPKG, Canonical is now developing their own package format. The new package format has promised highlights of having no dependencies between applications, each package would install to its own directory, root support wouldn't always be required, and overall a more self-contained and easier approach for developers than it stands now for Debian/Ubuntu packages. The primary users of the new packaging system would be those distributing applications built on the Ubuntu Touch/Phone SDK. The initial proof-of-concept package management system is written in Python and uses JSON representation." This quote from the post by Canonical's Colin Watson bears repeating: "We'll continue to use dpkg and apt for building the Ubuntu operating system, syncing with Debian, and so on."

Comments

Good

[MightyMartian]

Good, another reason to avoid Ubuntu like the plague.

Re:Good

[Penguinisto]

Indeed... this sentence:

The new package format has promised highlights of having no dependencies between applications

...tells me there's gonna be a whole shitload of bloat, duplicate binaries, and a performance hit from Hell.

I could be wrong, but...

Re:Good

[buchner.johannes]

Also, this might be the dawn of malware for Linux on the PC.

Re:Good

[Jeremiah Cornelius]

Go open a mac .app sometime. Libraries and resources galore can be found. The Systme libraries and frameworks can be over-ridden. like having ~/Library on a per-app basis.

Re:Good

[Anonymous Coward]

This is going to be the best thing for linux. Most users will enjoy and maybe switch to linux instead of windows. Dependecies that link multiple times to other dependancies get very convoluted. Normal peoples linux finally.

Re:Good

[Jeremiah Cornelius]

"Dunno if there's a way to specify that inside Xcode or not, but for our app we use a build script that includes some code like the following. The code uses Apple's install_name_tool utility to modify the application so that instead of pointing to /usr/lib/libsndfile.so, it points to a libsndfile.so path that is in the application's package instead.

Note this is just a cut-down script excerpt to give you an idea; it will probably require some tweaking before it works for you (and of course you'll need to modify it to operate on other libraries besides libsndfile if that is what you want):"

#!/bin/bash -e
 
BINARY="MyAppFolder/MyAppName"
FRAMEW_FOLDER="MyAppFolder/MyAppName/Contents/Frameworks/"
 
function DoInstallNameTool {
    xLIB="$1"
    xLIB_NAME="$2"
    xBINARY="$3"
    echo install_name_tool -change \"${xLIB}\" \"@executable_path/../Frameworks/${xLIB_NAME}\" \"${xBINARY}\"
    install_name_tool -change ${xLIB} "@executable_path/../Frameworks/${xLIB_NAME}" "${xBINARY}"
}
 
for LIB in $(otool -L "${BINARY}"|grep libsndfile|cut -d '(' -f -1)
do
    echo "Handling Lib: $LIB"
    LIB_NAME=$(basename "$LIB")
    echo " Adding ${LIB_NAME}"
    cp -Rf "${LIBSNDFILE_DIR}/src/.libs/${LIB_NAME}" "${FRAMEW_FOLDER}"
 
    DoInstallNameTool "$LIB" "$LIB_NAME" "$BINARY"
done

http://stackoverflow.com/questions/7470637/dynamic-library-in-application-bundle-mac-os-x

Re:Good

[rroman]

Actually, the "shitload of bloat duplicate binaries" is quite good. Nobody gives a damn about 10 MB of their disk space because the program takes it's libraries with it. However, everyone gives ten tones of damn when they can't install new application because of "dependency problem". Solving dependency problems costs time and hence money. Disk space is cheap.
Disclaimer: I'm not saying, that new Ubuntu does that, I'm just arguing against the philosophy of bad duplicate binaries.

Re:Good

[smash]

Which, given most people have upwards of half a terabyte of storage, and the OS consumes only 1% of that, storage space is no longer at a premium.

The benefits of totally self contained packages are huge. You'll probably find that the vast majority of your application is NOT duplicated data anyhow. It will be graphic resources, it's own code, etc.

Couple that with de-dup in modern filesystems (inline still a major overhead, but scheduled de-dup isn't a major problem) and the advantages to just linking to libraries everywhere are not so clear cut anymore.

Having lived with the OS X way of packaging for a few years, and dealt with both Linux, FreeBSD and Windows since the early 90s, I know which method I prefer.

Re:Good

[smash]

Yes. ZFS will do this, as will HAMMER. However, I suspect the gains REALLY aren't there. The core OS binaries (including all libraries) on a modern system account for what... 1-2% of the disk space? If that? The bulk of the code in your new application will be the new application. Even more disk will be consumed by the resources used only for that app (graphics, widget layouts, etc.).

Disk today is cheap. Having dealt with both linking (Windows, Unix) and self contained Applications (OS X), I much prefer the latter. Installation / un-installation is way, way easier. Compatibility issues are less common.

apt

[kthreadd]

Sounds like a complement to dpkg, not a replacement

Re:apt

[Anonymous Coward]

Yeah, Colin Watson knows nothing about apt. Maybe he wouldn't be so lazy if he would start maintaining some packages in debian.

Re:apt

[exomondo]

Ubuntu is building their own system for doing it, instead of using APT, which gets them 90% of the way there. Most likely, it will be poorly done, which is the common fate of those who are too lazy to understand existing systems.

So you're saying Colin Watson is too lazy to understand existing systems? You've never actually looked at commits for APT have you.

Bloat

[paulej72]

If everything has no dependencies, then all of the dependencies must be included in the install. I wounder how many copies of each low level program would be on a given machine.

Re:Bloat

[fph il quozientatore]

If everything has no dependencies, then all of the dependencies must be included in the install. I wounder how many copies of each low level program would be on a given machine.

And, especially, good luck installing a security update on *all* your copies of a core library.

Re:Bloat

[Anonymous Coward]

Quite frankly, disk space is cheap. If this eats another 10 gigabytes of hard drive space over the traditional approach, I personally wouldn't even notice. If it made things even 50% less likely to break because of dependency problems I would be all for it.

Re:Bloat

[Kjella]

Might as well use Windows with that philosophy. The big difference with Linux is its efficiency. Get rid of that, might as well get rid of linux.

Fast, cheap, good - pick two? >:-) If you have a modern system, no OS should take a significant amount of resources on its own. Don't waste just to waste but as long as developer time is a limited resource, I'd rather have more features, higher quality and better performance than lower resource usage. The hardware is there to be used, that's what it's bought for. Hell, I regularly run an application that likes to chew through it's whole 32 bit memory space (4GB) and I let it because I still have 12 to spare and there's no direct replacement and it mostly works. What you care about is simply nothing that I care about at all, but then so I took your advice on my machine in favor of Windows 7. If Microsoft continues down their path I might have to migrate to Linux again, but if so I'd like a few other things than efficiency. Otherwise it might be a Mac, but I'd rather not.

Re:Bloat

[timbo234]

How about comparing like-with-like instead of new software with software from 10 years ago:
Ubuntu 12.04 (released 2012): 384MB minimum
Windows 7 (released 2009): 1GB minimum for 32-bit, 2GB for 64-bit
Windows 8 (released 2012): 1GB minimum for 32-bit, 2GB for 64-bit

Plus the minimum requirement for XP was 64MB, with 128MB recommended (http://support.microsoft.com/kb/314865), not 32MB.

https://wiki.ubuntu.com/PrecisePangolin/ReleaseNotes/UbuntuDesktop#System_Requirements
http://windows.microsoft.com/en-us/windows-8/system-requirements
http://windows.microsoft.com/en-us/windows7/products/system-requirements

troll bait headline

[Anonymous Coward]

A better headline:

Ubuntu Phone apps will use a different package format than debian/dpkg/apt

I guess that's not really as exciting though

Re:troll bait headline

[MrEricSir]

Ubuntu Phone apps

Let's be clear: Canonical's vision doesn't involve "phone apps." They want the same apps running on your phone and on your desktop.

The good old days

[thomasdz]

Remember the good old days when everything was just a single -r-xr-xr-x executable in /bin or /usr/bin

sigh, the good old days

Re:The good old days

[phantomfive]

Talk about simplicity. Here is an example 'Makefile' from Unix 6. Compare it to today's auto-tools:

chdir lib
cc -c -O *.c
ar r /lib/liby.a *.o
rm *.o
chdir ../source
cc -s -O y?.c
cmp a.out /usr/bin/yacc
cp a.out /usr/bin/yacc
rm a.out *.o

Re:The good old days

[MavEtJu]

Call me old-fashioned, but I prefer a proper Makefile which only compiles the files required and which separates between the build and install phase.

More Flexibility?

[organgtool]

each package would install to its own directory

Would it allow users to install multiple versions of the same application from packages? One of my gripes with Linux is that it's not easy to test new or beta versions of software since there is no easy way to install from packages alongside the existing (stable) version. Yes, I know that I could build the app from source, but that can be quite a hassle sometimes.

Re:More Flexibility?

[interval1066]

Microsoft solved this (partially) using a centralized registry...

Um, the MS registry is a huge pain in the butt for developers and M$ knows it, but they can't get rid of it becuase its too ingrained. Getting rid of the registry was a huge selling point for Windows 8, as it was for Vista... and so on. I dare you to ask me why... if you don't realize its a huge honey pot for virii and hackers you have no business even asking. Linux DOES INDEED have a system for library control, its called pkg_config and it works very well. Its not my problem if developers are too lazy to use it. 90% of linux apps I've ever envountered use it, so don't come whining to me there's no soluton this lib hell of which you speak. I do quite well with Linux, thank you.

Re:More Flexibility?

[jedidiah]

> That's because Linux suffers from a similar problem that Windows 95/98, and XP to a lesser extent did: DLL hell.

"DLL hell" has squat to do with it. The package manager is going to want to replace one version of an app with another. That is the only real problem here. If you ignore the package manager, you can install what you want.

Linux has had versioned shared libraries for ever.

The registry is just crap and you're a moron for even bringing it up in this context.

Re:More Flexibility?

[julian67]

There is no such word as virii either in English or Latin. The plural of virus is viruses.

http://en.wikipedia.org/wiki/Plural_of_virus#Treating_v.C4.ABrus_as_2nd_declension_masculine

Re:More Flexibility?

[girlintraining]

Um, the MS registry is a huge pain in the butt for developers and M$ knows it, but they can't get rid of it becuase its too ingrained.

Hi. Developer here. No, no it is not. Pain in the ass is thunking, handlers that silently disconnect from message pumps, calls to win32 that are "unmanaged" and so often lead to memory leaks even when you diligently try to do garbage collection, undocumented exceptions that are thrown by system calls that even the debugger blanches at, and endless layers of class and interfaces nine levels deep, many of which only document the public methods and tell you nothing else. The registry though? The registry is stupid easy. If you think that's a pain in the ass, you fail as a programmer. Turn in your badge.

Getting rid of the registry was a huge selling point for Windows 8,

So was the 'Metro' interface. And I don't see many people around here talking about how great Windows 8 is going to be -- everyone says it's going to be a three coiled turd. And this is what you use to advance your argument?

I dare you to ask me why... if you don't realize its a huge honey pot for virii and hackers you have no business even asking.

There's no such thing as 'virii' and I don't think you know what a honey pot is. But perhaps you mean it's a rich repository of system and application settings that could be exploited by malicious people? Yes, that is correct. But then, scanning down the C: for a list of filenames can do that too. The registry has a rich set of access controls; Your lack of familiarity with other security concepts suggests you probably didn't know that.

Linux DOES INDEED have a system for library control, its called pkg_config and it works very well.

A tool used to assist in compiling is not a "library control" tool dude. It's main job is to tell whether a library is installed or not, what version it is, and where it is located. It doesn't do much else. It most certainly does not have any management functions.

Its not my problem if developers are too lazy to use it.

*facepalm* Who wrote the applications you're using right now? Are you sure their "lazyness" isn't your problem?

90% of linux apps I've ever envountered use it, so don't come whining to me there's no soluton this lib hell of which you speak.

That whooshing sound is the point flying over your head. 100% of windows applications have to go through the kernel to load dlls, and so it presents a standardized interface for doing so. Linux does not have this. Whether it's 90% of linux apps, or 99%, the lack of a standard means you, the developer, have to guess, pray, or assume, things about the environment. You can't rely on it.

I do quite well with Linux, thank you.

That's nice. I'm a neo-pagan who dances around poles naked, and it works quite well for me. But I'm not going to suggest it's for everyone, or that there aren't occasional problems in going "sky clad" in a public place. The police, for example, aren't as understanding as one might hope.

Re:More Flexibility?

[jedidiah]

> Does ldconfig allow for different versions of the same library to be requested by the application?

Yes.

> Does ldconfig warn you when a dependancy isn't met?

Better than that. You as a user can see what libraries are loaded from where and which ones are missing precisely.

The "superior alternative" is just a black box.

Re:More Flexibility?

[characterZer0]

You're just angry that I'm pointing out that linux lacks a central repository for application and kernel settings and you have to dig through /etc 's mass of files to do the same thing. Linux is still rocking the equivalent of ".ini" files, and yeah -- it is primitive. And I'm not a moron for bringing it up, you're a moron for not seeing that sometimes, your religion of choice, could benefit from looking outside of itself and seeing that other developers have done something better.

Text files for configuration are great. I can version control them, copy them from one system to another, see meaningful diffs between them, and invidual applications can choose formats that are sensible for them.

Why on earth would I want to cram everything into a central repository?

Re:More Flexibility?

[Anonymous Coward]

Your ignorance of Linux is showing. Please refrain from commenting about things you do not understand.

Can you please point out which setting in the man page allows you to set this, oh great wizard of Linux? Because I think you're just being a contrarian right now, and haven't actually read the damn page.

You won't find that in the ldconfig man page because it's provided by the filesystem. This is a small snippet from the contents of my /usr/lib:

libboost_python-2.7-1_49.so
libboost_python-2.7-1_49.so.1.49.0
libboost_python-2.7-mt-1_49.so
libboost_python-2.7-mt-1_49.so.1.49.0
libboost_python-2.7-mt.so
libboost_python-2.7.so
libboost_python-3.2-1_49.so
libboost_python-3.2-1_49.so.1.49.0
libboost_python-3.2-mt-1_49.so
libboost_python-3.2-mt-1_49.so.1.49.0

Do you see what is happening there? Have you ever actually looked inside /lib or /usr/lib of a *nix system? Did you grasp what you saw? One application may need /usr/lib/libboost_python-2.7.so while another needs /usr/lib/libboost_python-3.2-1_49.so. Both get what they need.

In Linux the library's version is part of its filename. There is no "dll hell". DLL Hell in Windows was caused by a dll with a given filename being replaced by a different version with the *same filename* in the *same location*. I don't think you really understand the DLL Hell situation.

There's no man page for knowing what you're talking about.

Re:More Flexibility?

[Anonymous Coward]

/etc is the registry, and it is no more or less centralized than the Windows registry (which consists of multiple files).

The difference is that /etc is implemented using two of the most mature and well understood pieces of the system: a file system directory hierarchy and text files.

The windows registry, in contrast, is implemented using a limited and clunky database, which is difficult to manage and drives even experienced users to frustration.

And speaking of "rich access control". /etc has extremely robust access control via the filesystem. Access control that is well tested, well documented and well understood.

Re:More Flexibility?

[alcourt]

Query: ldd
Control: see the various environment variables that specify which lib dirs are used in what order for that environment you just created. (LD_LIBRARY_...)

Applications can specify the exact version number of a library (.so.1 vs .so.2).

Note, none of this is Linux specific. That family of operating systems is far from perfect in shared library handling, but I won't pretend that it doesn't inherit at least some tools that have at least some flexibility from older Unix tools.

Re:More Flexibility?

[serviscope_minor]

Who are you and what have you done with girlintraining?

The dynamic loader on Linux is very flexible. There are actually several different systems of varying granularity availble.

The coarsest is using LD_LIBRARY_PATH to allow one program to use a different .so.

The next is versioning with libblah.so.?, which allows different .sos to be made available depending on how the program was compiled. This requires a little bit of care on the part of the library author to actually bother to increment the version numbers.

The next, more complex is symbol versioning within a .so. This allows a single .so to have multiple different versions within it, so a single.so can actually serve multiple different versions. This is really good for system libraries and allows a great deal of backwards compatibility without bloating the number of libraries, and while allowing the maximum amount of code sharing. It requires the most discipline, so is generally only done by the dedicated libc and libstdc++ people. It's particularly important for system libraries, since it allows libraries which depend on "different" versions of glibc or libstdc++ to be linked together without trouble.

This is why a modern libstdc++.so.6 can happily serve g++ compiled binaries from just after the last ABI change to now without trouble.

Keeping hundreds of linux boxes up to date and patched is a fucking nightmare.

What? With the popular, sane distros, you can install some version with some reasonable support term (DeadRat, CentOS, Ubuntu LTS, Suse) and just tell it to auto-update the packages at 3 AM. You can even point it at a local package mirror if you want to save on external bandwidth.

In fact one can easily create a PXE installer which will install a customised package list and set up that configuration for you. Once you've done that, you buy a new machine, mess with the BIOS just enough to PXE boot it, choose the version you want and hit go. 1 hour later, you'll have a working, freshly installed system which will keep itself up to date with security patches until the distro drops off support.

If you want to rely on 3rd party programs which sit in /opt, it is very easy to download the program, untar it and chuck the files into a dkpg or RPM file, and then just add that package to the list. In fact people like LibreOffice only make APTs and RPMs available making it a minor faff to install on a less popular distro.

Again, once you do that, it will automatically roll out to all machines. Of course there's a bit more interaction, but then it is up to you whether you want to upgrade to a new major version of LibreOffice or whatever. And this way, the users won't get exposed to those program's nasty auto-updater scripts hassling them for new versions or whatever.

Honestly, if you're having trouble keeping Linux boxes up to date, then you are doing something wrong.

Also known as application bundles.

[sidragon.net]

http://en.wikipedia.org/wiki/Application_bundle

Once again, racing as fast as we can towards where the puck was 20 years ago.

Goodbye Ubuntu

[FuzzNugget]

Hello Debian

I think this is for "apps" not applications

[jandrese]

I don't think we're going to see a completely separate install of Gnome for every single Gnome app on your system. I think this is intended for standalone apps like you see on phones and tablets.

Re:but... WHY?

[amorsen]

We need it because while current packaging systems are great for central control, they are bad for actually letting users contribute.

a) You cannot submit a bug to a developer, get a fixed beta release, and install that in the packaging system (unless you know how to handle spec files)
b) You cannot do parallel installations of newer (or older) versions for testing unless the package is built specifically for that
c) It is difficult to make distribution-independent packages, so users become dependent on the distribution for all software
d) Almost all packages require root, the packaging system cannot track software installed by users themselves

On the other hand, switching to a Mac-style packaging system has at least these problems:

1) Security updates to common code are unlikely to actually get applied to all packages
2) Some libraries will not be shared, costing extra memory and cache footprint
3) Centralized control over what software is installed suddenly becomes difficult
4) Without dependencies you need to define the minimal environment that software can depend on. LSB tried to do that and failed.

A crippled version of APT

[gringer]

From the email:

The proof of concept I wrote also isn't entirely new code. It's tiny due to using .deb as a container format (minus maintainer scripts, full dependencies, etc.), so I get to save effort by using dpkg to unpack things, which leaves us room to selectively use more of its features in future if we want to.

So they start off with what they think they need, then become more like APT as they need to add more features.

So the scope of what I've been considering is purely leaf apps built on a fixed "base system", which in the case of the initial target of the Ubuntu phone/tablet work would be the run-time part of the Ubuntu SDK.

In other words, this is something to be used in addition to APT (i.e. post-install), rather than instead of APT.

* no dependencies between apps; single implicit dependency on the base
      system by way of a Click-Base-System field

Just like Debian has an implicit dependency on the base system (except for base packages, which have more complicated rules). In other words, this system will only accept a single dependency, the Click-Base-System. I'm not quite sure why this is different from only accepting applications that only depend on Click-Base-System.

And note that the "each package will install to its own directory" bit is on the to-do list:

Obvious items I still need to work on:

• produce a strawman hooks implementation with some real worked examples
• integrate (or demonstrate how to integrate) the container isolation properties worked on elsewhere
• Click-Base-System field is skeletally simple right now, and may need to be expanded to at least leave open the possibility of multiple flavours of base system (see also GNOME's profiles idea)
• adjust unpack handling to avoid problems with project renames and name clashes, and to unpack each version into its own directory and flip symlinks to allow for multi-user independence
• integrate into the Ubuntu SDK, as well as providing examples of how it can be integrated into other build systems too

Re:I never got "packaging systems"

[cas2000]

actually, it's something that Windows gets dead-wrong, because executable installer apps (setup.exe and the like) are just plain fucking stupid.

they're an unfortunate necessity because windows doesn't have, and never has had a decent package management system. and is unlikely to ever get one because the windows software market is primarily commercial and proprietary.

when you have 10 (or 100 or 500) packages to upgrade on a single system (and then multiply that by 100 or 1000 systems), executing hundreds of installer packages one after another is the worst possible way to do it.

i've never understood why Windows (or Apple) users tolerate that shit. it's a tedious chore that's ripe for automation - exactly the kind of thing that computers are good at doing and users are bad at doing (due to boredom, fatigue, loss of attention, ignorance, or stupidity)

which is precisely why linux distros (and other *nixes) don't do it that way. they have packaging systems because systems are consistent, predictable, and easily automated.

windows users and windows developers often have just the wrong way of looking at things, the wrong mental model of how things work and how they should work.

I ran across a program for Windows recently called Ninite. It's a multi-app installer and updater. it sounds like a good idea and is. it's a big improvement over the usual click-and-execute for each individual program.

except the way it works is weird and clumsy:

you go to their website and select which apps you want to install (from a bunch of internet-available apps, including free software and proprietary freeware like adobe flash), and then it builds you an installer app that you download and run, and it installs and/or upgrades the apps you selected off their website.

WTF?

nice starting idea, but the implementation is idiotic. Why not just have one Ninite app that fetches a list of available apps and installer URLs and whatever custom installer scripts ninite needs for them) and allow the user to select which apps whenever they run it?

i.e. instead of a moronic implementation, actually make a smart and useful implementation that copies good ideas from linux distro installers like apt-get and yum, and re-purposes them for the Windows environment.

(oh, and adobe are sending cease-and-desist letters and threatening to sue if the ninite developer doesn't remove the ability to install & update downloadable adobe products like flash, so his good ideas and good intentions are fucked by the corporate vermin mindset that dominates the windows software market)

another thing windows devs don't get is shared libraries (DLLs in windows terminology). Why does every single app have to install their own copy of the MS C++ libraries? or .net? or nvidia physx? and numerous other common library packages? these things are supposed to be shared resources provided and kept up-to-date by the operating system, not bundled with every app that uses them.

Look into Debian's CUT project

[neiras]

Except Ubuntu users want cutting edge Debian, not tried and tested Debian...and unfortunately using Debian is not going to make it more cutting edge.

...Now if Debian decided to produce a (stable) cutting edge Desktop version (perhaps working with an existing Distribution team). To complete there ultra stable, you have me sold.

You asked for it, Debian delivers. The Debian CUT Project aims to publish usable snapshots of Debian Testing on a monthly basis. They're pretty new but picking up steam.

Nope..

[Junta]

*If* everyone picked exactly the same lib version, yes.

In practice, people aren't going to standardize on the same library version.

Bonus problem: Now each app provider is responsible for addressing a hypothetical libcrytpo vulnerability rather than the distro patching it in one place.

Re:Nope..

[Burz]

A typical desktop distro is an example of a system that is very 'fat' in terms of available components, but poor in terms of what can be considered integral to the system. There is gobs of duplicated functionality, and the packaging system treats all these pieces pretty equally and a lot of it can get in the way when you have independantly installed apps. So I think the difficulty for upgrading that you speak of is more acute in desktop distros than it is for, say, OS X.

Ubuntu is starting to limit the role of the managed repository: It will continue to be used for the core OS, but apps will increasingly use the new format which primarily specifies the OS version as a dependency to be checked. If the app needs more than what the OS offers, then they are free to include the extra libraries in their own tree. This makes a lot of sense and I hope it catches on with app developers.

Re:grep: command not found

[Anonymous Coward]

Anyone on this site who is stumped by grep missing need leave. Now.