Ubuntu Gets Container-Friendly "Snappy" Core

judgecorp writes: Canonical just announced Ubuntu Core, which uses containers instead of packages. It's the biggest Ubuntu shakeup for 20 years, says Canonical's Mark Shuttleworth, and is based on a tiny core, which will run Docker and other container technology better, quicker and with greater security than other Linux distros. Delivered as alpha code today, it's going to become a supported product, designed to compete with both CoreOS and Red Hat Atomic, the two leading container-friendly Linux approaches. Shuttleworth says it came about because Canonical found it had solved the "cloud" problems (delivering and updating apps and keeping security) by accident — in its work on a mobile version of Ubuntu.

Comments

Re:20 years?

[vux984]

Well TFA says:

"This is in a sense the biggest break with tradition in 10 years of Ubuntu..."

Editor fail.

This actually sounds pretty cool.

[Fwipp]

No dependency management or fooling around packages that require conflicting library versions, possibly near-instant "installation" (depending on if they're distributing Dockerfile-equivalents* or containers directly). Sounds good to me - I'll have to take a look sometime.

*Yes, I know that Docker is not the only way to do containers, but it's easy to imagine they could be using a similar "build" step.

Re:This actually sounds pretty cool.

[Fwipp]

Update: Playing with it, It looks like there isn't a build step. Following the steps here: http://www.ubuntu.com/cloud/to... - it seems that docker is installed under /apps/docker (~20 files total) - they're basically just distributing tarballs which contain all necessary dependencies.

From there, it puts a quick wrapper script at ~/snappy-bin/docker, containing:

#!/bin/sh
##TARGET=/apps/docker/1.3.2.007/bin/docker
#TMPDIR=/run/apps/docker
cd /apps/docker/1.3.2.007
aa-exec -p docker_docker_1.3.2.007 -- /apps/docker/1.3.2.007/bin/docker $@

To build your own "snappy package" looks as simple as "cd mydir; snappy build ."

Re:This actually sounds pretty cool.

[Imagix]

But... when a library is patched then you have to re-download your entire machine again. What's the point of shared libraries? Just static link everything and you get a lot of the same result. (Not all.. there are other features of containerization... but static linking solves your library hell. And then there's the GPL/LGPL issues....)

Re:This actually sounds pretty cool.

[Fwipp]

Yeah, there's a lot of drama 'round there. Cynically, it makes sense for CoreOS to roll their own solution, as Docker continues to reach into CoreOS's core areas. On the other hand, the Rocket announcement certainly sounded nice and desirable.

Neither the Rocket announcement nor the Docker response made me feel like I was reading a technical appraisal of either technology. Too much politics for me to dig through, so I'll wait for smarter people than I to weigh in before making a decision.

Re:This actually sounds pretty cool.

[geantvert]

The article indicates that containers can be upgraded by sending only the modified files so upgrading a lib should not cost too much.
If their system is not too stupid then it will manage the container at the file level and will try to share identical files between packages.

That should not be too difficult using checksums and hard links. If they don't do that then the kernel won't be able to share files between applications and the whole system will use a lot more memory. Linux application are sharing libs but also some data files such as TTF fonts.

Re:This actually sounds pretty cool.

[MMC Monster]

How does this mesh with the ideas that:
1. shared libraries allow a small memory footprint
and
2. allowing the OS to manage dependencies allows applications to be more secure since all applications that rely on a shared library benefit when a security update of the library get installed

Re:This actually sounds pretty cool.

[turbidostato]

"No dependency management or fooling around packages that require conflicting library versions, possibly near-instant "installation" (depending on if they're distributing Dockerfile-equivalents* or containers directly). Sounds good to me"

Congratulations. You have discovered static linking. Welcome to the fifties.

Now, in less than ten years you will find the problems with your approach and will also reinvent dynamic linking and I'll gladly welcome you to the sixties.

Re:This actually sounds pretty cool.

[DuckDodgers]

I wrote this elsewhere, but to repeat: the advantage of Docker and similar containers isn't static linking. The advantage is that you get most of the important benefits of whole operating system virtualization with much lower overhead in terms of disk space and resource use. Instead of building one 5GB CentOS or Ubuntu VM that runs a copy of your web server software and pushing that all over your network and having every running instance hold 100MB of RAM just for a virtual kernel in addition to the web server itself, you build a 500MB Apache/Nginx/Whatever Docker container with the same web server and push it everywhere, and its memory and CPU overhead on top of the web server is tiny. And since, yes, it uses statically linked libraries, it doesn't matter if the host OS has a different version of libevent as long as it has a compatible version of Docker or lmctfy or whatever container technology you use.

Now obviously if you wanted to make the virtualized environment available to a remote user as a remote desktop, or for remote login directly into the virtual environment with ssh for configuration changes, you want a full virtualized operating system. In that case you need VirtualBox, KVM, Xen, VMware, Hyper-V, etc... but for deploying identical instances of a configured application to a handful or even hundreds of machines with the smallest feasible overhead and a nice security layer to prevent most breaches of the hosted application from leaking into the host operating system, a container is great. If you look up how Docker and lmctfy work, they are mostly an API wrapper around the Linux kernel cgroup and selinux features for configurable restriction of a process's ability to utilize CPU, RAM, and IO.

Re:This actually sounds pretty cool.

[turbidostato]

"the advantage of Docker and similar containers isn't static linking. The advantage is that you get most of the important benefits of whole operating system virtualization with much lower overhead in terms of disk space and resource use [...] for deploying identical instances of a configured application to a handful or even hundreds of machines with the smallest feasible overhead"

Please take a time to think about what you wrote. Once you deploy your app to "hundreds of machines" you are back to square one: you have again the overhead of hundreds of kernels and basic OS. And if you think "but I avoid moving around a 5GB image for the complete VM!" you are doing it wrong, since you don't move the VM image, but the VM's contents' definition: the land of puppet, chef, et al.

No. The point _is_ -even yourself accept it, that this way "...it doesn't matter if the host OS has a different version of libevent", so you are using your docker as a way to "statically link" your application, which is the wrong approach, as it was already discovered quite a lot of time ago.

Now, what's in fact the nut of the problem? It is undisciplined developers. Undisciplined developers that publish libfoo, without any guarantee or description of what version of libfoo is it going to be long term supported and when told about a bug just tend to answer "please use our last version, since we don't want to spend our time correcting past ones but, of course we do have time to spent on introducing new bugs" and undisciplined users of libfoo which when start developing just link against the last version of libfoo, not minding that it is just an interim one.

On this environment you end up needing half a dozen different versions of libfoo to support half a dozen of otherwise functionally similar applications and, instead of trying to solve the problem at its root, which is using one, you cover this failure with a bigger one: "I know, let's statically link the application to the versions they need and it's done".

We see this syndrome all around: from docker to pip requirements, to gemspec files, to turbogears virtualenvs, to all the other versioning systems over there to install/compile exactly version x.y.z of foo or an equivalent abstraction.

So many goddamn layers.

[Anonymous Coward]

And here we go again, adding yet another layer to an already wobbling stack of layers.

First we have hardware. Then we're running Xen or some other supervisor on that hardware, so we can have numerous VMs running Linux running on one physical system. Then each of these Linux VMs is in turn running VirtualBox, which in turn is running Linux, which in turn is running some container system. Then each of these containers is running some set of software. In some cases these containers are running something like the Java VM, which is, of course, another layer. Then in some truly idiotic cases, we have something like JRuby running on this JVM. There's some half-baked Ruby code running on JRuby.

Let's visualize this stack:

- Ruby code
- JRuby
- JVM
- Container
- Linux
- VirtualBox
- Linux
- Xen
- Hardware

Now that there's all this compartmentalization, it becomes a royal pain in the ass to share data between the apps running in the containers running in the VMs running on the actual hardware. So we start seeing numerous networking hacks to try and make it all into something barely usable. So throw on Apache, Varnish, and other software into the mix, too.

I'm sure that within a few years, we'll start seeing containers within containers, if that isn't already being done. Then those will need sandboxing, so there will be sandboxes for the containers that contain the containers.

Meanwhile, it's just one hack after another to intentionally get around all of this isolation, in order to do something minimally useful with this stack. The performance of the system goes swirling down the shitter as a result of all of the layers, and all of the effort needed to bypass these layers.

What a fucking mess!

Re:So many goddamn layers.

[skids]

You think that's bad, you should see what goes on on the network side these days. All the layers of encapsulation can very often be larger than the innermost payload.

Re:20 years?

[Fwipp]

Two years ago's 20-years-ago would have been 1992? Unless leap years work very differently than I've been told...

Re:What about things like the JVM inside a contain

[Fwipp]

Each container would contain all of the stuff it needs to run - in this case, Java + associated modules.

It simplifies stuff, because if one server requires Foo v1.11.4 but another needs Foo v1.10.8, neither server "sees" the other. You simply configure each container separately, without worrying what the other container's doing. When distributing the container, all you have to do is send out one image. If you want to run 12 containers on a host, that's cool. If you want to run only 1, that's fine too. And that same container will work just fine whether it's running on the server or the new kid's development laptop.

It's not an all-or-nothing approach, so you can choose if you want the database to live in a container of its own, on the host, in the app container, or somewhere distant.

OK. I'm throroughly confused

[sconeu]

I'm just a casual user, not a sysadmin.

But I thought containers were kind of like VMs, not like packages.

What's the difference between a VM, a container, a chroot jail, and packages?

Auto analogies are always welcome.

I Don't Get It

[ewhac]

Am I getting hopelessly old and unable to appreciate new things, or is this not anywhere near as whoop-de-doo as they're making out?

"You can update transactionally!!" Great. What does that mean? Is it like git add newapp; git commit -a? If so, how do I back out a program I installed three installations ago?

Transactional updates have lots of useful properties: if they are done well, you can know EXACTLY what's running on a particular system, [ ... ]

dpkg -l

You can roll updates back, [ ... ]

dpkg -i <previous_version>

...lets you choose exactly the capabilities you want for yourself, rather than having someone else force you to use a particular tool.

#include <cheap_shots/systemd.h>

Because there is a single repository of frameworks and packages, and each of them has a digital fingerprint that cannot be faked, two people on opposite ends of the world can compare their systems and know that they are running exactly the same versions of the system and apps.

debsums

Developers of snappy apps get much more freedom to bundle the exact versions of libraries that they want to use with their apps.

...Did this guy just say he brought DLL Hell to Linux? Help me to understand how he didn't just say that.

I bet the average system on the cloud ends up with about three packages installed, total! Try this sort of output:

$ snappy info
release: ubuntu-core/devel
frameworks: docker, panamax
apps: owncloud

That's much easier to manage and reason about at scale.

No, it isn't!! What the hell is OwnCloud pulling in? What's it using as an HTTP server? As an SSL/TLS stack? Is it the one with the Heartbleed bug, the POODLE bug, or some new bug kluged in by the app vendor to add some pet feature that was rejected from upstream because it was plainly stupid?

Honestly, I'm really not getting this. It just sounds like they created a pile of tools that lets "cloud" administrators be supremely lazy. What am I missing here?

Re:I Don't Get It

[ArsonSmith]

being able to go from zero to fully functional and testable running application with mulitple tiers in minutes is a bit different.

Being able to completely uninstall without dpkg-old or random .bak files laying around is kinda nice.

rolling back 3 versions ago is as simple as
docker run

Honestly I've been doing this for 22 years and docker is the first time I've looked at a tech and it scared me. It is going to relegate systems administration to little more than rack and stackers. The current DevOps trend is going to become just Dev and everything will be code.

If you're an administrator of pretty much any type you better start learning to program. The days of long lived static apps with a full time support staff is going to go away. The art of setting and configuring the exact combination of packages, standards, access etc will be gone.

Re:20 years?

[binarylarry]

It'd be fucking great if old Mark decided to instead ship some stable, mature software once.

I'm fucking sick of Ubuntu Unity freezing, locking up or getting stuck (lol you're stuck between workspaces and can't do anything about it but reboot or bounce Xorg!)

Re:OK. I'm throroughly confused

[Anonymous Coward]

Ok let's get you up to speed on containers in 7 paragraphs and there is some pottering hiding somewhere in here to keep folks interested. A VM emulates the entire hardware layer. A container depends on cgroups and namespaces support in the Linux kernel to create a lightweight isolated OS environment with network support. So you could be running a Debian host and multiple Redhat, Centos, Ubuntu, Fedora etc containers and vice versa.

The advantage is because containers are not emulating a hardware layer you get near bare-metal performance, easier access to hosts filesystem via simple bind mounts and easy workload portability across any Linux system. The limitation is you can only run Linux OS containers as it depends on the Linux kernel.

The Linux containers (LXC) project has been baking since 2009, and has been supported by Ubuntu since 2012. However its been a low profile project. A lot of hardcore Linux users have not heard about LXC let alone used it. LXC gives you system containers, ie you get a near complete Linux environment like a lightweight VM.

Docker was using LXC in its previous avatar of dotcloud to run a public PAAS platform and experimented with LXCs support for overlay filesystems like overlayfs and aufs. They then released this as Docker in 2013. Docker basically used LXC as a base to abstract the container away to an app, which is composed of layers of aufs filesystems. Compared to LXC Docker gives you a more constrained OS environment, mainly a single app execution environment made up of aufs read only layers. Running multiple apps, daemons, cron, services will requirments will require workarounds, like a bash script launching multiple apps or processes, or using a process manager like runit or supervisor. And any data produced by the container in the top most aufs read-write layer is transient and has to be commited to a layer, bind mounted to the host eg mysql databases, logs etc or lost.

The big problem occupying brainy minds lately seems to be orchestration of containers and workloads in containers across multiple hosts and this is the problem Ubuntu's recently announced LXD project based on LXC, CoreOS, Docker libswarm, Mesosphere and multiple others are trying to address.

There are also efforts to run container only OSs - ie a bare minimum OS with apps, everything in isolated containers (which is where Lennart Pottering's recent blog post about using btrfs subvolumes, low cost COW snapshots, and containerized apps seems to suggest he is taking systemd to eventually). This is where CoreOS, Redhat's Project Atomic and now Ubuntu Core seem to be moving towards.

A chroot jails is barebones compared to a container. A container can use cgroups to limit memory and cpu per container and network namespaces. LXC can now use usernamespaces support in the kernel to let users run unprivileged containers by non root users.

Re:20 years?

[killkillkill]

That short hand also refers to BSD variants. I don't think its a literal regular expression.