[Lucid] fails to host and to be hosted using LXC/OpenVZ containers

[Thomas Bechtold]

Binary package hint: lxc

i tried to setup a container (lucid) on a lucid host system.

1) # debootstrap --arch i386 lucid lucid/ http://archive.ubuntu.com/ubuntu/
2) # mount -t cgroup cgroup /cgroup
3) # lxc-create -n ubuntu-lucid
4) # lxc-start -n ubuntu-lucid
init: Unable to listen for private connections: Failed to bind socket "/com/ubuntu/upstart": Address already in use
mountall: Could not connect to Plymouth
swapon: /dev/mapper/cryptswap1: swapon failed: Device or resource busy
mountall: swapon /dev/mapper/cryptswap1 [27] terminated with status 255
mountall: Problem activating swap: /dev/mapper/cryptswap1
5) CTRL+Z && kill %1
6) # lxc-start -n ubuntu-lucid
lxc-start: Device or resource busy - failed to remove previous cgroup '/cgroup/ubuntu-lucid'
lxc-start: failed to spawn '/sbin/init'
lxc-start: Device or resource busy - failed to remove cgroup '/cgroup/ubuntu-lucid'

Maybe i fogot some steps to do?

[C Anthony Risinger (extofme)]

there could be other problems, but you need to define a config file with lxc-create (the -f option). specifically you need the option:

lxc.rootfs = <wherever you just debootstrapped>

to be defined. otherwise, the rootfs will be the shared with the host. anything that is not defined in the config file is shared with the host (network stack, hostname, resources, etc) specifically in the rootfs case, newer revisions of lxc-* tools will not even allow you to get as far as you did (sharing the rootfs doesnt make sense for a system container). see this thread for more details:

http://ubuntuforums.org/showthread.php?t=1382823

specifically this post:

http://ubuntuforums.org/showpost.php?p=8736281&postcount=10

where i referenced this bug report.

[bodhi.zazen (bodhi.zazen)]

I also have this problem.

I build a container with debootstrap --variant minbase

I then confirmed the files in rootfs.lucid/dev were present and had the appropriate permissions.

I wrote a basic config defining the network, container name, root , etc

The container starts but does not mount proc sys dev/shm or dev/pts

If I chroot in and manually mount then restart the container, still no joy.

The container itself outputs

# lxc-start -n lucid

mountall: Could not connect to Plymouth

I attempted to use the upstart script I use in my karmic openvz VPS, but it did not help.

[C Anthony Risinger (extofme)]

also this probably should be handled more gracfully on lxc-*'s part, but the "resource busy" is a consequence of killing the lxc-start process, and thus the cgroups dont get cleanup up properly. make sure you kill the process lxc-start STARTED, i.e. /sbin/init (it will be the pid+1 of lxc-start), and not lxc-start itself

[C Anthony Risinger (extofme)]

to elaborate on the "resource busy" issue... you need to remove/kill the running processes from the cgroup in order to remove the folder.

view the tasks with:

$ cat /cgroup/ubuntu-lucid/tasks
<all the pids assigned to this cgroup>

$ kill -15 <all the pids assigned to above cgroup>

most processes should die, unless it was pid one in the container, then it will need a kill -9. after that you can:

$ rmdir /cgroup/ubuntu-lucid

so long as there are no more nested directories under /cgroup/ubuntu-lucid.

[Thomas Bechtold (toabctl)]

i used a config file now with the following content:

lxc.utsname = ubuntu
lxc.rootfs = /var/lib/lxc/ubuntu/root/

When i start the container with:

# lxc-start -n ubuntu -f /var/lib/lxc/ubuntu/config

nothing happens. But:

# lxc-info -n ubuntu
'ubuntu' is RUNNING

# lxc-ps -n ubuntu
CONTAINER PID TTY TIME CMD
            6753 pts/1 00:00:00 bash
            6847 pts/1 00:00:00 lxc-ps
            6848 pts/1 00:00:00 ps

So looks like the container is up and running. But how can i connect to the container? Like "vzctl enter container-name" in openvz?

[bodhi.zazen (bodhi.zazen)]

Normally you would enter the container with lxc-console -n ubuntu , however it is not working for me as of yet with a lucid container.

Your config file looks very very scant, that is probably about as minimal as you can get, what else did you do teo set up your container ? You need to do a bunch of manual configuration still, at least configure your networking.

Once you have a container up and running you can connect with lxc-console , ssh , VNC, you of course need to install the servers in the container first.

I was able to get a lucid container to boot last night, and it will start "OK" in that I can connect to the container via ssh.

I can not connect via ssh as above.

if I start the container with the -d option

lxc-start -d -n ubuntu

It does not boot properly.

It boots "OK" if I do not use the -d option

lxc-start -n ubuntu

It gets an IP address, and I can ssh in, but no luck with lxc-console.

[C Anthony Risinger (extofme)]

Thomas Bechtold and friends, the lxc-ps command i think is buggy, and the -n flag doesnt work, try using --name instead. the output you are seeing is NOT inside the container, it is the same as what you would see if you simply typed ps. if the process was running in a container/cgroup you would see the container name next to the process, like this:

cr@ph1 ~ $ lxc-ps --name guest-personal-tony
CONTAINER PID TTY TIME CMD
guest-personal-tony 4281 ? 00:00:01 init
guest-personal-tony 4318 ? 00:00:00 syslog-ng
guest-personal-tony 4319 ? 00:00:00 syslog-ng
guest-personal-tony 4370 ? 00:00:00 crond
guest-personal-tony 5768 pts/3 00:00:00 login
guest-personal-tony 5769 pts/3 00:00:00 bash
guest-personal-tony 25747 pts/3 00:00:00 su
guest-personal-tony 25748 pts/3 00:00:00 bash
guest-personal-tony 25851 pts/3 00:15:27 python
guest-personal-tony 31858 ? 00:00:00 dhcpcd
guest-personal-tony 32503 ? 00:00:00 git-daemon

another way of seeing the processes is to look at the cgroup itself:

cr@ph1 ~ $ cat /vps/var/cgroup/guest-personal-tony/tasks
<you will see a list of pids running within this cgroup>

cgroup is a psuedo filesystem, and everything you can do with containers/LXC can be managed by manipulating/echoing values into the files within a mounted cgroup. if you create a folder in a cgroup, you effectively begin a hierarchy of groups/aggregated processes, each with their own limits; notice that once you make a folder in a mounted cgroup, all the device files automagically appear inside it, ready to be manipulated. each process on the system must belong to one and only one node in the hierarchy. read this for a better understanding:

http://www.mjmwired.net/kernel/Documentation/cgroups.txt

to use lxc-console, you must enable tty's inside the container with this configuration option:

lxc.tty = N

where N is the number of tty's you want the container to have. lxc-* tools will connect a pty in the host to a tty in the container. the container is responsible for runny a getty or some kind of handler on the tty. if you enable 3 tty's, you can lxc-console into the container 3 independent times. i use an autologin binary and some agetty options to autologin as root anyone entering from the host.

[C Anthony Risinger (extofme)]

bodhi.zazen, im not sure if ubuntu has the latest lxc-* tools, the -d problem you having could be related to that, because i run all my containers with that option. however, i have noticed my container's (i can't confirm this for ubuntu based containers yet) networking does not come up right away for interfaces that are DHCP based on a bridge. i tried setting the bridge forward delay to zero:

brctl setfd <bridge name> 0

but that doesn't seem to help, and i'm still investigating this.

[bodhi.zazen (bodhi.zazen)]

I got it working, with a lot less tweaking then you posted on the forums sweetsinse.

I will work through my notes and post an update ASAP. Basically you do not need to worry about udev , just set up your devices and edit a few system files.

Personally I use a static IP on both the host node and in the containers, and it is very fast.

The default lxc in lucid is 0.6.4 I believe there is a ppa with a more recent version.

http://packages.ubuntu.com/lucid/lxc

I will try to compile 0.6.5 and see if the -d flag works, and if so, hopefully the lucid repo could be updated to 0.6.5 ? I am not sure about that.

[bodhi.zazen (bodhi.zazen)]

This is how I got it working :

http://blog.bodhizazen.net/linux/lxc-configure-ubuntu-lucid-containers/

[Eugene San (eugenesan)]

Lucid as LTS must support containers both as host and as hoster.
Not forfeiting that requirement will have huge impact on Ubuntu

[Scott James Remnant (Canonical) (canonical-scott)]

Upstart (and many other key parts of the system) require a minimum kernel version of 2.6.32, and a configuration compatible with Ubuntu's.

The problem with OpenVZ is that the kernel is too old, and has important features disabled (such as the uevent netlink socket). There are far better virtualisation technologies out there.

There are also known "workarounds" to the problems, which I have given to OpenVZ in the past - you can replace the majority of the boot with a small shell script that emits the appropriate filesystem events

[Eugene San (eugenesan)]

I suppose we should be discussing LXC support in current Lucid's Linux and upcoming OpenVZ support in 2.6.32 which probably will be ported to Lucid (officially or not) very soon.

And what "Better virtualisation technologies" are? KVM? Xen?
They are totally different beasts.

There is no permanent workaround, for example installing package with dependency for udev will ruin the VPS.

New init scheme is the reason for inability to boot under containers.
Upstart is major player here, and developers, with all due respect, can't just ignore the situation they created.

I am really happy to see my Laptop booting Lucid under 30sec, but I also have much more of virtualized instances.
Do you aware of how many Ubuntu insances are running inside containers? My guess is close to a million.
And what do you think people will do, after realizing they can't upgrade/install?
Definitely not praising Canonical.

I really hope Canonical will not endorse such a nonchalance.

[Scott James Remnant (Canonical) (canonical-scott)]

On Wed, 2010-04-28 at 06:30 +0000, Eugene San wrote:

> I suppose we should be discussing LXC support in current Lucid's Linux
> and upcoming OpenVZ support in 2.6.32 which probably will be ported to
> Lucid (officially or not) very soon.
>
If OpenVZ is ported to 2.6.32, then the problem will be fixed (assuming
the config enables the udev netlink socket).

> There is no permanent workaround, for example installing package with
> dependency for udev will ruin the VPS.
>
udev is a dependency of many parts of the system, including the X
server.

> New init scheme is the reason for inability to boot under containers.
> Upstart is major player here, and developers, with all due respect, can't just ignore the situation they created.
>
Upstart isn't actually the cause of the problem, it's quite compatible
with OpenVZ. The problem is that OpenVZ is *incompatible* with udev -
which is a fundamental component of the modern Linux system.

If you want to discuss further, this is entirely the wrong place to do
it! You'd need to take it upstream with the Linux kernel and udev, e.g.
on the LKML mailing list.

> I am really happy to see my Laptop booting Lucid under 30sec, but I also have much more of virtualized instances.
> Do you aware of how many Ubuntu insances are running inside containers? My guess is close to a million.
> And what do you think people will do, after realizing they can't upgrade/install?
> Definitely not praising Canonical.
>
> I really hope Canonical will not endorse such a nonchalance.
>
This has nothing to do with Canonical. No new release of any Linux
distribution will run under OpenVZ. We've all adopted the same basic
set of changes, and all made our distributions reliant on udev.

Sorry, but your complaint is misdirected. It needs to be dealt with
between the upstream Linux kernel, OpenVZ and udev developers -
Canonical are not involved here - we're just the first to release this
generation of kernel/udev/etc.

(Fedora 13, RHEL 6 and OpenSuSE 11.3 are all due to be released not long
after, and will have similar problems with the current generation of
OpenVZ.)

From a purely Ubuntu point of view, I'd like to point out that Ubuntu
8.04 LTS works just fine with the current OpenVZ kernel, and continues
to be supported for three more years to come!

There's no rush to upgrade to 10.04 LTS, you can carry on running 8.04
LTS until such time as OpenVZ is updated to be compatible with Linux
2.6.32 and its userland.

Scott
--
Have you ever, ever felt like this?
Had strange things happen? Are you going round the twist?

[Scott James Remnant (Canonical) (canonical-scott)]

It looks like OpenVZ are definitely planning to update their kernel
patches to 2.6.32:

http://community.livejournal.com/openvz/30934.html

As far as I'm aware, this is the only problem that's caused difficulty
running Ubuntu 10.04 under OpenVZ without workarounds.

Scott
--
Have you ever, ever felt like this?
Had strange things happen? Are you going round the twist?

[Christoph Lechleitner (lech)]

I am not so sure if OpenVZ as a project is going to "survive" the fact that Ubuntu 10.04 LTS and RHEL6 both are switching to LXC, I am afraid it might starve to death.

However it should become (a lot) easier to get a current distro (like lucid) running (and up to date) as guest under any kernel virtualization, at least under those kernel virtualizations officially supported, like LXC in lucid.

Especially with the extremely young LXC many guest-specific details are far from clear, i.e. what parts of /dev/ should be bind-mounted, what services not to start or to remove, how to have apt-get update survive dpkg-reconfigure udev, and so on.
Yes, LXC needs to do far more documentation for people other than kernel developers, and yes I kind of dislike both udev and upstart, but the distros should throw in more support, too.

There are several HowTos across the web and it's possible to get it running somehow, but by far not as easy and stable as OpenVZ under hardy. We run >30 hardy guests on CentOS hosts, but at the moment I would'nt dare using lucid.

Is there any chance of getting an "official" Lucid-as-LXC-guest.template?

[Serge Hallyn (serge-hallyn)]

> Is there any chance of getting an "official" Lucid-as-LXC-guest.template?

If you're running a maverick guest, then this exists:
 lxc-create -f lxc_net_conf.conf -t ubuntu -n ubuntu1
will create a working lucid guest by default. I'm therefore marking this bug
'fix released.'

The ubuntu 'template' is really just a shell script that uses debootstrap
to setup the basic system and then tweaks the guest configuration to work
nicely as a lxc guest. Much of it could be reused on a lucid host, but
not all, especially as the lxc guest job control (used in /etc/init/lxc.conf)
does not exist in lxc 0.6.5.

If you are interested in using this on a lucid guest, then I would
recommend sending an email to the lxc-users mailing list, and posting the
result on https://help.ubuntu.com/community/LXC.

[Christoph Lechleitner (lech)]

Excuse me!?
How does a workaround for the next non-LTS release solve this for the current LTS release!?

Besides, telling a few services not to start has nothing to do with a stable setup in the long term, because any update in the area of udev or upstarted will undo part of that and thereby make the guest unusable.

[Paul van Genderen (paulvg)]

You can install the maverick version of lxc using prevu, but good luck getting it to actually work. (At least the busybox template does something when running the container.)

[User Virtual (virtualusr)]

Hi guys

I've spent too much of my time making this work. You can download install demo scripts with minimally patched lxc-ubuntu template script that works perfectly here in PV Xen 4.0.1rc5 guest and remotely on another physical machine. "One click" solution, which should be easy enough to use in coming releases by sed -i 's/lucid/maverick/g' .
Well, easy excluded upstart changes, /dev population and similar. No guarantees here. But LTS Lucid inside LTS Lucid (inside Xen 4.0.* or physical box) flies great.

Regards Virtual User

[User Virtual (virtualusr)]

Here: http://virtualusr.wordpress.com/2010/09/07/flying-linux-containers-on-xen-4-0-on-pv-ubuntu-10-04-lucid/