LXC containers fail to start when upgraded to 14.04 LTS

[Sean Clarke]

After running 14.04 LTS through beta etc. I updated to teh latest this morning and all was well. I then went into a running container, changed it's sources.list to pint to the trusty repos and upgraded (the container itself was running 13.10).

All went well, I restart the container and it hangs on a restart.

When I start it in the forground it just hangs :

sudo lxc-start -n <container name>
<4>init: plymouth-upstart-bridge main process (5) terminated with status 1
<4>init: plymouth-upstart-bridge main process ended, respawning
<4>init: plymouth-upstart-bridge main process (15) terminated with status 1
<4>init: plymouth-upstart-bridge main process ended, respawning
<4>init: ureadahead main process (8) terminated with status 5
<4>init: plymouth-upstart-bridge main process (19) terminated with status 1
<4>init: plymouth-upstart-bridge main process ended, respawning
 * Stopping Send an event to indicate plymouth is up ...done.
 * Starting Mount filesystems on boot ...done.
 * Starting Signal sysvinit that the rootfs is mounted ...done.
 * Starting Clean /tmp directory ...done.
 * Stopping Clean /tmp directory ...done.
 * Starting Populate and link to /run filesystem ...done.
 * Starting Track if upstart is running in a container ...done.
 * Stopping Populate and link to /run filesystem ...done.
 * Starting load fallback graphics devices ...done.
 * Starting workaround for missing events in container ...done.
 * Stopping workaround for missing events in container ...done.
<4>init: udev-fallback-graphics main process (79) terminated with status 1
 * Starting load fallback graphics devices ...fail!
 * Starting configure network device security ...done.
 * Starting set console font ...done.
<4>init: console-font main process (113) terminated with status 1
 * Starting set console font ...fail!
 * Starting userspace bootsplash ...done.
 * Starting Initialize or finalize resolvconf ...done.
<4>init: setvtrgb main process (126) terminated with status 1
 * Stopping userspace bootsplash ...done.
 * Starting Send an event to indicate plymouth is up ...done.
 * Starting Mount network filesystems ...done.
 * Stopping Send an event to indicate plymouth is up ...done.
 * Stopping Mount network filesystems ...done.
 * Starting Bridge socket events into upstart ...done.
 * Starting configure network device ...done.

<hangs forever>

This is in the host syslog:

Apr 14 11:48:19 sentinel kernel: [59143.539884] device vethJQQMQG entered promiscuous mode
Apr 14 11:48:19 sentinel kernel: [59143.609423] IPv6: ADDRCONF(NETDEV_CHANGE): vethJQQMQG: link becomes ready
Apr 14 11:48:19 sentinel kernel: [59143.609526] br0: port 2(vethJQQMQG) entered forwarding state
Apr 14 11:48:19 sentinel kernel: [59143.609539] br0: port 2(vethJQQMQG) entered forwarding state
Apr 14 11:48:20 sentinel kernel: [59144.885192] type=1400 audit(1397472500.484:38): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/pstore/" pid=8430 comm="mount" fstype="pstore" srcname="none" flags="rw"
Apr 14 11:48:20 sentinel kernel: [59144.885349] type=1400 audit(1397472500.484:39): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/pstore/" pid=8430 comm="mount" fstype="pstore" srcname="none" flags="ro"
Apr 14 11:48:22 sentinel ntpd[2085]: Listen normally on 18 vethJQQMQG fe80::fc93:63ff:feb1:3c5e UDP 123
Apr 14 11:48:22 sentinel ntpd[2085]: peers refreshed
Apr 14 11:48:22 sentinel ntpd[2085]: new interface(s) found: waking up resolver

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: lxc 1.0.3-0ubuntu2
ProcVersionSignature: Ubuntu 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu2
Architecture: amd64
Date: Mon Apr 14 11:49:45 2014
InstallationDate: Installed on 2012-09-05 (585 days ago)
InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Release amd64 (20120424.1)
SourcePackage: lxc
UpgradeStatus: Upgraded to trusty on 2014-02-08 (64 days ago)
defaults.conf:
 lxc.network.type = veth
 lxc.network.link = br0
 lxc.network.flags = up
modified.conffile..etc.lxc.default.conf: [modified]
mtime.conffile..etc.lxc.default.conf: 2013-03-07T17:09:54.680483

[Sean Clarke (sean-clarke)]

When teh container is hung and i execute an lxc-stop (from a different terminal) I see a:

mount: cannot mount block device <nfs server>/export/lxc-lib/<container name>/rootfs read-only

Might be connected?

[Stéphane Graber (stgraber)]

So the problem here, as is shown by your kernel log is that your container even though the rootfs upgraded properly is still using an old config...

That old config doesn't mount pstore which is then causing the hang at boot time which you are seeing.

LXC 1.0 fixes that issue going forward through a set of config includes, but for old containers, you may have to update their config by hand to be closer to what you'd get if you just created a new container.

So edit /var/lib/lxc/<container>/config and make it look like something like that:
"""
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: -d ubuntu -r trusty -a amd64
# For additional config options, please look at lxc.conf(5)

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.arch = x86_64

# Container specific configuration
lxc.rootfs = /var/lib/lxc/YOUR-CONTAINER-NAME/rootfs
lxc.utsname = YOUR-CONTAINER-NAME

# Network configuration
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.network.hwaddr = YOUR-CONTAINER-MAC-ADDRESS
"""

Then start it again and things should be back to normal.

There's sadly no way we can programmatically do that for existing containers on upgrade due to all the possible combinations and hand made changes, but as I said, the new lxc.includes we added should let us cover most of those cases moving forward and hopefully we won't have a similar problem with 16.04.

[Sean Clarke (sean-clarke)]

Yes, that works!

Many thanks for your help.