lxc on Wily cannot dist-upgrade Xenial

[Robie Basak]

This affects 1.1.5-0ubuntu0.15.10.3 on Wily. I believe it also affects 1.1.5-0ubuntu0.15.04.3 on Vivid.

Downgrading to 1.1.4-0ubuntu1 fixes it, so it does not affect 1.1.4-0ubuntu1 on Wily.

My failure case is that I am running adt-run --apt-pocket=proposed -U with adt-virt-lxc and a test needs to upgrade systemd in the container which fails in the postinst. I think this comes down to lxc being unable to dist-upgrade xenial-proposed.

I created the container with: sudo lxc-create -n xenial -t ubuntu-cloud -- -r xenial -T http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz -F

Reproduce with: adt-run -U --apt-pocket=proposed libapache2-mod-perl2 --- adt-virt-lxc -es xenial

Expected: test pass (or fail for unrelated reason)

Actual: test fail due to:

Processing triggers for systemd (228-2ubuntu2) ...
Failed to execute operation: Failed to activate service 'org.freedesktop.systemd1': timed out
dpkg: error processing package systemd (--unpack):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for ufw (0.34-2) ...
Errors were encountered while processing:
 systemd
E: Sub-process /usr/bin/dpkg returned an error code (1)
blame: libapache2-mod-perl2
badpkg: failed to run apt-get to satisfy adt-satdep.deb dependencies
adt-run [15:57:15]: ERROR: erroneous package: failed to run apt-get to satisfy adt-satdep.deb dependencies

I tried to reduce the failure case but I'm not sure how to manually activate a trigger for systemd.

Workaround: downgrade using: sudo apt-get install lxc=1.1.4-0ubuntu1 lxcfs=0.10-0ubuntu2 liblxc1=1.1.4-0ubuntu1 python3-lxc=1.1.4-0ubuntu1

Presumably the failure case will stop working once systemd lands in the xenial release pocket, but perhaps reconfiguring it would still reproduce?

Confirmed since pitti has experienced this as well.

[Robie Basak (racb)]

Correction. Rather than systemd being upgraded, I think there's something that fires a trigger in it that sets off the failure.

[Martin Pitt (pitti)]

I have this on the armhf production autopkgtest machines, but I wasn't aware that it is a general problem. These still run an ancient kernel (3.13) under wily userspace. But it seems it happens under a full wily amd64 too then.

For me I bisected it to downgrading lxcfs. 0.10-0ubuntu2 works, 0.10-0ubuntu2.1 breaks. I also had to downgrade lxc itself (like Robie), but I don't remember ATM why.

FTR, this is all fine on xenial, this just happens on wily. Perhaps wily-updates is missing some accompanying cgmanager fix?

[Martin Pitt (pitti)]

> something that fires a trigger in it that sets off the failure.

The trigger is most likely stuck in "systemctl daemon-reload", which likely exercises some cgroup reconfiguration. In lxc these are being redirected to lxcfs/cgmanager.

[Stéphane Graber (stgraber)]

Sounds like a lxcfs or cgmanager issue, moving to lxcfs for now.

[Serge Hallyn (serge-hallyn)]

Failing to reproduce this.

I do notice that now systemd is also placing itself into init.scope for devices cgroup, which i don't think it did before, but that should be fine (and isn't causing issues here).

[Serge Hallyn (serge-hallyn)]

Let me rephrase - I can't reproduce by hand, even doing apt-get install --reinstall systemd. adt-run does indeed fail.

[Serge Hallyn (serge-hallyn)]

I've tried downgrading systemd and re-installing, but that did not suffice.

[Robie Basak (racb)]

On Mon, Dec 21, 2015 at 07:38:29PM -0000, Serge Hallyn wrote:
> Let me rephrase - I can't reproduce by hand, even doing apt-get install
> --reinstall systemd. adt-run does indeed fail.

I couldn't reproduce that either in the end. Sorry if the report is
contradictory, I made assumptions and then realised they were wrong,
amended my report but missed few places.

My suspicion is that reinstalling systemd doesn't cause the same changes
as whatever is dist-upgrading in the systemd trigger that fires.