changes in >= 234-2ubuntu7 for artful breaks kubuntu CI image build in docker

[Rik Mills]

Our live image build in kubuntu CI using docker, now fails since changes in 234-2ubuntu7 onwards.

log: https://kci.pangea.pub/job/iso_artful_stable_amd64/4/console

14:52:28 ln: cannot remove '/etc/resolv.conf': Device or resource busy
14:52:28 dpkg: error processing package systemd (--configure):
14:52:28 subprocess installed post-installation script returned error exit status 1

234-2ubuntu7 changes: https://launchpad.net/ubuntu/+source/systemd/234-2ubuntu7

  * Always setup /etc/resolv.conf on new installations.
    On new installations, /etc/resolv.conf will always exist. Move it to /run
    and replace it with the desired final symlink. (LP: #1712283)
  * Create /etc/resolv.conf on resolved start, if it is an empty file.

I doubt eventually we will be the only ones to hit the issue.

[Steve Langasek (vorlon)]

Has this still been an issue since resolvconf and ifupdown were re-promoted to Priority: important?

[Rik Mills (rikmills)]

No, builds are still failing with the same issue.

[Dimitri John Ledkov (xnox)]

I'm not sure how to fix this, or rather how to make src:systemd match what src:resolvconf used to do.

In the end the system should end up with /etc/resolv.conf pointing at /run/systemd/resolve/stub-resolv.conf. Yet from the log, it seems that debootstrap (?!) is running, however, one is not allowed to overwrite /etc/resolv.conf.

Do you have a copy/result of older deboostraps? Do they end up having a valid /etc/resolv.conf?

[Rik Mills (rikmills)]

A bit old, as we were not able to run the builds for artful while we had the Qt 5.9 landing PPA enabled to test build our packages against.

https://kci.pangea.pub/job/iso_artful_stable_amd64/3/consoleFull

http://kci.pangea.pub/images/iso_artful_stable_amd64/20170722-2012/

[Dimitri John Ledkov (xnox)]

Inside the old kubuntu-ci image I see:

# ls -latr squashfs-root/etc/resolv.conf
lrwxrwxrwx 1 root root 29 Aug 29 12:54 squashfs-root/etc/resolv.conf -> ../run/resolvconf/resolv.conf

I am confused how resolvconf managed to replace /etc/resolv.conf, yet systemd did not.
Do you somewhere later modify or update /etc/resolv.conf? or is it never touched manually?

[Philip Muškovac (yofel)]

For clarification, the environment the containers run with is:

privileged: false,
cap_add: ['SYS_ADMIN'],
security_opts: ['apparmor:unconfined']

(see https://git.launchpad.net/~kubuntu-ci-admins/kubuntu-ci/+git/pangea-tooling/tree/kci/imager.rb)

what's not helpful is that running debootstrap in a container started on the shell with
run --cap-add SYS_ADMIN --privileged=false --security-opt 'apparmor:unconfined'
seems to work fine... (result: artful/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf)

so this might be related to some of the environment setup before live-build starts running - or that fact that it's running headless, but I did not have time to take a closer look at that.

As for touching resolv.conf, live-build does mess with it later on in some way during the chroot build, but that happens far later during the build.

[Philip Muškovac (yofel)]

I did manage to reproduce this after all.

First thing: 234-2ubuntu10 indeed solves the problem in my manual test at least.

All you need to do to reproduce this is start an artful container:
docker run -ti ubuntu:artful bash
then install systemd in it:
Setting up systemd (234-2ubuntu9) ...
ln: cannot remove '/etc/resolv.conf': Device or resource busy

which I'm not really suprised about as the file is managed by docker. Debootstrap itself isn't directly affected by this when you run it yourself as it creates its own resolv.conf in the chroot, so I guess live-build had it's hand in this failure.

In the builds for older releases this was a non-issue because we never install 'resolvconf' during the build, once you try to do this in the docker container itself, it crashes the same way.

Xenial:
Setting up resolvconf (1.78ubuntu4) ...
ln: cannot remove '/etc/resolv.conf': Device or resource busy
dpkg: error processing package resolvconf (--configure):

so what happened here is really just that systemd started doing something it wasn't responsible for in the past.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 234-2ubuntu10

---------------
systemd (234-2ubuntu10) artful; urgency=medium

  * Do not fail debootstrap if /etc/resolv.conf is immutable. (LP: #1713212)
  * Revert "Create /etc/resolv.conf on resolved start, if it is an empty file."
    As it is ineffective, and correct creation of /etc/resolv.conf has been fixed.
    This reverts commit ccba42504f216f6ffbc54eb2c9af347355f8d86b.
  * initramfs-tools: trigger udevadm add actions with subsystems first.
    This updates the initramfs-tools init-top udev script to trigger udevadm
    actions with type specified. This mimicks the
    systemd-udev-trigger.service. Without type specified only devices are
    triggered, but triggering subsystems may also be required and should happen
    before triggering the devices. This is the case for example on s390x with zdev
    generated udev rules. (LP: #1713536)

 -- Dimitri John Ledkov <email address hidden> Wed, 30 Aug 2017 11:22:41 +0100