newton/trusty lxc package install failure

[Jesse Pretorius]

Today when testing an installation with the current head of stable/newton I hit the following failure:

failed: [rantdsp-65-e84d] (item=[u'apparmor', u'apparmor-utils', u'bridge-utils', u'cgmanager', u'cgroup-lite', u'debootstrap', u'dnsmasq', u'git', u'irqbalance', u'language-pack-en', u'liblxc1', u'lxc', u'lxc-dev', u'lxc-templates', u'python-dev', u'python3-lxc', u'pxz']) =>
{"cache_update_time": 0, "cache_updated": false, "failed": true, "item": ["apparmor", "apparmor-utils", "bridge-utils", "cgmanager", "cgroup-lite", "debootstrap", "dnsmasq", "git", "irqbalance", "language-pack-en", "liblxc1", "lxc", "lxc-dev", "lxc-templates", "python-dev", "python3-lxc", "pxz"], "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'lxc' 'lxc-templates' -t 'trusty-backports'' failed: start: Job failed to start\ninvoke-rc.d: initscript lxc-net, action \"start\" failed.\ndpkg: error processing package lxc1 (--configure):\n subprocess installed post-installation script returned error exit status 1\ndpkg: dependency problems prevent configuration of lxc-templates:\n lxc-templates depends on lxc1 (= 2.0.8-0ubuntu1~14.04.1); however:\n Package lxc1 is not configured yet.\n\ndpkg: error processing package lxc-templates (--configure):\n dependency problems - leaving unconfigured\ndpkg: dependency problems prevent configuration of lxc:\n lxc depends on lxc1 (>= 2.0.8-0ubuntu1~14.04.1); however:\n Package lxc1 is not configured yet.\n\ndpkg: error processing package lxc (--configure):\n dependency problems - leaving unconfigured\nNo apport report written because the error message indicates its a followup error from a previous failure.\nNo apport report written because the error message indicates its a followup error from a previous failure.\nErrors were encountered while processing:\n lxc1\n lxc-templates\n lxc\nE: Sub-process /usr/bin/dpkg returned an error code (1)\n", "stderr": "start: Job failed to start\ninvoke-rc.d: initscript lxc-net, action \"start\" failed.\ndpkg: error processing package lxc1 (--configure):\n subprocess installed post-installation script returned error exit status 1\ndpkg: dependency problems prevent configuration of lxc-templates:\n lxc-templates depends on lxc1 (= 2.0.8-0ubuntu1~14.04.1); however:\n Package lxc1 is not configured yet.\n\ndpkg: error processing package lxc-templates (--configure):\n dependency problems - leaving unconfigured\ndpkg: dependency problems prevent configuration of lxc:\n lxc depends on lxc1 (>= 2.0.8-0ubuntu1~14.04.1); however:\n Package lxc1 is not configured yet.\n\ndpkg: error processing package lxc (--configure):\n dependency problems - leaving unconfigured\nNo apport report written because the error message indicates its a followup error from a previous failure.\nNo apport report written because the error message indicates its a followup error from a previous failure.\nErrors were encountered while processing:\n lxc1\n lxc-templates\n lxc\nE: Sub-process /usr/bin/dpkg returned an error code (1)\n", "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nlxc is already the newest version.\nlxc-templates is already the newest version.\n0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded.\n3 not fully installed or removed.\nAfter this operation, 0 B of additional disk space will be used.\nSetting up lxc1 (2.0.8-0ubuntu1~14.04.1) ...\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "lxc is already the newest version.", "lxc-templates is already the newest version.", "0 upgraded, 0 newly installed, 0 to remove and 88 not upgraded.", "3 not fully installed or removed.", "After this operation, 0 B of additional disk space will be used.", "Setting up lxc1 (2.0.8-0ubuntu1~14.04.1) ..."]}

Trying it manually got me:

# /usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'lxc' 'lxc-templates'
 -t 'trusty-backports'
Reading package lists... Done
Building dependency tree
Reading state information... Done
lxc is already the newest version.
lxc-templates is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 44 not upgraded.
3 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up lxc1 (2.0.8-0ubuntu1~14.04.1) ...
start: Job failed to start
invoke-rc.d: initscript lxc-net, action "start" failed.
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          No apport report written because the error message indica
tes its a followup error from a previous failure.
                                                 dpkg: error processing package lxc1 (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of lxc-templates:
 lxc-templates depends on lxc1 (= 2.0.8-0ubuntu1~14.04.1); however:
  Package lxc1 is not configured yet.

dpkg: error processing package lxc-templates (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of lxc:
 lxc depends on lxc1 (>= 2.0.8-0ubuntu1~14.04.1); however:
  Package lxc1 is not configured yet.

dpkg: error processing package lxc (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 lxc1
 lxc-templates
 lxc
E: Sub-process /usr/bin/dpkg returned an error code (1)

# apt-cache policy lxc
lxc:
  Installed: 2.0.8-0ubuntu1~14.04.1
  Candidate: 2.0.8-0ubuntu1~14.04.1
  Version table:
 *** 2.0.8-0ubuntu1~14.04.1 0
        100 http://rackspace.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.7-0ubuntu1~16.04.2~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ trusty-updates/mitaka/main amd64 Packages
     1.0.10-0ubuntu1.1 0
        500 http://rackspace.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.0.3-0ubuntu3 0
        500 http://rackspace.clouds.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

from /var/log/apt/history.log:

Start-Date: 2017-08-16 12:30:55
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install lxc lxc-templates -t trusty-backports
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2017-08-16 12:30:56

from /var/log/upstart/lxc-net.log:

dnsmasq: failed to create listening socket for 10.0.3.1: Address already in use
Failed to setup lxc-net.
Failed to setup lxc-net.
lxc-net isn't running

I tried doing the following and the install worked as expected:
echo manual > /etc/init/lxc-net.override

Then I realised that dnsmasq should not actually be running in the first place. It's used by neutron and LXC but they both start it up in their own way, so there should be no need for dnsmasq to be running on its own.

I confirmed that it was running:

# ps -ef | grep dns
dnsmasq 1754 1 0 13:02 ? 00:00:00 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpk
g-dist,.dpkg-old,.dpkg-new

Then removed the lxc-net override and put a dnsmasq override in place instead:

# rm -f /etc/init/lxc-net.override
# echo manual > /etc/init/dnsmasq.override
# apt-get purge -y lxc* dnsmasq
# apt-get autoremove -y

I then re-installed LXC again and it worked as expected. Except this time lxc-net had started a dnsmasq instance of its own, which is looks more correct.

# ps -ef | grep dns
lxc-dns+ 18396 1 0 13:22 ? 00:00:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address 10.0.3.1 -
-dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leas
es --dhcp-authoritative

I think what needs to happen is that the package dnsmasq (which contains the upstart/systemd init scripts and config as a forwarder) should be removed or replaced with just 'dnsmasq-base' for Ubuntu (which only has the required binaries). This will resolve the conflict between the services and allow the package to install without further workarounds/masks.

It may also make sense to revisit our masking of lxc-net and all the tasks/config associated to see if we can't simplify it to use the service as packaged.

[Jesse Pretorius (jesse-pretorius)]

Patch in progress: https://review.openstack.org/494212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494212
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=d5bef82dfedc72eb2d51ef899578098c35738085
Submitter: Jenkins
Branch: master

commit d5bef82dfedc72eb2d51ef899578098c35738085
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 16 14:43:45 2017 +0100

    Ubuntu: Use dnsmasq-base, not dnsmasq

    In Ubuntu the 'dnsmasq' package actually includes
    init scripts and service configuration which
    conflict with LXC and are best not included. The
    actual dependent package is 'dnsmasq-base'.

    With this change, the cleanup is no longer
    required because the files are removed by the
    purge or not installed in the first place.

    Closes-Bug: #1711140
    Change-Id: I0b0a8dec3943651f379479ea1a3b8038736b9219

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/494266

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/494267

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494266
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=35cb1ffd23b194491cde02312d3066362d9e5c13
Submitter: Jenkins
Branch: stable/ocata

commit 35cb1ffd23b194491cde02312d3066362d9e5c13
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 16 14:43:45 2017 +0100

    Ubuntu: Use dnsmasq-base, not dnsmasq

    In Ubuntu the 'dnsmasq' package actually includes
    init scripts and service configuration which
    conflict with LXC and are best not included. The
    actual dependent package is 'dnsmasq-base'.

    With this change, the cleanup is no longer
    required because the files are removed by the
    purge or not installed in the first place.

    Closes-Bug: #1711140
    Change-Id: I0b0a8dec3943651f379479ea1a3b8038736b9219
    (cherry picked from commit d5bef82dfedc72eb2d51ef899578098c35738085)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494267
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=09a9ff6297038f4eba49c4cac31ca78f4abb9d51
Submitter: Jenkins
Branch: stable/newton

commit 09a9ff6297038f4eba49c4cac31ca78f4abb9d51
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 16 14:43:45 2017 +0100

    Ubuntu: Use dnsmasq-base, not dnsmasq

    In Ubuntu the 'dnsmasq' package actually includes
    init scripts and service configuration which
    conflict with LXC and are best not included. The
    actual dependent package is 'dnsmasq-base'.

    With this change, the cleanup is no longer
    required because the files are removed by the
    purge or not installed in the first place.

    Closes-Bug: #1711140
    Change-Id: I0b0a8dec3943651f379479ea1a3b8038736b9219
    (cherry picked from commit d5bef82dfedc72eb2d51ef899578098c35738085)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/495346

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/495346
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=af36d8458dee92ef94c7b544dcc2c5894024e233
Submitter: Jenkins
Branch: stable/pike

commit af36d8458dee92ef94c7b544dcc2c5894024e233
Author: Jesse Pretorius <email address hidden>
Date: Wed Aug 16 14:43:45 2017 +0100

    Ubuntu: Use dnsmasq-base, not dnsmasq

    In Ubuntu the 'dnsmasq' package actually includes
    init scripts and service configuration which
    conflict with LXC and are best not included. The
    actual dependent package is 'dnsmasq-base'.

    With this change, the cleanup is no longer
    required because the files are removed by the
    purge or not installed in the first place.

    Closes-Bug: #1711140
    Change-Id: I0b0a8dec3943651f379479ea1a3b8038736b9219
    (cherry picked from commit d5bef82dfedc72eb2d51ef899578098c35738085)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts 15.1.8 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts 14.2.8 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts 16.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-lxc_hosts 17.0.0.0b1 development milestone.