During do-release-upgrade from 18.04 to 18.10: package lxd 3.0.2-0ubuntu1~18.04.1 failed to install/upgrade: new lxd package pre-installation script subprocess returned error exit status 1

[Karl Stenerud]

[Impact]

Upgrading apt version of lxd to snap version during do-release-upgrade failed.

I attempted to update my ubuntu server 18.04 system to 18.10 as root:

apt update
apt dist-upgrade -y
apt autoremove -y
do-release-upgrade

I have an existing LXD installed via apt, and running containers.

It asked me which snap LXD to upgrade to, and I selected "latest".

Console log:

==> Installing the LXD snap from the latest track for ubuntu-18.10
error: cannot perform the following tasks:
- Run install hook of "lxd" snap if present (run hook "install": cannot perform operation: mount /var/lib/snapd/hostfs/var/lib/lxd /var/lib/lxd -o nosuid,nodev,noexec,rbind,rslave: Permission denied)
dpkg: error processing archive /tmp/apt-dpkg-install-gdCSiW/32-lxd_1%3a0.4_all.deb (--unpack):
 new lxd package pre-installation script subprocess returned error exit status 1
Traceback (most recent call last):
                                    File "/usr/lib/python3/dist-packages/apt/cache.py", line 265, in __getitem__
                                                                                                                    rawpkg = self._cache[key]
                                                                                                                                             KeyError: '32-lxd'

                                                                                                                                                               During handling of the above exception, another exception occurred:

              Traceback (most recent call last):
                                                  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 126, in _apt_pkg
                                                                                                                                           return self._cache()[package]
                                                                                                                                                                          File "/usr/lib/python3/dist-packages/apt/cache.py", line 267, in __getitem__
                                      raise KeyError('The cache has no package named %r' % key)
                                                                                               KeyError: "The cache has no package named '32-lxd'"

                                                                                                                                                  During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
                                     File "/usr/share/apport/package_hook", line 48, in <module>
                                                                                                    pr['SourcePackage'] = apport.packaging.get_source(options.package)
                                                                                                                                                                        File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 159, in get_source
                                               if self._apt_pkg(package).installed:
                                                                                     File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 128, in _apt_pkg
                                                                                                                                                                              raise ValueError('package %s does not exist' % package)
                 ValueError: package 32-lxd does not exist
                                                          Preparing to unpack .../33-open-iscsi_2.0.874-5ubuntu9.1_amd64.deb ...
Unpacking open-iscsi (2.0.874-5ubuntu9.1) over (2.0.874-5ubuntu2.4) ...
Preparing to unpack .../34-poppler-data_0.4.9-2_all.deb ...
Unpacking poppler-data (0.4.9-2) over (0.4.8-2) ...
Preparing to unpack .../35-tasksel-data_3.34ubuntu12_all.deb ...
Unpacking tasksel-data (3.34ubuntu12) over (3.34ubuntu11) ...
Preparing to unpack .../36-tasksel_3.34ubuntu12_all.deb ...
Unpacking tasksel (3.34ubuntu12) over (3.34ubuntu11) ...
Preparing to unpack .../37-libsepol1_2.8-1_amd64.deb ...
Unpacking libsepol1:amd64 (2.8-1) over (2.7-1) ...
Errors were encountered while processing:
 /tmp/apt-dpkg-install-gdCSiW/32-lxd_1%3a0.4_all.deb
Exception during pm.DoInstall(): E:Sub-process /usr/bin/dpkg returned an error code (1)

*** Send problem report to the developers?

There also appears to be a secondary bug in the error report collection software, causing it to process an illegal UTF-8 sequence:

*** Collecting problem information

The collected information can be sent to the developers to improve the
application. This might take a few minutes.
.....ERROR: hook /usr/share/apport/general-hooks/ubuntu.py crashed:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport/report.py", line 198, in _run_hook
    symb['add_info'](report, ui)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 89, in add_info
    check_attachment_for_errors(report, attachment)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 208, in check_attachment_for_errors
    trim_dpkg_log(report)
  File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in trim_dpkg_log
    report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
  File "/usr/share/apport/general-hooks/ubuntu.py", line 508, in <listcomp>
    report['DpkgTerminalLog'] = '\n'.join([str(line.decode('utf-8')) for line in lines])
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 14333-14334: invalid continuation byte

I'm afraid I don't know which software package this second error falls under.

[Test Case]

I'm not sure about this, but I think it may have to do with me already bind-mounting /var/lib/lxd off the boot drive because it's too small to support the data (using https://github.com/kstenerud/ubuntu-dev-installer/blob/master/map-vm-paths.sh). The exact sequence of commands I used to trigger the bug are:

apt update
apt dist-upgrade -y
apt autoremove -y
do-release-upgrade

[Regression Potential]

TBD

ProblemType: Package
DistroRelease: Ubuntu 18.10
Package: lxd 3.0.2-0ubuntu1~18.04.1
ProcVersionSignature: Ubuntu 4.15.0-39.42-generic 4.15.18
Uname: Linux 4.15.0-39-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
Date: Tue Dec 11 09:25:57 2018
ErrorMessage: new lxd package pre-installation script subprocess returned error exit status 1
InstallationDate: Installed on 2018-03-23 (262 days ago)
InstallationMedia: Ubuntu-Server 18.04 LTS "Bionic Beaver" - Alpha amd64 (20180322.1)
Python3Details: /usr/bin/python3.6, Python 3.6.7, python3-minimal, 3.6.7-1~18.10
PythonDetails: /usr/bin/python2.7, Python 2.7.15+, python-minimal, 2.7.15-3
RelatedPackageVersions:
 dpkg 1.19.0.5ubuntu5
 apt 1.7.0
SourcePackage: lxd
Title: package lxd 3.0.2-0ubuntu1~18.04.1 failed to install/upgrade: new lxd package pre-installation script subprocess returned error exit status 1
UpgradeStatus: Upgraded to cosmic on 2018-12-11 (0 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stéphane Graber (stgraber)]

Moving over to snapd as the failure actually hits before any of the LXD migration code runs.

This shows a failure in setting up the environment for the configure hook, likely related to snapd passing /var/lib/lxd into the mount namespace. I don't know if this is still a thing that it does, but if it is, then this bug shows a potential issue with the current approach.

[Seth Arnold (seth-arnold)]

I added apport for the python2 -> python3 bug.

Thanks

[Brian Murray (brian-murray)]

What version of apport did you have installed? This should have been fixed when bug 1766337 was fixed.