Upgrade from xenial to yakkety fails due to a UnicodeDecodeError

Bug #1614576 reported by Max Brustkern on 2016-08-18
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ubuntu-release-upgrader (Ubuntu)
High
Brian Murray
Xenial
High
Brian Murray

Bug Description

Test Case
---------
1) Install the package shim-signed on an Ubuntu 14.04 system
2) Run do-release-upgrade -f DistUpgradeViewNonInteractive
3) Observe the following Traceback:

Setting up shim-signed (1.18~16.04.1+0.8-0ubuntu2) ...
find: Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 15, in <module>
    import xml.dom, xml.dom.minidom
ImportError: No module named 'xml.dom'

With the version of ubuntu-release-upgrader from -proposed you will not encounter the Traceback.

Regression Potential
--------------------
The change modifies the code so that errors are ignored when decoding output, so some messages may be less readable. However, this is better than having the release upgrader crash and the upgrade fail.

Original Description
--------------------
When doing an automated upgrade from xenial to yakkey, I get:
Preparing to unpack .../policykit-1_0.105-16_amd64.deb ...
Created symlink /run/systemd/system/polkitd.service auto-upgrade [10:48:43]: ERROR: ERROR: Something went wrong with the upgrade.

ProblemType: BugDistroRelease: Ubuntu 16.10
Package: policykit-1 0.105-16
ProcVersionSignature: User Name 4.4.0-34.53-generic 4.4.15
Uname: Linux 4.4.0-34-generic x86_64
ApportVersion: 2.20.3-0ubuntu7
Architecture: amd64
Date: Thu Aug 18 11:02:06 2016
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bashSourcePackage: policykit-1
UpgradeStatus: Upgraded to yakkety on 2016-08-18 (0 days ago)

Related branches

Max Brustkern (nuclearbob) wrote :
Max Brustkern (nuclearbob) wrote :

This only appears to occur on unattended upgrades. Additional logs are attached, and I'll attach some more shortly.

Max Brustkern (nuclearbob) wrote :
Max Brustkern (nuclearbob) wrote :
Download full text (8.0 KiB)

The end of /var/log/dist-upgrade/main.log is showing a unicode decode error:
2016-09-09 14:21:30,594 DEBUG linux-image-4.4.0-9136-generic (new-install) added with 45669149 to boot space
2016-09-09 14:21:31,638 DEBUG dir '/var/cache/apt/archives' needs '180762526' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f94752ca240>' (21087518720.000000)
2016-09-09 14:21:31,638 DEBUG dir '/usr' needs '354729984' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f94752ca240>' (20906756194.000000)
2016-09-09 14:21:31,639 DEBUG dir '/usr' needs '52428800' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f94752ca240>' (20552026210.000000)
2016-09-09 14:21:31,639 DEBUG dir '/boot' needs '91338298' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f94752ca240>' (20499597410.000000)
2016-09-09 14:21:31,639 DEBUG dir '/tmp' needs '5242880' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f94752ca240>' (20408259112.000000)
2016-09-09 14:21:31,639 DEBUG dir '/' needs '10485760' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f94752ca240>' (20403016232.000000)
2016-09-09 14:21:31,640 DEBUG dir '/tmp' needs '0.0' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f94752ca240>' (20392530472.000000)
2016-09-09 14:21:31,640 DEBUG dir '/usr' needs '0.0' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f94752ca240>' (20392530472.000000)
2016-09-09 14:21:31,643 DEBUG demoted: 'makedev'
2016-09-09 14:21:31,643 DEBUG found components: {'yakkety-security': set(), 'yakkety': {'multiverse', 'universe', 'restricted', 'main'}, 'yakkety-updates': {'multiverse', 'universe', 'restricted', 'main'}
}
2016-09-09 14:21:31,810 DEBUG toinstall: '['btrfs-progs', 'dirmngr', 'gnupg-agent', 'gnupg-l10n', 'libassuan0', 'libicu57', 'libip4tc0', 'libip6tc0', 'libiptc0', 'libjson-c3', 'libksba8', 'libnpth0', 'libpng16-16', 'libprocps6', 'linux-headers-4.4.0-9136', 'linux-headers-4.4.0-9136-generic', 'linux-image-4.4.0-9136-generic', 'linux-image-extra-4.4.0-9136-generic', 'pinentry-curses', 'python3-crypto', 'python3-distro-info', 'python3-httplib2', 'python3-keyring', 'python3-keyrings.alt', 'python3-launchpadlib', 'python3-lazr.restfulclient', 'python3-lazr.uri', 'python3-oauth', 'python3-secretstorage', 'python3-setuptools', 'python3-simplejson', 'python3-wadllib', 'python3-zope.interface']'
2016-09-09 14:21:31,811 DEBUG toupgrade: '['apparmor', 'apport', 'apt', 'apt-transport-https', 'apt-utils', 'base-files', 'base-passwd', 'bash', 'bash-completion', 'bind9-host', 'bsdutils', 'btrfs-tools', 'busybox-initramfs', 'busybox-static', 'byobu', 'bzip2', 'cloud-guest-utils', 'cloud-initramfs-copymods', 'cloud-initramfs-dyn-netconf', 'console-setup', 'console-setup-linux', 'cryptsetup-bin', 'curl', 'dash', 'dbus', 'debconf', 'debconf-i18n', 'debianutils', 'dh-python', 'distro-info-data', 'dmidecode', 'dmsetup', 'dnsmasq-base', 'dnsutils', 'dosfstools', 'dpkg', 'e2fslibs', 'e2fsprogs', ...

Read more...

Brian Murray (brian-murray) wrote :

It's not really a bug with policykit-1, its just that the upgrader trips over this specific package.

affects: policykit-1 (Ubuntu) → ubuntu-release-upgrader (Ubuntu)
Changed in ubuntu-release-upgrader (Ubuntu):
importance: Undecided → High
summary: - Upgrade from xenial to yakkety fails when install policykit-1
+ Upgrade from xenial to yakkety fails due to a UnicodeDecodeError
Brian Murray (brian-murray) wrote :

For future reference one annoying thing about the non-interactive version of the upgrader is that it has its own exception handler e.g.:

class DistUpgradeViewNonInteractive(DistUpgradeView):
    " non-interactive version of the upgrade view "
    def __init__(self, datadir=None, logdir=None):
        DistUpgradeView.__init__(self)
        self.config = DistUpgradeConfig(".")
        self._acquireProgress = NonInteractiveAcquireProgress()
        self._installProgress = NonInteractiveInstallProgress(logdir)
        self._opProgress = apt.progress.base.OpProgress()
        sys.__excepthook__ = self.excepthook

Disabling sys.__excepthook__ allowed the real error, UnicodeDecodeError, to surface.

Brian Murray (brian-murray) wrote :

Using errors='ignore' allows the upgrade to complete and doesn't seem too horrible given that's its non-interactive anyway.

               sys.stdout.write("%s" % s.decode(locale.getpreferredencoding(), errors='ignore'))

We can see the errors in the terminal output:

Preparing to unpack .../inputattach_1%3a1.5.1-2_amd64.deb ...
Unpacking inputattach (1:1.5.1-2) over (1:1.4.9-1) ...

Notice the ':' vs '%3a'. The same errors also appear in the log - /var/log/dist-upgrade/main.log.

Changed in ubuntu-release-upgrader (Ubuntu):
status: New → Triaged
Brian Murray (brian-murray) wrote :

Here's the full Traceback:

Preparing to unpack .../policykit-1_0.105-16_amd64.deb ...
Created symlink /run/systemd/system/polkitd.service Traceback (most recent call last):
  File "./yakkety", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-uavelnl5/DistUpgrade/DistUpgradeMain.py", line 242, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-uavelnl5/DistUpgrade/DistUpgradeController.py", line 1880, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-uavelnl5/DistUpgrade/DistUpgradeController.py", line 1845, in fullUpgrade
    if not self.doDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-uavelnl5/DistUpgrade/DistUpgradeController.py", line 1183, in doDistUpgrade
    res = self.cache.commit(fprogress,iprogress)
  File "/tmp/ubuntu-release-upgrader-uavelnl5/DistUpgrade/DistUpgradeCache.py", line 267, in commit
    apt.Cache.commit(self, fprogress, iprogress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 515, in commit
    res = self.install_archives(pm, install_progress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 479, in install_archives
    res = install_progress.run(pm)
  File "/tmp/ubuntu-release-upgrader-uavelnl5/DistUpgrade/DistUpgradeView.py", line 234, in run
    res = os.WEXITSTATUS(self.wait_child())
  File "/tmp/ubuntu-release-upgrader-uavelnl5/DistUpgrade/DistUpgradeView.py", line 200, in wait_child
    self.update_interface()
  File "/tmp/ubuntu-release-upgrader-uavelnl5/DistUpgrade/DistUpgradeViewNonInteractive.py", line 229, in update_interface
    sys.stdout.write("%s" % s.decode(locale.getpreferredencoding()))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 0: unexpected end of data

Brian Murray (brian-murray) wrote :

Looking a /var/log/dist-upgrade/main.log though maybe its the arrow:

reparing to unpack .../policykit-1_0.105-16_amd64.deb ...^M
Created symlink /run/systemd/system/polkitd.service → /dev/null.^M
Unpacking policykit-1 (0.105-16) over (0.105-14.1) ...^M

Changed in ubuntu-release-upgrader (Ubuntu):
assignee: nobody → Brian Murray (brian-murray)
status: Triaged → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-release-upgrader - 1:16.10.5

---------------
ubuntu-release-upgrader (1:16.10.5) yakkety; urgency=medium

  * DistUpgradeViewNonInteractive.py: Instead of tripping over errors when
    writing progress to stdout just ignore them so that the upgrade can
    continue. (LP: #1614576)

 -- Brian Murray <email address hidden> Wed, 05 Oct 2016 10:40:39 -0700

Changed in ubuntu-release-upgrader (Ubuntu):
status: In Progress → Fix Released
Changed in ubuntu-release-upgrader (Ubuntu Xenial):
importance: Undecided → High
assignee: nobody → Brian Murray (brian-murray)
status: New → In Progress
description: updated
description: updated

Hello Max, or anyone else affected,

Accepted ubuntu-release-upgrader into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:16.04.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ubuntu-release-upgrader (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
Max Brustkern (nuclearbob) wrote :

My tests, including the ones that involve upgrading systems running UEFI, are working at the moment. I'll comment on the other bug so eole-team can see it.

Brian Murray (brian-murray) wrote :

I followed the test case and upgraded from Trusty to Xenial without an issue.

2016-11-14 11:16:21,125 INFO release-upgrader version '16.04.18' started
2016-11-14 11:16:21,141 INFO locale: 'en_US' 'UTF-8'
2016-11-14 11:16:21,188 DEBUG setting up environ for non-interactive use
...
2016-11-14 11:39:55,996 DEBUG quirks: running PostCleanup
2016-11-14 11:39:55,996 DEBUG running Quirks.PostCleanup
2016-11-14 11:39:55,996 DEBUG Running PostInstallScript: './xorg_fix_proprietary.py'
2016-11-14 11:39:56,011 DEBUG confirmRestart() called

tags: added: verification-done
removed: verification-needed

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-release-upgrader - 1:16.04.18

---------------
ubuntu-release-upgrader (1:16.04.18) xenial; urgency=medium

  * DistUpgrade/ReleaseNotesViewerWebkit.py: fix indentation of function
    _on_navigation_policy_decision_requested. (LP: #1635155)
  * DistUpgradeViewNonInteractive.py: Instead of tripping over errors when
    writing progress to stdout just ignore them so that the upgrade can
    continue. (LP: #1614576)

 -- Brian Murray <email address hidden> Tue, 08 Nov 2016 09:55:26 -0800

Changed in ubuntu-release-upgrader (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers