Unicode decode error in noninteractive frontend on t->x

[Max Brustkern]

I've got a number of automatic upgrade tests running daily, which all use the noninteractive frontend. As of a few days ago, the trusty->xenial amd64 desktop qemu uefi upgrade testing is having problems:

Setting up sane-utils (1.0.25+git20150528-1ubuntu2) ...
Installing new version of config file /etc/default/saned ...
Installing new version of config file /etc/init.d/saned ...
Test for running scanbd
Move homedir from /home/saned to /var/lib/saned
 * saned disabled; edit /etc/default/saned
Setting up shim-signed (1.17~16.04.1+0.8-0ubuntu2) ...
Installing for x86_64-efi platform.
Installation finished. No error reported.
find: Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 15, in <module>
    import xml.dom, xml.dom.minidom
ImportError: No module named 'xml.dom'

Original exception was:
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/xenial", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/DistUpgrade/DistUpgradeMain.py", line 242, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/DistUpgrade/DistUpgradeController.py", line 1876, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/DistUpgrade/DistUpgradeController.py", line 1841, in fullUpgrade
    if not self.doDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/DistUpgrade/DistUpgradeController.py", line 1182, in doDistUpgrade
    res = self.cache.commit(fprogress,iprogress)
  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/DistUpgrade/DistUpgradeCache.py", line 267, in commit
    apt.Cache.commit(self, fprogress, iprogress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 501, in commit
    # which is less than optimal!
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 465, in install_archives

  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/DistUpgrade/DistUpgradeView.py", line 234, in run
    res = os.WEXITSTATUS(self.wait_child())
  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/DistUpgrade/DistUpgradeView.py", line 200, in wait_child
    self.update_interface()
  File "/tmp/ubuntu-release-upgrader-xv9kbqfl/DistUpgrade/DistUpgradeViewNonInteractive.py", line 229, in update_interface
    sys.stdout.write("%s" % s.decode(locale.getpreferredencoding()))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 0: unexpected end of data
auto-upgrade [16:53:31]: ERROR: ERROR: Something went wrong with the upgrade.

I can get more output if necessary. This problem doesn't appear to occur on other trusty->xenial amd64 desktop qemu installs, so I'm uncertain if it's related to a package installed to support the UEFI boot or some other difference between that and the other images.

[Brian Murray (brian-murray)]

Line 229 of DistUpgradeViewNonInteractive.py was changed due to bug https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1538880.

[Max Brustkern (nuclearbob)]

My most recent upgrade tests have passed, FWIW.

[Brian Murray (brian-murray)]

Let's set this to Incomplete then and wait for any additional failures.

[Max Brustkern (nuclearbob)]

Sounds good!

[Max Brustkern (nuclearbob)]

Here's an apport report from a system where an interactive upgrade stalled. I can work on getting a noninteractive one there as well.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[EOLE team (eole-team)]

We have a hundred trusty servers with the same configuration and need to upgrade them without question.
The "do-release-upgrade -m server -f DistUpgradeViewNonInteractive" always ends with the traceback written in the Bug Description.
Is there a workaround for this bug or an another way to upgrade without question ?

[Max Brustkern (nuclearbob)]

Is your traceback the same as the one I originally posted? If not, could you upload it so we can see what package might be generating the problem?

[Brian Murray (brian-murray)]

The fix for bug 1614576 will also fix this bug, so I'll backport it to the 16.04 release upgrader shortly.

[Brian Murray (brian-murray)]

That being said it would help to know what package is triggering this error, so that we can write a test case for the Stable Release Update process. You might find more information in the log file /var/log/dist-upgrade/apt-term.log or /var/log/dist-upgrade/main.log.

[Brian Murray (brian-murray)]

After reading carefully, I was able to recreate this by installing shim-signed.

etting up shim-signed (1.18~16.04.1+0.8-0ubuntu2) ...
find: Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 15, in <module>
    import xml.dom, xml.dom.minidom
ImportError: No module named 'xml.dom'

Original exception was:
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-op_78f0n/xenial", line 8, in <module>
    sys.exit(main())
  File "/tmp/ubuntu-release-upgrader-op_78f0n/DistUpgrade/DistUpgradeMain.py", line 242, in main
    if app.run():
  File "/tmp/ubuntu-release-upgrader-op_78f0n/DistUpgrade/DistUpgradeController.py", line 1876, in run
    return self.fullUpgrade()
  File "/tmp/ubuntu-release-upgrader-op_78f0n/DistUpgrade/DistUpgradeController.py", line 1841, in fullUpgrade
    if not self.doDistUpgrade():
  File "/tmp/ubuntu-release-upgrader-op_78f0n/DistUpgrade/DistUpgradeController.py", line 1182, in doDistUpgrade
    res = self.cache.commit(fprogress,iprogress)
  File "/tmp/ubuntu-release-upgrader-op_78f0n/DistUpgrade/DistUpgradeCache.py", line 267, in commit
    apt.Cache.commit(self, fprogress, iprogress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 501, in commit
    # which is less than optimal!
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 465, in install_archives

  File "/tmp/ubuntu-release-upgrader-op_78f0n/DistUpgrade/DistUpgradeView.py", line 234, in run
    res = os.WEXITSTATUS(self.wait_child())
  File "/tmp/ubuntu-release-upgrader-op_78f0n/DistUpgrade/DistUpgradeView.py", line 200, in wait_child
    self.update_interface()
  File "/tmp/ubuntu-release-upgrader-op_78f0n/DistUpgrade/DistUpgradeViewNonInteractive.py", line 229, in update_interface
    sys.stdout.write("%s" % s.decode(locale.getpreferredencoding()))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 0: unexpected end of data

[Brian Murray (brian-murray)]

I'm going to mark this as a duplicate of bug 1614576 given that the other bug was used for the changelog entry fixing it in Yakkety+.

[Max Brustkern (nuclearbob)]

From the other bug, in case this information is needed:

Hello Max, or anyone else affected,

Accepted ubuntu-release-upgrader into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:16.04.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

To test the new release upgrader you'll need to use 'do-release-upgrade' with the '-p' switch.