'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128) (curtin can't handle non-ascii characters in dpkg --list output)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
curtin |
Fix Released
|
High
|
Blake Rouse | ||
curtin (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Trusty |
Fix Released
|
Undecided
|
Unassigned | ||
Utopic |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
I was trying to deploy an image based on an Ubuntu Desktop 14.04 install using MAAS 1.7.0~beta2+
I made a "curtin-compatible" tar.gz which includes cloud-init, put it in place of root-tgz in my maas server, and tried to deploy a node (using juju bootstrap).
Juju ended up complaining and timing out, and the node never fully completed installation. On closer look, I found curtin crashes when running "curthooks", specifically at some point it does dpkg --list, and the code that handles this (utils.subp) assumes output from its commands is pure ascii and so crashes when the output contains non-ascii. For Ubuntu Desktop, in particular, the output contains "Déjȧ Dup", which is what causes things to barf.
I SSHd into the half-installed node and ran curtin manually like so:
sudo CURTIN_STACKTRACE=1 /curtin/bin/curtin --verbose --verbose --log-file /media/
It ended in this:
Running command ['chroot', '/tmp/tmpQ_
Traceback (most recent call last):
File "/curtin/
sys.
File "/curtin/
apply_
File "/curtin/
pkgs_installed = get_installed_
File "/curtin/
(out, _err) = util.subp(cmd, capture=True)
File "/curtin/
out = out.decode()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128)
'ascii' codec can't decode byte 0xc3 in position 19027: ordinal not in range(128)
Traceback (most recent call last):
File "/curtin/
sys.
File "/curtin/
stage.run()
File "/curtin/
util.subp(cmd, shell=shell, env=self.env)
File "/curtin/
cmd=args)
ProcessExecutio
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: ''
Stderr: ''
Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: ''
Stderr: ''
I'm attaching the full log as well.
SRU curtin
==========
[Impact]
curtin can't handle non-ascii characters in dpkg --list output as mentioned in the original bug report descripition. This bug doesn't allow curtin to continue and terminates the installation.
This fix is needed so that we can make subprocess calls with output that contains non-ascii characters.
The patch supplied by the bug reporter enables non-ascii character subprocess output decoding for curtin.
[Test Case]
To reproduce the bug, follow the same steps as the bug report description mentions. Namely, making sure that dpkg --list will contain output with non-ascii characters.
[Regression Potential]
utf-8 is backwards compatible with ascii, so the likely hood of regressions is small.
Changed in curtin: | |
status: | Triaged → Fix Committed |
Changed in curtin (Ubuntu): | |
status: | New → Fix Released |
description: | updated |
description: | updated |
tags: |
added: verification-done removed: verification-needed |
The attached patch makes things work, but it may be too naive. I don't think explicitly expecting utf-8 is riskier than the old behavior (implicitly assuming ascii) but I'm not an expert with this codebase and may be missing something. Review appreciated :)