Curtin failure with 'utf8' codec can't decode byte 0x80 in position 373: invalid start byte

[Dave Chiluk]

Curtin fails to install with the following returned in install.log

______________________________________________________________
'utf8' codec can't decode byte 0x80 in position 373: invalid start byte
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'simple']
Exit code: 3
Reason: -
Stdout: "'utf8' codec can't decode byte 0x80 in position 373: invalid start byte\n"
Stderr: ''
______________________________________________________________

This looks to be related to the existence or detection of fakeraid on the machine. We're still trying to determine if there is a way to disable fakeraid so that the install will succeed. Additionally this happens fairly early in the install process.

Machines are available to debug this within the Canonical Lab.

[Dave Chiluk (chiluk)]

Additionally it should be noted that this was experienced using maas 1.7.1+bzr3341-0ubuntu1~trusty1.

[Dave Chiluk (chiluk)]

This might be a duplicate of 1394335

[Dave Chiluk (chiluk)]

So we discovered a workaround for this.

It turns out that disks had fakeraid superblocks. When d-i or curtin go to install, they are getting screwed up by this.

So, our solution was to boot the servers with an Ubuntu livecd. We used https://help.ubuntu.com/community/Installation/MinimalCD

Proceed into rescue mode, run through the prompts, and execute a shell.

From the shell prompt. You'll want to stop all md devices. *(please forgive the freehand scripting), and then run mdadm --zero-superblock /dev/sd* on the remaining disk drives.
_________________________________________________________________
#!/bin/bash
for i in `ls /dev/md*`; do
mdadm --stop $i
done

for i in `ls /dev/sd*`; do
mdadm --zero-superblock $i
done

That should do it.
_________________________________________________________________

It would be nice if we fix curtin to detect errant superblocks, and possibly clear them. It might also be nice to be able to select some sort of fakeraid functionality from maas for each machine. I haven't fully thought this through, and I'll leave that up to the maas team to sort out.

[Scott Moser (smoser)]

You should be able to work around this with something like:
--- /etc/maas/preseeds/curtin_userdata.dist 2015-03-25 19:39:19.667272991 +0000
+++ /etc/maas/preseeds/curtin_userdata 2015-03-25 19:39:21.591291807 +0000
@@ -12,6 +12,7 @@
   driver_02_add: ["add-apt-repository", "-y", "deb {{driver['repository']}} {{node.get_distro_series()}} main"]
   driver_03_update_install: ["sh", "-c", "apt-get update --quiet && apt-get --assume-yes install {{driver['package']}}"]
   driver_04_load: ["sh", "-c", "depmod && modprobe {{driver['module']}}"]
+ wipe_mdadm: ["sh", "-c", 'apt-get install -qy mdadm; for i in /dev/md*; do [ -b "$i" ] || continue; mdadm --stop "$i"; done; udevadm settle; for i in /dev/[xsv]d*; do [ -b "$i" ] || mdadm --zero-superblock "$i"; done']
 {{endif}}
 late_commands:
   maas: [wget, '--no-proxy', '{{node_disable_pxe_url|escape.shell}}', '--post-data', '{{node_disable_pxe_data|escape.shell}}', '-O', '/dev/null']

[Dave Chiluk (chiluk)]

So Scott, that would allow curtin to proceed, but I realized that it might be nice for maas to be able to install to a fakeraid device as well. Last time I tried that it didn't work, but I haven't attempted recently.

[Ryan Harper (raharper)]

RAID support (not sure about fakeraid and maas) is available via custom storage in MAAS 1.8+

We'd need to figure out what is needed for fakeraid, and how to communicate that vs. plain mdadm.