Nodes fail to deploy with custom preseed

[Rod Smith]

Nodes are failing to deploy again -- noticed on 13 March, 2019. The problem seems to be something in our custom preseed, although it can be worked around by disabling the ppa:hardware-certification/public PPA, too.

[Rod Smith (rodsmith)]

An update: This appears to be caused by a recent change to MAAS and/or curtin. Our preseed installs canonical-certification-server and its dependencies, and something about this is causing the latest MAAS/curtin to fail when it tries to set up the network.

As a workaround, disable the ppa:hardware-certification/public PPA in MAAS. The deployment will succeed, but canonical-certification-server will not be installed. It can be installed manually, as described in Appendix A of the Self-Test Guide.

[Andres Rodriguez (andreserl)]

From looking at the logs in bug #1819994, I see this:

Mar 11 06:55:15 one-weevil cloud-init[4010]: Get:1 http://ppa.launchpad.net/hardware-certification/public/ubuntu xenial/main amd64 dmidecode amd64 3.0-2ubuntu1~x1 [52.1 kB]
Mar 11 06:55:26 one-weevil cloud-init[4010]: debconf: unable to initialize frontend: Dialog
Mar 11 06:55:26 one-weevil cloud-init[4010]: debconf: (TERM is not set, so the dialog frontend is not usable.)
Mar 11 06:55:26 one-weevil cloud-init[4010]: debconf: falling back to frontend: Readline
Mar 11 06:55:26 one-weevil cloud-init[4010]: debconf: unable to initialize frontend: Readline
Mar 11 06:55:26 one-weevil cloud-init[4010]: debconf: (This frontend requires a controlling tty.)
Mar 11 06:55:26 one-weevil cloud-init[4010]: debconf: falling back to frontend: Teletype
Mar 11 06:55:26 one-weevil cloud-init[4010]: dpkg-preconfigure: unable to re-open stdin:
Mar 11 06:55:26 one-weevil cloud-init[4010]: Fetched 52.1 kB in 30s (1730 B/s)
Mar 11 06:55:26 one-weevil cloud-init[4010]: E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)
Mar 11 06:55:26 one-weevil cloud-init[4010]: (Reading database ... 90302 files and directories currently installed.)
Mar 11 06:55:26 one-weevil cloud-init[4010]: Preparing to unpack .../dmidecode_3.0-2ubuntu1~x1_amd64.deb ...
Mar 11 06:55:27 one-weevil cloud-init[4010]: Unpacking dmidecode (3.0-2ubuntu1~x1) over (3.0-2ubuntu0.1) ...
Mar 11 06:55:27 one-weevil cloud-init[4010]: Processing triggers for man-db (2.7.5-1) ...
Mar 11 06:55:28 one-weevil cloud-init[4010]: Setting up dmidecode (3.0-2ubuntu1~x1) ...
Mar 11 06:56:29 one-weevil cloud-init[4010]: Done.
Mar 11 06:56:30 one-weevil cloud-init[4010]: Setting up to use KVM image on MAAS server...
Mar 11 06:56:30 one-weevil cloud-init[4010]: --2019-03-11 06:56:30-- http://192.168.120.1/cloud/
Mar 11 06:56:30 one-weevil cloud-init[4010]: Connecting to 192.168.120.1:80... connected.
Mar 11 06:56:30 one-weevil cloud-init[4010]: HTTP request sent, awaiting response... 200 OK
Mar 11 06:56:30 one-weevil cloud-init[4010]: Length: 4828 (4.7K) [text/html]
Mar 11 06:56:30 one-weevil cloud-init[4010]: Saving to: '/tmp/index.html'
Mar 11 06:56:30 one-weevil cloud-init[4010]: 0K .... 100% 491M=0s
Mar 11 06:56:30 one-weevil cloud-init[4010]: 2019-03-11 06:56:30 (491 MB/s) - '/tmp/index.html' saved [4828/4828]
Mar 11 06:56:30 one-weevil cloud-init[4010]: Setting up to use LXD images on MAAS server...
Mar 11 06:56:30 one-weevil cloud-init[4010]: --2019-03-11 06:56:30-- http://192.168.120.1/cloud/
Mar 11 06:56:30 one-weevil cloud-init[4010]: Connecting to 192.168.120.1:80... connected.
Mar 11 06:56:30 one-weevil cloud-init[4010]: HTTP request sent, awaiting response... 200 OK
Mar 11 06:56:30 one-weevil cloud-init[4010]: Length: 4828 (4.7K) [text/html]
Mar 11 06:56:30 one-weevil cloud-init[4010]: Saving to: '/tmp/index.html'
Mar 11 06:56:30 one-weevil cloud-init[4010]: 0K .... 100% 472M=0s
Mar 11 06:56:30 one-weevil cloud-init[4010]: 2019-03-11 06:56:30 (472 MB/s) - '/tmp/index.html' saved [4828/4828]
Mar 11 06:56:30 one-weevil ...

[Rod Smith (rodsmith)]

Andres, we've traced the problem to the network-manager package, which gets pulled in by a dependency in canonical-certification-server. Apparently, curtin or cloud-init (I'm not sure which) is now skipping netplan configuration when the network-manager package is installed.

[Rod Smith (rodsmith)]

For anybody following this, we have a workaround in maas-cert-server 0.3.9. If you're affected by this bug, please update maas-cert-server and you should be able to deploy. The drawback is that the Debian package database will be left in an inconsistent state, so you'll need to do a "sudo apt --fix-broken install" on the node after deploying it before you can install new packages. (This won't normally be necessary to run certification tests.) A true fix is expected shortly.

[Rod Smith (rodsmith)]

The cloud-init team has released a proper fix for the problem, and it's been pushed out, so starting with maas-cert-server 0.4.3, we've reverted the band-aid workaround introduced in 0.3.9. Problems might still occur if you start up a MAAS server that hasn't been booted in a while and immediately begin a deployment, though. If you experience this problem, wait a while, or temporarily disable the PPA, as noted in comment #1. Reverting this band-aid solution means "sudo apt --fix-broken install" should no longer be necessary when trying to install new software.