curtin fails to configure archive in trusty MAAS deployment

[Andres Rodriguez]

When deploying trusty, curtin fails to configure the archive mirror to the one it is being told to. When deploying xenial with the same configuration, this is not the case.

MAAS sends the config to curtin like:

apt:
  preserve_sources_list: false
  primary:
  - arches: [default]
    uri: http://us.archive.ubuntu.com/ubuntu
  proxy: http://10.90.90.1:8000/
  security:
  - arches: [default]
    uri: http://us.archive.ubuntu.com/ubuntu

The result of this is that curtin starts the installation, but fallbacks to 'archive.ubuntu.com' instead of accessing the defined mirror. Excerpt cloud-init logs:

[...]
Setting up python-crypto (2.6.1-4ubuntu0.2) ...
Setting up python-oauthlib (0.6.1-1) ...
Setting up xfsprogs (3.1.9ubuntu2) ...
Processing triggers for libc-bin (2.19-0ubuntu6.13) ...
Processing triggers for initramfs-tools (0.103ubuntu4.10) ...
update-initramfs: Generating /boot/initrd.img-3.13.0-139-generic
cryptsetup: WARNING: failed to detect canonical device of /media/root-ro/
cryptsetup: WARNING: could not determine root device from /etc/fstab
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
Processing triggers for ureadahead (0.100.0-16) ...
curtin: Installation started. (0.1.0~bzr532-0ubuntu1~16.04.1)
third party drivers not installed or necessary.
Error: /dev/sda: unrecognised disk label
Creating new GPT entries.
The operation has completed successfully.
Get:1 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Ign http://archive.ubuntu.com trusty InRelease
Get:2 http://ppa.launchpad.net trusty InRelease [16.0 kB]
Get:3 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:4 http://ppa.launchpad.net trusty/main amd64 Packages [1139 B]
Get:5 http://security.ubuntu.com trusty-security/main amd64 Packages [704 kB]
Get:6 http://archive.ubuntu.com trusty-backports InRelease [65.9 kB]
Get:7 http://archive.ubuntu.com trusty Release.gpg [933 B]
Get:8 http://archive.ubuntu.com trusty-updates/main amd64 Packages [1050 kB]
Get:9 http://security.ubuntu.com trusty-security/restricted amd64 Packages [14.1 kB]
Get:10 http://security.ubuntu.com trusty-security/universe amd64 Packages [201 kB]
Get:11 http://security.ubuntu.com trusty-security/multiverse amd64 Packages [4803 B]
Get:12 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [17.2 kB]
Get:13 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [439 kB]
Get:14 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.6 kB]
Get:15 http://archive.ubuntu.com trusty-backports/main amd64 Packages [13.3 kB]
Get:16 http://archive.ubuntu.com trusty-backports/restricted amd64 Packages [28 B]
Get:17 http://archive.ubuntu.com trusty-backports/universe amd64 Packages [43.1 kB]
Get:18 http://archive.ubuntu.com trusty-backports/multiverse amd64 Packages [1567 B]
Get:19 http://archive.ubuntu.com trusty Release [58.5 kB]
Get:20 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]
Get:21 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]
Get:22 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]
Get:23 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]
Fetched 10.1 MB in 5s (1843 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  linux-signed-image-3.13.0-139-generic linux-signed-image-generic sbsigntool
The following NEW packages will be installed:
  linux-signed-generic linux-signed-image-3.13.0-139-generic
  linux-signed-image-generic sbsigntool
0 upgraded, 4 newly installed, 0 to remove and 12 not upgraded.
Need to get 62.2 kB of archives.
After this operation, 468 kB of additional disk space will be used.
[...].

Now, If I change the curtin configuration to the old format, I experience the same issue:

apt_mirrors: {ubuntu_archive: 'http://us.archive.ubuntu.com/ubuntu', ubuntu_security: 'http://us.archive.ubuntu.com/ubuntu'}

The result is the same as above.

Again, this does *NOT* happen when deploying xenial, only trusty (and hence precise).

[Ryan Harper (raharper)]

To document my understanding of what happened:

Specifying a mirror URI and no template directs curtin to load the on-disk /etc/apt/sources.list file to be used as the template. The process of turning the apt/sources.list contents into a template uses curtin's default mirror values:

# Default archive mirrors
PRIMARY_ARCH_MIRRORS = {"PRIMARY": "http://archive.ubuntu.com/ubuntu/",
                        "SECURITY": "http://security.ubuntu.com/ubuntu/"}

which include a trailing slash for the mirror value.

Curtin then attempts to match the primary mirror url in the apt/sources.list file and if found, will replace the url with the literal string: $MIRROR which will then later get replaced with the uri value specifed in the yaml.

Since curtin's default primary mirror url included the trailing slash, but the trusty apt/sources.list did not, curtin failed to create a template file which would inject the supplied value.

The fix in the proposed branch relaxes the trailing slash restriction to handle the specific case where the on-disk sources.list is missing a trailing slash.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 17.1-11-ga4c9636b-0ubuntu1

---------------
curtin (17.1-11-ga4c9636b-0ubuntu1) bionic; urgency=medium

  * New upstream snapshot.
    - tests: cleanup the apt custom sources test.
    - apt: Be more lenient when replacing mirrors in /etc/apt/sources.list
      (LP: #1744038)
    - vmtest: Drop Zesty release
    - vmtest: initialize logger with class names for easy parsing
    - packaging: Do not mention primary contributors in debian/changelog.

 -- Scott Moser <email address hidden> Thu, 18 Jan 2018 16:31:27 -0500

[Scott Moser (smoser)]

A fix for this bug is being SRU to Ubuntu 16.04 and 17.10 under bug 1743618.

[Scott Moser (smoser)]

This bug is believed to be fixed in curtin in 18.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.