apt setting overrides on custom image deploy not working

[Conrad Hoffmann]

We are using MAAS to deploy custom Debian images. This worked fine with 2.4, but stopped working when we upgraded to 2.6.

We use the following /etc/maas/preseeds/curtin_userdata_custom to achieve this:

-------------------------------------------
#cloud-config
kernel:
  fallback-package: linux-image-amd64
  package: linux-image-5.0.8-17.18+soundcloud1

apt:
  preserve_sources_list: true

debconf_selections:
 maas: |
  {{for line in str(curtin_preseed).splitlines()}}
  {{line}}
  {{endfor}}

late_commands:
  maas: [wget, '--no-proxy', '{{node_disable_pxe_url}}', '--post-data', '{{node_disable_pxe_data}}', '-O', '/dev/null']
-------------------------------------------

Deploying a node with 2.4, the install log shows that the apt repos pre-configured on the image are used. The deploy finishes successfully.

Using the same curtin_userdata_custom on 2.6, we observe the following in the install log (some parts omitted for brevity and/or sensitivity):

-------------------------------------------
Running curtin builtin curthooks
Configuring target system for distro: debian osfamily: debian
start: cmd-install/stage-curthooks/builtin/cmd-curthooks/writing-apt-config: configuring apt configuring apt
Transferred {'grub2': 'grub2 grub2/update_nvram boolean false', 'maas': "..."} into new format: {'preserve_sources_list': False, ...
curthooks handling apt to target /tmp/tmp1affjmir/target with config {'preserve_sources_list': False,
-------------------------------------------

Then, later on, there is a lot of failed attempts to get "stretch" repos from archive.ubuntu.com:

> Ign:1 http://archive.ubuntu.com/ubuntu stretch InRelease

Adn eventually the installation will fail installing the kernel, since there are no valid apt repos:

> finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-kernel: FAIL: installing kernel

Output of 'dpkg -l '*maas*'|cat':

-------------------------------------------
root@ip-10-33-23-28:/# dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-======================================-============-=================================================
ii maas 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all MAAS server common files
ii maas-dhcp 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all MAAS DHCP server
un maas-dns <none> <none> (no description available)
ii maas-proxy 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all Rack Controller for MAAS
ii maas-region-api 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.6.0-7802-g59416a869-0ubuntu1~18.04.1 all MAAS server provisioning libraries (Python 3)
-------------------------------------------

Steps to reproduce:
 - Use MAAS 2.6.0 (7802-g59416a869-0ubuntu1~18.04.1)
 - Deploy a custom OS image that requires apt/preserve_sources_list to be true during installation

[Conrad Hoffmann (bitfehler)]

Please note that I created this bug in response to https://code.launchpad.net/~bitfehler/maas/+git/maas/+merge/370182

The patch there did fix the issue for me, but having read the 2.4 source code as well now that code path has not really changed the way I thought it did, so I am uncertain about the root cause.

Also, this is my first bug report in launchpad, and if you have any pointers on how to properly format output or code snippets, I would appreciate it. Should they all be attachments? I was confused because the initial form only allows for a single attachment (or I didn't see the button to add more).

Thanks, Conrad

[Conrad Hoffmann (bitfehler)]

I did a bit more investigation, and here is what I have so far, maybe someone more familiar with the matter can make an educated guess based on this:

I initially figured that the order of the curtin configs matters, and to some extend that is certainly true (see the merge proposal linked above).

However, the order has not changed from 2.4 to 2.6, so I wanted to know what exactly changed that made this work in 2.4. I wrote out all the curtin config files in a similar way curtin does when it `pack()`s the image here: https://git.launchpad.net/curtin/tree/curtin/pack.py#n231

The `main_config`, which includes the snippets I added in `/etc/maas/preseeds/curtin_userdata_custom` (including setting preserve_sources_list to true) is always one of the first files. The next file is generated from the `archive_config`, which _always_ contains `preserve_sources_list: false`, so given nothing else, assuming a fairly standard config merge algorithm, it is no surprise that the latter would overwrite the former.

That begs the question: why does it work in 2.4 then? The only difference in the curtin configs that I could find that are possibly related to this is the debconf section that is also part of the `main_config`.

This is the debconf section in 2.4:

debconf_selections: {grub2: grub2 grub2/update_nvram boolean false, maas: 'cloud-init cloud-init/datasources multiselect
    MAAS

    cloud-init cloud-init/maas-metadata-url string http://10.33.152.28:5240/MAAS/metadata/

    cloud-init cloud-init/maas-metadata-credentials string oauth_consumer_key=...&oauth_token_key=...&oauth_token_secret=...

    cloud-init cloud-init/local-cloud-config string apt:\n preserve_sources_list:
    false\n primary:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\n proxy:
    http://10.33.152.28:8000/\n security:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\napt_preserve_sources_list:
    true\napt_proxy: http://10.33.152.28:8000/\nmanage_etc_hosts: false\nmanual_cache_clean:
    true\nreporting:\n maas: {consumer_key: ..., endpoint: ''http://10.33.152.28:5240/MAAS/metadata/status/mg84hc'',\n token_key:
    ..., token_secret: ...,\n type:
    webhook}\nsystem_info:\n package_mirrors:\n - arches: [i386, amd64]\n failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n search:\n primary:
    [''http://archive.ubuntu.com/ubuntu'']\n security: [''http://archive.ubuntu.com/ubuntu'']\n -
    arches: [default]\n failsafe: {primary: ''http://ports.ubuntu.com/ubuntu-ports'',
    security: ''http://ports.ubuntu.com/ubuntu-ports''}\n search:\n primary:
    [''http://ports.ubuntu.com/ubuntu-ports'']\n security: [''http://ports.ubuntu.com/ubuntu-ports'']\n

    '

Note that this contains the setting to NOT preserve the sources first, and then again to DO preserve the sources, potentially overriding the first one.

The debconf section in 2.6 does not contain anything related to apt at all.

So just from looking at it I would guess that this it _might_ be related, though it does not quite make sense to me. From what I understand, thi...

[Adam Collard (adam-collard)]

Fixed in 2.6 in d4c3a712eea5dfdb6eb4a2aecbab46f335f01ecf