container-inherit-properties=apt-primary,apt-security,apt-sources doesn't work on MAAS provider

[Nobuto Murata]

According to:
https://docs.jujucharms.com/2.5/en/models-config#container-inheritance
Juju can inherit apt sources.list from the host to containers basically.

However it doesn't work at least on MAAS provider.

juju: 2.5.0-bionic-amd64
maas: 2.5.0-7442-gdf68e30a5-0ubuntu1~18.04.1

$ maas root package-repositories read
...
    {
        "name": "main_archive",
        "url": "http://my-custom.archive.ubuntu.com/ubuntu/",
        "distributions": [],
        "disabled_pockets": [],
        "disabled_components": [],
        "disable_sources": true,
        "components": [],
        "arches": [
            "amd64"
        ],
        "key": "",
        "enabled": true,
        "id": 1,
        "resource_uri": "/MAAS/api/2.0/package-repositories/1/"
    }

$ juju model-config container-inherit-properties
apt-primary,apt-security,apt-sources

$ juju deploy --series bionic --bind oam-space ubuntu
$ juju add-unit ubuntu --to lxd:0

[/etc/apt/sources.list on host (machine-0)]
deb http://my-custom.archive.ubuntu.com/ubuntu/ bionic multiverse restricted universe main
# deb-src http://my-custom.archive.ubuntu.com/ubuntu/ bionic multiverse restricted universe main
deb http://my-custom.archive.ubuntu.com/ubuntu/ bionic-updates multiverse restricted universe main
# deb-src http://my-custom.archive.ubuntu.com/ubuntu/ bionic-updates multiverse restricted universe main
deb http://my-custom.archive.ubuntu.com/ubuntu/ bionic-security multiverse restricted universe main
# deb-src http://my-custom.archive.ubuntu.com/ubuntu/ bionic-security multiverse restricted universe main
deb http://my-custom.archive.ubuntu.com/ubuntu/ bionic-backports multiverse restricted universe main
# deb-src http://my-custom.archive.ubuntu.com/ubuntu/ bionic-backports multiverse restricted universe main

[/etc/apt/sources.list on container (machine-0-lxd-0)]
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
## if you wish to make changes you can:
## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
## or do the same in user-data
## b.) add sources in /etc/apt/sources.list.d
## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://archive.ubuntu.com/ubuntu bionic main restricted
deb-src http://archive.ubuntu.com/ubuntu bionic main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted
deb-src http://archive.ubuntu.com/ubuntu bionic-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu bionic universe
deb-src http://archive.ubuntu.com/ubuntu bionic universe
deb http://archive.ubuntu.com/ubuntu bionic-updates universe
deb-src http://archive.ubuntu.com/ubuntu bionic-updates universe
...

^^^ expected URL is http://my-custom.archive.ubuntu.com/ubuntu/ instead of http://archive.ubuntu.com/ubuntu

[Nobuto Murata (nobuto)]

Subscribing ~field-high since
1. documented behavior doesn't work
2. changing the default archive URL is required for a customer delivery as the env doesn't have connectivity to archive.ubuntu.com

[Nobuto Murata (nobuto)]

Ok, it seems MAAS 2.5 changed the behavior how to set a custom archive URL with curtin compared to MAAS 2.4.2. Juju needs to be updated to adapt the new behavior of MAAS 2.5.

This issue is a regression from Juju 2.5 + MAAS 2.4.2 from a user point of view.

https://paste.ubuntu.com/p/FgdBTfXswX/

$ grep -r -l my-custom.archive.ubuntu.com maas_2-4-2/ maas_2-5-0/ | grep -v log
maas_2-4-2/0/lxd/0/var/lib/cloud/instance/user-data.txt.i
maas_2-4-2/0/lxd/0/var/lib/cloud/instance/user-data.txt
maas_2-4-2/0/lxd/0/var/lib/cloud/instance/obj.pkl
maas_2-4-2/0/lxd/0/var/lib/cloud/instance/cloud-config.txt
maas_2-4-2/0/baremetal/etc/cloud/cloud.cfg.d/90_dpkg_local_cloud_config.cfg
maas_2-4-2/0/baremetal/var/lib/cloud/instance/obj.pkl
maas_2-4-2/0/baremetal/root/curtin-install-cfg.yaml
maas_2-5-0/0/baremetal/root/curtin-install-cfg.yaml

[Nobuto Murata (nobuto)]

attaching a juju-crashdump from Juju 2.5 + MAAS 2.4.2 deployment so that you can compare it with the original crashdump to see what exactly has been changed.

[Richard Harding (rharding)]

Thanks, we'll try to get a fix into 2.5.2 with the updated behavior.

[Joseph Phillips (manadart)]

Thanks for the detailed info.

I have confirmed with the MAAS team that there behaviour has changed in 2.5. It comes with this patch:
https://github.com/maas/maas/commit/9035a2410ba2c344bf4de886058d75988bd08e94

I'll set about making the necessary Juju modifications to accommodate it.

[Joseph Phillips (manadart)]

*their*

[Joseph Phillips (manadart)]

A likely work-around in the interim is to set cloud-init user-data with the apt values required when provisioning the metal hosts.

Something like:

 apt:
  primary:
  - arches:
    - default
    uri: http://my-custom.archive.ubuntu.com/ubuntu/
  proxy: http://192.168.153.10:8000/
  security:
  - arches:
    - default
    uri: http://my-custom.archive.ubuntu.com/ubuntu/

I will verify if this works.

[Joseph Phillips (manadart)]

I have proposed https://github.com/juju/juju/pull/9764 to address this issue.

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/9819 merges this into develop