[2.4-beta3] juju relies on deprecated cloud-init apt_mirror cloud-config key: "Old and New apt format defined with unequal values True vs False"

[Dmitrii Shcherbakov]

cloud-init has supported apt v3 configuration for a while now but Juju still uses the old format.

https://github.com/juju/juju/blob/8b3a6eb703f06fb9f57a37a27afaee9aac1e8d59/cloudconfig/cloudinit/cloudinit_ubuntu.go#L43-L57
 cfg.SetAttr("apt_mirror", url)

While cloud-init converts the old format entries for compatibility reasons there are still some issues that occur.
https://github.com/cloud-init/cloud-init/blob/ubuntu/18.2-59-gcd1de5f4-0ubuntu1/cloudinit/config/cc_apt_configure.py#L667-L682
https://github.com/cloud-init/cloud-init/blob/ubuntu/18.2-59-gcd1de5f4-0ubuntu1/cloudinit/config/cc_apt_configure.py#L888-L909
def find_apt_mirror_info(cfg, cloud, arch=None):
    """find_apt_mirror_info
       find an apt_mirror given the cfg provided.
       It can check for separate config of primary and security mirrors
       If only primary is given security is assumed to be equal to primary
       If the generic apt_mirror is given that is defining for both
    """

The new format looks like this:
http://cloudinit.readthedocs.io/en/latest/topics/modules.html
apt:
  primary:
    - arches: [default]
      search:
        - http://local-mirror.mydomain
        - http://archive.ubuntu.com

Now to the problem that I encountered.

MAAS has "xenial-proposed" added for "main" and "universe":
https://paste.ubuntu.com/p/WT3Z3VgsDg/
maas maas package-repositories read
#...
        "name": "proposed",
        "distributions": [
            "xenial-proposed"
        ],
        "resource_uri": "/MAAS/api/2.0/package-repositories/8/",
        "components": [
            "main",
            "universe"
        ],

At the same time, a mirror is configured at the model level:

juju model-config apt-mirror
http://mirror.yandex.ru/ubuntu/

This results in the following 2 config files on the target machine:

What came from MAAS in v3 format:
https://paste.ubuntu.com/p/ywqZDz9ghb/
/etc/cloud/cloud.cfg.d/90_dpkg_local_cloud_config.cfg
apt:
  preserve_sources_list: false
  primary:
  - arches: [default]
    uri: http://archive.ubuntu.com/ubuntu
  proxy: http://10.10.10.88:3128
  security:
  - arches: [default]
    uri: http://archive.ubuntu.com/ubuntu
  sources:
    proposed_8: {source: 'deb http://archive.ubuntu.com/ubuntu xenial-proposed main
        universe'} # <---- proposed

Added by juju in the old format:

cat /var/lib/cloud/instances/xcgtpc/cloud-config.txt
#cloud-config
# ...
apt_mirror: http://mirror.yandex.ru/ubuntu/
# ...

MAAS configs take priority in the end but cloud-init errors out:

018-05-28 19:15:45,392 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/xcgtpc/sem/config_apt_configure - wb: [644] 25 bytes
2018-05-28 19:15:45,392 - helpers.py[DEBUG]: Running config-apt-configure using lock (<FileLock using file '/var/lib/cloud/instances/xcgtpc/sem/config_apt_configure'>)
2018-05-28 19:15:45,392 - cc_apt_configure.py[DEBUG]: apt config: convert V2 to V3 format for keys 'apt_mirror, apt_preserve_sources_list, apt_proxy'
2018-05-28 19:15:45,393 - cc_apt_configure.py[DEBUG]: apt config: V1/2 and V3 format specified, preferring V3
2018-05-28 19:15:45,393 - handlers.py[DEBUG]: finish: modules-config/config-apt-configure: FAIL: running config-apt-configure with frequency once-per-instance
2018-05-28 19:15:45,393 - url_helper.py[DEBUG]: [0/1] open 'http://10.10.101.2/MAAS/metadata/status/xcgtpc' with {'method': 'POST', 'headers': {'Authorization': 'OAuth oauth_nonce="36428931178945893021527534945", oauth_timestamp="1527534945", oauth_version="1.0", oauth_signature_method="PLAINTEXT", oauth_consumer_key="p2cZN7tatHWVrr699E", oauth_token="nWjQufkQC2eBZFBVut", oauth_signature="%26qdUxpdvS9a4BLgqpGhnf4KWNKUUjWwjG"'}, 'allow_redirects': True, 'url': 'http://10.10.101.2/MAAS/metadata/status/xcgtpc'} configuration
2018-05-28 19:15:45,398 - url_helper.py[DEBUG]: Read from http://10.10.101.2/MAAS/metadata/status/xcgtpc (204, 0b) after 1 attempts
2018-05-28 19:15:45,398 - util.py[WARNING]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
2018-05-28 19:15:45,398 - util.py[DEBUG]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 792, in _run_modules
    freq=freq)
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 274, in handle
    ocfg = convert_to_v3_apt_format(ocfg)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 759, in convert_to_v3_apt_format
    cfg = convert_v2_to_v3_apt_format(cfg)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 725, in convert_v2_to_v3_apt_format
    oldkey))
ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list

I think something should take priority (Juju seems to have higher priority to me) and Juju needs to use the new format as by this time cloud images contain cloud-init that supports apt v3 config.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.

[Trent Lloyd (lathiat)]

I get the following warning when booting now:
cloud-init[1330]: 2023-08-21 04:50:48,176 - util.py[DEPRECATED]: The following config key(s): ['apt_mirror'] is deprecated in 22.1 and scheduled to be removed in 27.1.

While that seems a few years away, since cloud-init is regularly updated in instance images it may become an issue later.

I think this should be triaged higher than Low and I can't actually update the importance. So I am setting this from Triaged back to New for re-triage.