Trusty deployments fail when custom archive keys are configured

[Christian Reis]

TLDR: when MAAS is configured to use custom archives, Trusty deployments fail.

When custom archive configurations are in use, MAAS renders cloud-init config which cloud-init reads in order to set up apt details.

The cloud-init config format it renders is only compatible with the cloud-init in Xenial. That version is currently:

  0.7.9-153-g16a7302f-0ubuntu1~16.04.2

It looks something like this:

        apt:
          source1:
              source: "deb http://packages.chef.io/repos/apt/stable
              $RELEASE main"
                  key: |
                        -----BEGIN PGP PUBLIC KEY BLOCK-----

I fetched that example from http://cloudinit.readthedocs.io/en/latest/topics/examples.html

The cloud-init version in Trusty is 0.7.5-0ubuntu1.22 and the equivalent stanza looks like this:

        apt_sources:
         - source: "deb http://apt.opscode.com/ $RELEASE-0.10 main"
            key: |
                 -----BEGIN PGP PUBLIC KEY BLOCK-----

I fetched that example from http://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

Looking at src/maasserver/compose_preseed.py:get_archive_config() it seems that MAAS only supports the Xenial-hosted version's format. That breaks the custom archives feature.

[Christian Reis (kiko)]

There are at least two ways this could be fixed:

1. To backport the cloud-init change to Trusty; note that cloud-init is backwards-compatible and widely tested, but also widely in use
2. To have MAAS generate stanzas in both formats, selecting the right one according to the ephemeral environment being used.

It would be worth considering what will happen as cloud-init continues to evolve and our LTS backlog grows due to ESM.

[Christian Reis (kiko)]

Note that https://help.ubuntu.com/community/CloudInit still documents the old stanza format.

[Christian Reis (kiko)]

(Presumably because it's still accepted by cloud-init)

[Andres Rodriguez (andreserl)]

@Kiko,

I've tested a deployment of trusty and confirm that the deployed system does end up with the configuration of the custom repository. This is because the configuration happens by curtin.

However, the difference between trusty and xenial is that the custom repository is not available in the ephemeral environment, which I'm guessing its what's affecting you.

As such, can you confirm what customizations you are doing that are causing the deployment failure ?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Scott Moser (smoser)]

I'm wondering if you possibly were hitting bug 1744038.

MAAS can declare (and actually, i think *does* declare) both old format configuration of apt and new format configuration. That would cover trusty and later. If you declare both, though, there is code in cloud-init that insists they are consistent with each other. Because otherwise it wouldnt know which value was "correct".

I'm marking incomplete. Can you give example of *what* is faling ?

[Christian Reis (kiko)]

MAAS does? I am looking at src/maasserver/compose_preseed.py:get_archive_config() and I don't see apt_sources anyway.

[Andres Rodriguez (andreserl)]

For posterity, MAAS trusty configuration uses package_mirrors to change the archive the machines should access. Such configuration does not support specifying a key for a custom mirror.

That said, the deployed machine will end up with the correct configuration, but the deployment ephemeral environment doesn't.

The reason this is an issue is becuase that instead of creating an additional reposiutory with the software, a custom mirror is being created, with additional packages, and completely re-signed.

[Ondrej Kuchar (ondrej-kuchar)]

I applied the patch on MaaS 2.2.3 and seems it broke deployment of nodes where `if third_party_drivers` condition is true in /etc/maas/preseeds/curtin_userdata.

I'm attaching part of /var/log/maas/regiond.log where it failed.

[Ondrej Kuchar (ondrej-kuchar)]

our curtin_userdata file

[Ryan Harper (raharper)]

I'm closing the cloud-init task; If there is further information about what cloud-init would need to do here, please reopen.