Ubuntu Pro 18.04 on Azure does not configure azure.archive in sources.list

[Aaron Whitehouse]

On "Ubuntu Server 18.04 LTS - Gen 1" or "Ubuntu Server 18.04 LTS - Gen 2" on Azure, the sources.list is correctly modified to use the Azure archive mirrors:

## Note, this file is written by cloud-init on first boot of an instance
[...]
deb http://azure.archive.ubuntu.com/ubuntu/ bionic main restricted
[...]
deb http://azure.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
[...]

On "Ubuntu Pro 18.04 LTS - Gen 2" (Publisher canonical Offer 0001-com-ubuntu-pro-bionic Plan
pro-18_04-lts-gen2 VM generation V2), there is no Note at the top sources.list saying that it has been written by cloud-init and it has not been correctly updated:

deb http://archive.ubuntu.com/ubuntu/ bionic main restricted
[...]
deb http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted
[...]

East US (Zone 1) in both cases.

[Lucas Albuquerque Medeiros de Moura (lamoura)]

Hello Aaron and thanks for this bug report. Could you please run on the affected PRO machine:

  cloud-init collect-logs

and attach the generated tarball to this bug? I think this is a valid bug report, but additional logs/context will help us working on it. Thanks!

[Aaron Whitehouse (aaron-whitehouse)]

It looks like Focal (gen 1) is working correctly:

[Aaron Whitehouse (aaron-whitehouse)]

Ubuntu Pro 18.04 gen 1 is also not correct. Output of cloud-init collect-logs attached.

[Chad Smith (chad.smith)]

@Aaron, I think this may be a race condition where people are able to ssh into the instance before cloud-init completes. Launching Azure Ubuntu PRO bionic gen1 instances, I was able to easily reproduce this same "invalid" stock apt source by sshing into the vm before cloud-init completed.

On first ssh into the machine I can see the example stock /etc/apt/sources.list without azure-specific mirrors.
I can also block until cloud-init finishes it's setup work with
`cloud-init status --wait --long` which will print "." until cloud-init is done as well as output any errors.

Once cloud-init finished the content in apt sources.list reflected azure mirrors

On PRO images something to note is that ubuntu-advantage-tools is involved in initial system boot too, and the setup/attach of those services takes a while and adds some cost to initial boot time.

`ua auto-attach` blocks later stages of cloud-init (like modules:config) from starting.

On Azure Bionic, I can see that this auto-attach setup cost an additional 2 mins

ubuntu@testpro-bgen1:~$ systemd-analyze blame
    1min 51.944s ua-auto-attach.service
         17.921s cloud-init.service

So checking before that completes would mean that you won't yet see cloud-init apt sources written.

Looking into /var/log/ubuntu-advantage.log on this system we can see the majority of it's setup time is spent waiting on snaps being seeded

2021-03-31 20:27:14,328 - util.py:(509) [DEBUG]: Ran cmd: /usr/bin/snap wait system seed.loaded, rc: 0 stderr: b''
2021-03-31 20:28:37,799 - util.py:(509) [DEBUG]: Ran cmd: /usr/bin/snap install canonical-livepatch, rc: 0 stderr: b''

Note the 1 min 23 second blocking call to /usr/bin/snap wait system seed.loaded in logs.

I'm guessting this is probably why on some PRO image flavors we can see delayed processing/writing of apt config writes.

[Chad Smith (chad.smith)]

In further triage with Dan, this boot timing cost is mostly due to the download of the 'core' snap on which `canonical-livepatch` depends.

this issue has been taken against the ubuntu-advantage-tools project and we are sorting improving that initial boot performance separately from this issue.

https://github.com/canonical/ubuntu-advantage-client/issues/1502

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3862

[Aaron Whitehouse (aaron-whitehouse)]

@Chad, sorry I never saw this until the recent update.

Yes, it looks as though you are right. Starting a 20.04 Pro instance:

$ cat /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://archive.ubuntu.com/ubuntu/ focal main restricted
# deb-src http://archive.ubuntu.com/ubuntu/ focal main restricted

A few minutes later:
$ cat /etc/apt/sources.list
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
## if you wish to make changes you can:
## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
## or do the same in user-data
## b.) add sources in /etc/apt/sources.list.d
## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://azure.archive.ubuntu.com/ubuntu/ focal main restricted
# deb-src http://azure.archive.ubuntu.com/ubuntu/ focal main restricted