Comment 7 for bug 1921429

Revision history for this message
Chad Smith (chad.smith) wrote :

@Aaron, I think this may be a race condition where people are able to ssh into the instance before cloud-init completes. Launching Azure Ubuntu PRO bionic gen1 instances, I was able to easily reproduce this same "invalid" stock apt source by sshing into the vm before cloud-init completed.

On first ssh into the machine I can see the example stock /etc/apt/sources.list without azure-specific mirrors.
I can also block until cloud-init finishes it's setup work with
`cloud-init status --wait --long` which will print "." until cloud-init is done as well as output any errors.

Once cloud-init finished the content in apt sources.list reflected azure mirrors

On PRO images something to note is that ubuntu-advantage-tools is involved in initial system boot too, and the setup/attach of those services takes a while and adds some cost to initial boot time.

`ua auto-attach` blocks later stages of cloud-init (like modules:config) from starting.

On Azure Bionic, I can see that this auto-attach setup cost an additional 2 mins

ubuntu@testpro-bgen1:~$ systemd-analyze blame
    1min 51.944s ua-auto-attach.service
         17.921s cloud-init.service

So checking before that completes would mean that you won't yet see cloud-init apt sources written.

Looking into /var/log/ubuntu-advantage.log on this system we can see the majority of it's setup time is spent waiting on snaps being seeded

2021-03-31 20:27:14,328 - util.py:(509) [DEBUG]: Ran cmd: /usr/bin/snap wait system seed.loaded, rc: 0 stderr: b''
2021-03-31 20:28:37,799 - util.py:(509) [DEBUG]: Ran cmd: /usr/bin/snap install canonical-livepatch, rc: 0 stderr: b''

Note the 1 min 23 second blocking call to /usr/bin/snap wait system seed.loaded in logs.

I'm guessting this is probably why on some PRO image flavors we can see delayed processing/writing of apt config writes.