@Aaron, I think this may be a race condition where people are able to ssh into the instance before cloud-init completes. Launching Azure Ubuntu PRO bionic gen1 instances, I was able to easily reproduce this same "invalid" stock apt source by sshing into the vm before cloud-init completed.
On first ssh into the machine I can see the example stock /etc/apt/sources.list without azure-specific mirrors.
I can also block until cloud-init finishes it's setup work with
`cloud-init status --wait --long` which will print "." until cloud-init is done as well as output any errors.
Once cloud-init finished the content in apt sources.list reflected azure mirrors
On PRO images something to note is that ubuntu-advantage-tools is involved in initial system boot too, and the setup/attach of those services takes a while and adds some cost to initial boot time.
`ua auto-attach` blocks later stages of cloud-init (like modules:config) from starting.
On Azure Bionic, I can see that this auto-attach setup cost an additional 2 mins
@Aaron, I think this may be a race condition where people are able to ssh into the instance before cloud-init completes. Launching Azure Ubuntu PRO bionic gen1 instances, I was able to easily reproduce this same "invalid" stock apt source by sshing into the vm before cloud-init completed.
On first ssh into the machine I can see the example stock /etc/apt/ sources. list without azure-specific mirrors.
I can also block until cloud-init finishes it's setup work with
`cloud-init status --wait --long` which will print "." until cloud-init is done as well as output any errors.
Once cloud-init finished the content in apt sources.list reflected azure mirrors
On PRO images something to note is that ubuntu- advantage- tools is involved in initial system boot too, and the setup/attach of those services takes a while and adds some cost to initial boot time.
`ua auto-attach` blocks later stages of cloud-init (like modules:config) from starting.
On Azure Bionic, I can see that this auto-attach setup cost an additional 2 mins
ubuntu@ testpro- bgen1:~ $ systemd-analyze blame attach. service
1min 51.944s ua-auto-
17.921s cloud-init.service
So checking before that completes would mean that you won't yet see cloud-init apt sources written.
Looking into /var/log/ ubuntu- advantage. log on this system we can see the majority of it's setup time is spent waiting on snaps being seeded
2021-03-31 20:27:14,328 - util.py:(509) [DEBUG]: Ran cmd: /usr/bin/snap wait system seed.loaded, rc: 0 stderr: b'' livepatch, rc: 0 stderr: b''
2021-03-31 20:28:37,799 - util.py:(509) [DEBUG]: Ran cmd: /usr/bin/snap install canonical-
Note the 1 min 23 second blocking call to /usr/bin/snap wait system seed.loaded in logs.
I'm guessting this is probably why on some PRO image flavors we can see delayed processing/writing of apt config writes.