Hirsute/Azure: cloud-init-local sometimes very slow to initialize

[Gauthier Jolly]

Sometimes, cloud-init-local takes a few min to start on Azure instances. Eg:

$ systemd-analyze blame
3min 33.770s cloud-init-local.service
      5.441s snapd.seeded.service
      5.146s cloud-init.service

After quickly checking cloud-init logs, I see the following time gap:

2021-04-22 15:00:55,211 - netlink.py[DEBUG]: Wait for media disconnect and reconnect to happen
2021-04-22 15:04:20,783 - netlink.py[DEBUG]: netlink socket ready for read

This only seems to be affecting Hirsute, cloud-init 21.1-19-gbad84ad4-0ubuntu1~20.10.1.

[Paride Legovini (paride)]

Hello Gauthier and thanks for this bug report. What you describe looks very much like this issue:

https://github.com/canonical/ubuntu-advantage-client/issues/1502

Do you think that could be it? I.e, do you hit the problem using Pro images on Azure?

[Dan Watkins (oddbloke)]

An alternative explanation: Azure has a pre-provisioning system, whereby they'll partially boot a machine and then hold it in that state until a user requests a system which corresponds. This is implemented using the netlink socket you see: the fabric will reconnect that socket once it is ready for cloud-init to continue.

So: this logging is not unexpected, it indicates a pre-provisioned VM. Are you seeing an actual 3m30 wait happen when you launch instances, or is this report based only this logging?

[Gauthier Jolly (gjolly)]

Hi,

I am not using Pro offers here.

@oddbloke, that's very interesting, thanks for the explanation. You might be right here, I'm not sure I actually have to wait 3m30 to create the instance. I will try to reproduce the issue while carefully monitoring the delay to be sure.

[Gauthier Jolly (gjolly)]

I was unable to reproduce this, and I think @oddbloke is right. Please feel free to close this and I will post something new if I see the issue happening again.

[Gauthier Jolly (gjolly)]

I was actually able to reproduce today and I can confirm that the
instance creation didn't take more than a few seconds. This bug can be
closed

[James Falcon (falcojr)]

Thanks Gauthier. Based on your comment I closed this.