[fan-network] Race-condition between "apt update" and dhcp request causes cloud-init error

[Pedro Guimarães]

Hi,

Using manual provider on top of VMWare.

Juju: 2.9.14
VM: Focal
Containers: Bionic

I've noticed that, in a given Focal host, 2x Bionic containers may have different behavior: one successfully completes cloud-init whereas the other fails. The failed container errors at "apt update" in cloud-init with:

Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
  Something wicked happened resolving 'PROXY_IP:<PORT>' (-9 - Address family for hostname not supported)
(Likely this is the line where it breaks: https://github.com/Debian/apt/blob/766b24b7f7484751950c76bc66d3d6cdeaf949a5/methods/connect.cc#L436)

Failed container, full cloud-init-output.log: https://pastebin.canonical.com/p/hKNw49mvJP/
Successful container, full cloud-init-output.log: https://pastebin.canonical.com/p/P6MZdQPrWd/
Host syslog: https://pastebin.canonical.com/p/jjMmWRFd9y/

I can see that, on the host, dnsmasq served the FAN IP to the broken container after the "apt-update" was called.

----- FAILED CONTAINER -----:
Sep 19 13:02:14 lma-stack-2 dnsmasq-dhcp[1243]: DHCPOFFER(fan-252) 252.252.15.198 00:16:3e:da:dd:c4
apt-update on cloud-init was ran:
Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~18.04.1 running 'modules:config' at Sun, 19 Sep 2021 13:02:13 +0000. Up 3.90 seconds.

----- SUCCESSFUL CONTAINER -----:
Sep 19 13:09:40 lma-stack-2 dnsmasq-dhcp[1243]: DHCPOFFER(fan-252) 252.252.15.253 00:16:3e:d7:6b:ef
apt-update on cloud-init was ran:
Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~18.04.1 running 'modules:config' at Sun, 19 Sep 2021 13:10:10 +0000. Up 35.99 seconds.

In the case of the Failed Container, the DHCPOFFER arrived 1s after the "apt update" was requested, whereas the Successful Container logged a DHCPOFFER 30s before the "apt update".

[Pedro Guimarães (pguimaraes)]

Also added charm-rabbitmq-server, where this error is more clear. Rabbitmq-server fails on: https://github.com/openstack/charm-rabbitmq-server/blob/master/hooks/install#L9 because it runs a "dpkg" command before a successful execution of "apt update" on the container. Install eventually fails with:

# ./hooks/install
dpkg-query: package 'python3-pip' is not installed and no information is available
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package python3-pip

[Liam Young (gnuoy)]

Perhaps I'm missing something but this does not seem to be a bug in the rabbitmq-server charm. It may be easier to observe there but the root cause is elsewhere.

[Harry Pidcock (hpidcock)]

https://github.com/juju/juju/pull/14028