Artful to Bionic upgrade kills all LXD container networking

[David Favor]

Upgrading machine from Artful -> Bionic all LXD containers fail, as no DHCP addresses are assigned.

[David Favor (davidfavor)]

This is a high priority bug as it means any machine running many containers, like hosting companies will destroy all connectivity to containers if upgrade is done.

This means Bionic upgrades should not occur (when LXD containers running), till this bug is fixed.

[Scott Moser (smoser)]

Hi David,
Thank you for filing the bug.

Could you please run 'cloud-init collect-logs' and attach the output?

We're interested also in knowing what platform you are running on.
Is cloud-init installed on your container host ? Is that running on a cloud platform?

Or, are you only having problems network problems in the containers?

[Scott Moser (smoser)]

David,
After you've provided the information requested, please set the bug back to 'New' status.
Thanks,
Scott

[Ryan Harper (raharper)]

What's the process for upgrading from Artful to Bionic?

do-release-upgrade -d ?

[David Favor (davidfavor)]

Additional info.

Correct. I did a do-release-upgrade -d to update from Artful to Bionic.

Also, this machine installed Artful from scratch, so was using Netplan networking.

I've run cloud-init collect-logs + this produces no output + even using strace it's not clear where logs end up being deposited.

Let me know + I'll provide the logs.

Only network problems are in containers.

Also, if I simply copy a failing container over to another Artful machine the container works fine, so it appears something's wrong with container <-> machine plumbing.

[David Favor (davidfavor)]

Ah... Maybe you're referring to standard cloud-init logs.

I've attached these two logs. If other logs are required, let me know.

[David Favor (davidfavor)]

If you look closely at the cloud-init-output.log you'll see the DHCP address never gets assigned + there's no errors or warnings about any problems.

[Scott Moser (smoser)]

cloud-init collect-logs

writes output to the current working directory named 'cloud-init.tar.gz'.

Can you please attach that?

thank you.

[Scott Moser (smoser)]

Hi David,
Thank you for the information.

Its not clear to me what the problem is.

Please answer:
   Which OS did you ugprade from artful to bionic? The host? the container? both?

I just successfully attempted a upgrade from an artful lxd container to bionic lxd container (my host is bionic, my lxd is from the snap packaging of lxd version 3.0.0).

% lxc launch ubuntu-daily:artful a1
% lxc exec a1 -- do-release-upgrade -d
# say 'Y' (yes) several times, reboot
% lxc list a1
$ lxc list a1
+------+---------+-----------------------+-----------------------------------------------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+---------+-----------------------+-----------------------------------------------+------------+-----------+
| a1 | RUNNING | 10.145.210.191 (eth0) | fd42:6470:c1cd:249f:216:3eff:fe85:80bb (eth0) | PERSISTENT | 0 |
+------+---------+-----------------------+-----------------------------------------------+------------+-----------+

[David Favor (davidfavor)]

Upgrade was done at host/machine level only.

When this is done all containers fail.

https://github.com/lxc/lxd/issues/4510 covers more information.

There appear to be many problems with bionic + LXD.

[David Favor (davidfavor)]

Problem seems to be in Ubuntu packaging, as this fixes problem.

1) copy all containers to another LXD machine

2) do fresh Bionic install

3) copy all containers back

At this point all problems resolve, so some sort of packaging problem in the Artful to Bionic upgrade.

[David Favor (davidfavor)]

https://github.com/lxc/lxd/issues/4510 is the LXD related bug with additional information + fix LXD dev is rolling in to attempt avoiding problem.

[Ryan Harper (raharper)]

Reading through the lxd issue log, it appears that a host networking issue/packaging issue triggered on upgrade; unrelated to cloud-init inside the containers. If you feel that cloud-init isn't doing something correct, please re-open this bug with new information.

Thanks.