error upgrading from 2.7 to 2.8: cannot get all link layer devices

[Laurent Sesquès]

On a non-ha juju controller, with just the default model, but managing machines with lots of network devices (openstack neutron) being created or deleted, upgrading to 2.8.3 ended up with a very slow controller, which errors when just trying to get a juju status on the default model:

ERROR could not fetch IP addresses and link layer devices: cannot get all link layer devices

https://pastebin.canonical.com/p/hXkDM32dTn/

The size of the linklayerdevices collection:
`db.linklayerdevices.find().explain("executionStats")`:
     "totalDocsExamined" : 795837

These are all the qvb*, tap*, qvo*, qbr* interfaces.

We started investigating with manadart (to whom I proveided a dump) and achilleasa. This bug is to capture further investigations.

[Pen Gale (pengale)]

Triaged as high priority and dropped into juju 2.8.6 milestone. Conversations are currently ongoing. Please post updates to this bug!

[Joseph Phillips (manadart)]

Adds an index to the collection for access by model and/or machine:
https://github.com/juju/juju/pull/12133

[Joseph Phillips (manadart)]

Prior releases had an issue where discovered devices were added to the collection, but not removed, which explains the accrual of these ephemeral virtual devices.

The 2.8 series addresses this issue in the logic for setting each machine's link-layer devices.

Where neither the provider nor the machine observes an interface, it is now deleted. So this should be a transient problem.

We are experimenting with indexes on the collection. The access-by-machine case is sped up greatly, but we're seeing worse performance where the index is not very selective, such as by model in this case.

This is only used by "juju status" though, so once the machines strangle out the devices, this should resolve.

[Joseph Phillips (manadart)]

https://bugs.launchpad.net/juju/+bug/1899536