Comparing the kernel logs, on Xenial, the second nic comes up: May 22 15:00:27 aurorus kernel: [ 24.840500] IPv6: ADDRCONF(NETDEV_UP): enP2p1s0f2: link is not ready May 22 15:00:27 aurorus kernel: [ 25.472391] thunder-nicvf 0002:01:00.2 enP2p1s0f2: Link is Up 10000 Mbps Full duplex But on bionic, we only ever have f3 up. Note this isn't a network configuration, but rather the state of the Nic and the switch. It doesn't appear to matter, 0f3 is what get's bridged by juju anyhow. But it does suggest that something is different. There is a slight kernel version variance as well: Xenial: May 22 15:00:27 aurorus kernel: [ 0.000000] Linux version 4.15.0-22-generic (buildd@bos02-arm64-038) (gcc version 5.4.0 20160609 (Ubuntu/Lin Bionic: May 17 18:03:47 aurorus kernel: [ 0.000000] Linux version 4.15.0-20-generic (buildd@bos02-arm64-029) (gcc version 7.3.0 (Ubuntu/Linaro 7.3. Looks like Xenial does not use unified cgroup namespaces, not sure what affect this may have on what's running in those lxd juju containers. % grep DENIED *.log bionic.log:May 17 18:19:33 aurorus kernel: [ 983.592228] audit: type=1400 audit(1526581173.043:70): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-1_" name="/sys/fs/cgroup/unified/" pid=24143 comm="systemd" fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec" bionic.log:May 17 18:19:33 aurorus kernel: [ 983.592476] audit: type=1400 audit(1526581173.043:71): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-1_" name="/sys/fs/cgroup/unified/" pid=24143 comm="systemd" fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec" bionic.log:May 17 18:19:41 aurorus kernel: [ 991.818402] audit: type=1400 audit(1526581181.267:88): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-1_" name="/run/systemd/unit-root/var/lib/lxcfs/" pid=24757 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind" bionic.log:May 17 18:19:46 aurorus kernel: [ 997.271203] audit: type=1400 audit(1526581186.719:90): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-2_" name="/sys/fs/cgroup/unified/" pid=25227 comm="systemd" fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec" bionic.log:May 17 18:19:46 aurorus kernel: [ 997.271425] audit: type=1400 audit(1526581186.723:91): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-2_" name="/sys/fs/cgroup/unified/" pid=25227 comm="systemd" fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec" bionic.log:May 17 18:19:55 aurorus kernel: [ 1006.285863] audit: type=1400 audit(1526581195.735:108): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-2_" name="/run/systemd/unit-root/" pid=26209 comm="(networkd)" flags="ro, remount, bind" bionic.log:May 17 18:20:12 aurorus kernel: [ 1022.760512] audit: type=1400 audit(1526581212.211:110): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-0_" name="/sys/fs/cgroup/unified/" pid=28344 comm="systemd" fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec" bionic.log:May 17 18:20:12 aurorus kernel: [ 1022.760713] audit: type=1400 audit(1526581212.211:111): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-0_" name="/sys/fs/cgroup/unified/" pid=28344 comm="systemd" fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec" bionic.log:May 17 18:20:20 aurorus kernel: [ 1031.256448] audit: type=1400 audit(1526581220.707:128): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-0_" name="/run/systemd/unit-root/" pid=29205 comm="(networkd)" flags="ro, remount, bind" bionic.log:May 17 18:30:03 aurorus kernel: [ 1613.787782] audit: type=1400 audit(1526581803.277:151): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-0_" name="/bin/" pid=91926 comm="(arter.sh)" flags="ro, remount, bind" bionic.log:May 17 18:30:03 aurorus kernel: [ 1613.832621] audit: type=1400 audit(1526581803.321:152): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-juju-657fe9-1-lxd-0_" name="/bin/" pid=91949 comm="(y-helper)" flags="ro, remount, bind" xenial.log:May 22 15:15:10 aurorus kernel: [ 918.311740] audit: type=1400 audit(1527002110.131:109): apparmor="DENIED" operation="file_mmap" namespace="root//lxd-juju-878ab5-1-lxd-1_" profile="/usr/lib/lxd/lxd-bridge-proxy" name="/usr/lib/lxd/lxd-bridge-proxy" pid=40973 comm="lxd-bridge-prox" requested_mask="m" denied_mask="m" fsuid=100000 ouid=100000 xenial.log:May 22 15:15:11 aurorus kernel: [ 919.605481] audit: type=1400 audit(1527002111.427:115): apparmor="DENIED" operation="file_mmap" namespace="root//lxd-juju-878ab5-1-lxd-2_" profile="/usr/lib/lxd/lxd-bridge-proxy" name="/usr/lib/lxd/lxd-bridge-proxy" pid=41233 comm="lxd-bridge-prox" requested_mask="m" denied_mask="m" fsuid=100000 ouid=100000 Looking at the nova.pci.utils code, the different errors seem to be related to sysfs entries: https://git.openstack.org/cgit/openstack/nova/tree/nova/pci/utils.py?id=e919720e08fae5c07cecda00ac2d51b0a09f533e#n196 If the sysfs path exists, then we go "further down" the hole and get an error like in bionic, but if the sysfs path does not exist, then we get the exception we see in Xenial. Can we get a tar of /sys for both to see if this confirms the suspicion that we're taking different paths due to differing kernels? On Tue, May 22, 2018 at 3:27 PM, Jason Hobbs