Comparing the kernel logs, on Xenial, the second nic comes up:
May 22 15:00:27 aurorus kernel: [ 24.840500] IPv6:
ADDRCONF(NETDEV_UP): enP2p1s0f2: link is not ready
May 22 15:00:27 aurorus kernel: [ 25.472391] thunder-nicvf
0002:01:00.2 enP2p1s0f2: Link is Up 10000 Mbps Full duplex
But on bionic, we only ever have f3 up. Note this isn't a network
configuration, but rather the state of the Nic and the switch.
It doesn't appear to matter, 0f3 is what get's bridged by juju anyhow.
But it does suggest that something is different.
There is a slight kernel version variance as well:
Xenial:
May 22 15:00:27 aurorus kernel: [ 0.000000] Linux version
4.15.0-22-generic (buildd@bos02-arm64-038) (gcc version 5.4.0 20160609
(Ubuntu/Lin
Bionic:
May 17 18:03:47 aurorus kernel: [ 0.000000] Linux version
4.15.0-20-generic (buildd@bos02-arm64-029) (gcc version 7.3.0
(Ubuntu/Linaro 7.3.
Looks like Xenial does not use unified cgroup namespaces, not sure
what affect this may have on what's running in those lxd juju
containers.
If the sysfs path exists, then we go "further down" the hole and get
an error like in bionic, but if the sysfs path does not exist, then we
get
the exception we see in Xenial.
Can we get a tar of /sys for both to see if this confirms the
suspicion that we're taking different paths due to differing kernels?
On Tue, May 22, 2018 at 3:27 PM, Jason Hobbs <email address hidden> wrote:
> marked new on nova-compute-charm due to rharper's comment #18, and new
> on libvirt because I've posted all the requested logs now.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1771662
>
> Title:
> libvirtError: Node device not found: no node device with matching name
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-nova-compute/+bug/1771662/+subscriptions
Comparing the kernel logs, on Xenial, the second nic comes up:
May 22 15:00:27 aurorus kernel: [ 24.840500] IPv6: NETDEV_ UP): enP2p1s0f2: link is not ready
ADDRCONF(
May 22 15:00:27 aurorus kernel: [ 25.472391] thunder-nicvf
0002:01:00.2 enP2p1s0f2: Link is Up 10000 Mbps Full duplex
But on bionic, we only ever have f3 up. Note this isn't a network
configuration, but rather the state of the Nic and the switch.
It doesn't appear to matter, 0f3 is what get's bridged by juju anyhow.
But it does suggest that something is different.
There is a slight kernel version variance as well:
Xenial: bos02-arm64- 038) (gcc version 5.4.0 20160609
May 22 15:00:27 aurorus kernel: [ 0.000000] Linux version
4.15.0-22-generic (buildd@
(Ubuntu/Lin
Bionic: bos02-arm64- 029) (gcc version 7.3.0
May 17 18:03:47 aurorus kernel: [ 0.000000] Linux version
4.15.0-20-generic (buildd@
(Ubuntu/Linaro 7.3.
Looks like Xenial does not use unified cgroup namespaces, not sure
what affect this may have on what's running in those lxd juju
containers.
% grep DENIED *.log 3.043:70) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-1_ </var/lib/ lxd>" sys/fs/ cgroup/ unified/ " pid=24143 comm="systemd" 3.043:71) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-1_ </var/lib/ lxd>" sys/fs/ cgroup/ unified/ " pid=24143 comm="systemd" 1.267:88) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-1_ </var/lib/ lxd>" run/systemd/ unit-root/ var/lib/ lxcfs/" pid=24757 6.719:90) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-2_ </var/lib/ lxd>" sys/fs/ cgroup/ unified/ " pid=25227 comm="systemd" 6.723:91) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-2_ </var/lib/ lxd>" sys/fs/ cgroup/ unified/ " pid=25227 comm="systemd" 5.735:108) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-2_ </var/lib/ lxd>" run/systemd/ unit-root/ " pid=26209 comm="(networkd)" flags="ro, 2.211:110) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-0_ </var/lib/ lxd>" sys/fs/ cgroup/ unified/ " pid=28344 comm="systemd" 2.211:111) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-0_ </var/lib/ lxd>" sys/fs/ cgroup/ unified/ " pid=28344 comm="systemd" 0.707:128) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-0_ </var/lib/ lxd>" run/systemd/ unit-root/ " pid=29205 comm="(networkd)" flags="ro, 3.277:151) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-0_ </var/lib/ lxd>" name="/bin/" 3.321:152) : apparmor="DENIED" "lxd-juju- 657fe9- 1-lxd-0_ </var/lib/ lxd>" name="/bin/"
bionic.log:May 17 18:19:33 aurorus kernel: [ 983.592228] audit:
type=1400 audit(152658117
operation="mount" info="failed flags match" error=-13
profile=
name="/
fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
bionic.log:May 17 18:19:33 aurorus kernel: [ 983.592476] audit:
type=1400 audit(152658117
operation="mount" info="failed flags match" error=-13
profile=
name="/
fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
bionic.log:May 17 18:19:41 aurorus kernel: [ 991.818402] audit:
type=1400 audit(152658118
operation="mount" info="failed flags match" error=-13
profile=
name="/
comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
bionic.log:May 17 18:19:46 aurorus kernel: [ 997.271203] audit:
type=1400 audit(152658118
operation="mount" info="failed flags match" error=-13
profile=
name="/
fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
bionic.log:May 17 18:19:46 aurorus kernel: [ 997.271425] audit:
type=1400 audit(152658118
operation="mount" info="failed flags match" error=-13
profile=
name="/
fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
bionic.log:May 17 18:19:55 aurorus kernel: [ 1006.285863] audit:
type=1400 audit(152658119
operation="mount" info="failed flags match" error=-13
profile=
name="/
remount, bind"
bionic.log:May 17 18:20:12 aurorus kernel: [ 1022.760512] audit:
type=1400 audit(152658121
operation="mount" info="failed flags match" error=-13
profile=
name="/
fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
bionic.log:May 17 18:20:12 aurorus kernel: [ 1022.760713] audit:
type=1400 audit(152658121
operation="mount" info="failed flags match" error=-13
profile=
name="/
fstype="cgroup2" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
bionic.log:May 17 18:20:20 aurorus kernel: [ 1031.256448] audit:
type=1400 audit(152658122
operation="mount" info="failed flags match" error=-13
profile=
name="/
remount, bind"
bionic.log:May 17 18:30:03 aurorus kernel: [ 1613.787782] audit:
type=1400 audit(152658180
operation="mount" info="failed flags match" error=-13
profile=
pid=91926 comm="(arter.sh)" flags="ro, remount, bind"
bionic.log:May 17 18:30:03 aurorus kernel: [ 1613.832621] audit:
type=1400 audit(152658180
operation="mount" info="failed flags match" error=-13
profile=
pid=91949 comm="(y-helper)" flags="ro, remount, bind"
xenial.log:May 22 15:15:10 aurorus kernel: [ 918.311740] audit: 0.131:109) : apparmor="DENIED" "file_mmap" "root// lxd-juju- 878ab5- 1-lxd-1_ <var-lib- lxd>" "/usr/lib/ lxd/lxd- bridge- proxy" usr/lib/ lxd/lxd- bridge- proxy" pid=40973 comm="lxd- bridge- prox" 1.427:115) : apparmor="DENIED" "file_mmap" "root// lxd-juju- 878ab5- 1-lxd-2_ <var-lib- lxd>" "/usr/lib/ lxd/lxd- bridge- proxy" usr/lib/ lxd/lxd- bridge- proxy" pid=41233 comm="lxd- bridge- prox"
type=1400 audit(152700211
operation=
namespace=
profile=
name="/
requested_mask="m" denied_mask="m" fsuid=100000 ouid=100000
xenial.log:May 22 15:15:11 aurorus kernel: [ 919.605481] audit:
type=1400 audit(152700211
operation=
namespace=
profile=
name="/
requested_mask="m" denied_mask="m" fsuid=100000 ouid=100000
Looking at the nova.pci.utils code, the different errors seem to be
related to sysfs entries:
https:/ /git.openstack. org/cgit/ openstack/ nova/tree/ nova/pci/ utils.py? id=e919720e08fa e5c07cecda00ac2 d51b0a09f533e# n196
If the sysfs path exists, then we go "further down" the hole and get
an error like in bionic, but if the sysfs path does not exist, then we
get
the exception we see in Xenial.
Can we get a tar of /sys for both to see if this confirms the
suspicion that we're taking different paths due to differing kernels?
On Tue, May 22, 2018 at 3:27 PM, Jason Hobbs <email address hidden> wrote: /bugs.launchpad .net/bugs/ 1771662 /bugs.launchpad .net/charm- nova-compute/ +bug/1771662/ +subscriptions
> marked new on nova-compute-charm due to rharper's comment #18, and new
> on libvirt because I've posted all the requested logs now.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> libvirtError: Node device not found: no node device with matching name
>
> To manage notifications about this bug go to:
> https:/