Activity log for bug #1882519

Date Who What changed Old value New value Message
2020-06-08 11:38:51 David O Neill bug added bug
2020-06-08 11:40:47 Dan Ackerson description Short description ==================================== We have intermittent re-accurance of what appears to be OVS being dead. Clients loose connection to the networks (vxlan,gre) with no root cause and any visible sign of issue. "Lights are on but non-one is home" Steps to reproduce =================== unknown Steps to reproduce ================= restart the VM openstack server restart creating new vms are not impacted and work immediately Sospreorts =========== ubuntu@files.support.canonical.com sosreport-..............-00282744-2020-06-08-typrlwu.tar.xz sosreport-...........-00282744-2020-06-08-qdvfplm.tar.xz openvswitch =========== dpkg --list | grep -i openv | awk '{print $2 " " $3 }' neutron-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0 neutron-plugin-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0 openvswitch-common 2.9.5-0ubuntu0.18.04.1~cloud0 openvswitch-switch 2.9.5-0ubuntu0.18.04.1~cloud0 python-openvswitch 2.9.5-0ubuntu0.18.04.1~cloud0 qemu ==== dpkg --list | grep -i qemu | awk '{print $2 " " $3 }' ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2.2~cloud0 ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2~cloud0 qemu 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-kvm 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-slof 20151103+dfsg-1ubuntu1.1 qemu-system 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-arm 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-common 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-mips 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-misc 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-ppc 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-s390x 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-sparc 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-x86 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-user 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-utils 1:2.11+dfsg-1ubuntu7.23~cloud0 Details ======= openstack server show f0c5b261-73ed-43cc-bd5d-82d181680d87 +-------------------------------------+------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | hidden to protect customer | | OS-EXT-SRV-ATTR:host | hidden to protect customer | | OS-EXT-SRV-ATTR:hypervisor_hostname | hidden to protect customer | | OS-EXT-SRV-ATTR:instance_name | instance-0000b9fc | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2020-01-28T13:16:46.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | DCS1-BO-SERVICES=172.16.15.22, xxxxxxxxxxxxxx | VM interface 172.16.15.22 =============================== virsh domiflist instance-0000b9fc Interface Type Source Model MAC ------------------------------------------------------- tapba67f7c2-bc bridge br-int virtio fa:16:3e:b5:56:26 OVS stat info ============= icmp, dns, dhcp, flows are visible /snap/bin/ovs-stat -p results --host dcs1-clp-nod23 --tree https://pastebin.canonical.com/p/NMZSRCqdPh/ Dead icmp ========= ping 172.16.15.22 PING 172.16.15.22 (172.16.15.22) 56(84) bytes of data. ^C --- 172.16.15.22 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1031ms Short description ==================================== We have intermittent reoccurance of what appears to be OVS being dead. Clients lose connection to the networks (vxlan,gre) with no root cause and any visible sign of issue. "Lights are on but noone is home" Steps to reproduce =================== unknown Steps to resolve ================= restart the VM `openstack server restart <id>` Newly created vms are not impacted and work immediately. Sospreorts =========== ubuntu@files.support.canonical.com sosreport-..............-00282744-2020-06-08-typrlwu.tar.xz sosreport-...........-00282744-2020-06-08-qdvfplm.tar.xz openvswitch =========== dpkg --list | grep -i openv | awk '{print $2 " " $3 }' neutron-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0 neutron-plugin-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0 openvswitch-common 2.9.5-0ubuntu0.18.04.1~cloud0 openvswitch-switch 2.9.5-0ubuntu0.18.04.1~cloud0 python-openvswitch 2.9.5-0ubuntu0.18.04.1~cloud0 qemu ==== dpkg --list | grep -i qemu | awk '{print $2 " " $3 }' ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2.2~cloud0 ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2~cloud0 qemu 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-kvm 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-slof 20151103+dfsg-1ubuntu1.1 qemu-system 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-arm 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-common 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-mips 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-misc 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-ppc 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-s390x 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-sparc 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-system-x86 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-user 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.23~cloud0 qemu-utils 1:2.11+dfsg-1ubuntu7.23~cloud0 Details ======= openstack server show f0c5b261-73ed-43cc-bd5d-82d181680d87 +-------------------------------------+------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | hidden to protect customer | | OS-EXT-SRV-ATTR:host | hidden to protect customer | | OS-EXT-SRV-ATTR:hypervisor_hostname | hidden to protect customer | | OS-EXT-SRV-ATTR:instance_name | instance-0000b9fc | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2020-01-28T13:16:46.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | DCS1-BO-SERVICES=172.16.15.22, xxxxxxxxxxxxxx | VM interface 172.16.15.22 =============================== virsh domiflist instance-0000b9fc Interface Type Source Model MAC ------------------------------------------------------- tapba67f7c2-bc bridge br-int virtio fa:16:3e:b5:56:26 OVS stat info ============= icmp, dns, dhcp, flows are visible /snap/bin/ovs-stat -p results --host dcs1-clp-nod23 --tree https://pastebin.canonical.com/p/NMZSRCqdPh/ Dead icmp ========= ping 172.16.15.22 PING 172.16.15.22 (172.16.15.22) 56(84) bytes of data. ^C --- 172.16.15.22 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1031ms
2021-06-30 10:14:17 James Page openvswitch (Ubuntu): status New Incomplete
2021-08-30 04:17:18 Launchpad Janitor openvswitch (Ubuntu): status Incomplete Expired