2020-06-08 11:40:47 |
Dan Ackerson |
description |
Short description
====================================
We have intermittent re-accurance of what appears to be OVS being dead.
Clients loose connection to the networks (vxlan,gre) with no root cause and any visible sign of issue. "Lights are on but non-one is home"
Steps to reproduce
===================
unknown
Steps to reproduce
=================
restart the VM
openstack server restart
creating new vms are not impacted and work immediately
Sospreorts
===========
ubuntu@files.support.canonical.com
sosreport-..............-00282744-2020-06-08-typrlwu.tar.xz
sosreport-...........-00282744-2020-06-08-qdvfplm.tar.xz
openvswitch
===========
dpkg --list | grep -i openv | awk '{print $2 " " $3 }'
neutron-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0
neutron-plugin-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0
openvswitch-common 2.9.5-0ubuntu0.18.04.1~cloud0
openvswitch-switch 2.9.5-0ubuntu0.18.04.1~cloud0
python-openvswitch 2.9.5-0ubuntu0.18.04.1~cloud0
qemu
====
dpkg --list | grep -i qemu | awk '{print $2 " " $3 }'
ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2.2~cloud0
ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2~cloud0
qemu 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-kvm 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-slof 20151103+dfsg-1ubuntu1.1
qemu-system 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-arm 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-common 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-mips 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-misc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-ppc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-s390x 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-sparc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-x86 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-user 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-utils 1:2.11+dfsg-1ubuntu7.23~cloud0
Details
=======
openstack server show f0c5b261-73ed-43cc-bd5d-82d181680d87
+-------------------------------------+------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | hidden to protect customer |
| OS-EXT-SRV-ATTR:host | hidden to protect customer |
| OS-EXT-SRV-ATTR:hypervisor_hostname | hidden to protect customer |
| OS-EXT-SRV-ATTR:instance_name | instance-0000b9fc |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-01-28T13:16:46.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | DCS1-BO-SERVICES=172.16.15.22, xxxxxxxxxxxxxx |
VM interface 172.16.15.22
===============================
virsh domiflist instance-0000b9fc
Interface Type Source Model MAC
-------------------------------------------------------
tapba67f7c2-bc bridge br-int virtio fa:16:3e:b5:56:26
OVS stat info
=============
icmp, dns, dhcp, flows are visible
/snap/bin/ovs-stat -p results --host dcs1-clp-nod23 --tree
https://pastebin.canonical.com/p/NMZSRCqdPh/
Dead icmp
=========
ping 172.16.15.22
PING 172.16.15.22 (172.16.15.22) 56(84) bytes of data.
^C
--- 172.16.15.22 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1031ms |
Short description
====================================
We have intermittent reoccurance of what appears to be OVS being dead.
Clients lose connection to the networks (vxlan,gre) with no root cause and any visible sign of issue. "Lights are on but noone is home"
Steps to reproduce
===================
unknown
Steps to resolve
=================
restart the VM
`openstack server restart <id>`
Newly created vms are not impacted and work immediately.
Sospreorts
===========
ubuntu@files.support.canonical.com
sosreport-..............-00282744-2020-06-08-typrlwu.tar.xz
sosreport-...........-00282744-2020-06-08-qdvfplm.tar.xz
openvswitch
===========
dpkg --list | grep -i openv | awk '{print $2 " " $3 }'
neutron-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0
neutron-plugin-openvswitch-agent 2:12.1.0-0ubuntu1~cloud0
openvswitch-common 2.9.5-0ubuntu0.18.04.1~cloud0
openvswitch-switch 2.9.5-0ubuntu0.18.04.1~cloud0
python-openvswitch 2.9.5-0ubuntu0.18.04.1~cloud0
qemu
====
dpkg --list | grep -i qemu | awk '{print $2 " " $3 }'
ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2.2~cloud0
ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2~cloud0
qemu 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-kvm 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-slof 20151103+dfsg-1ubuntu1.1
qemu-system 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-arm 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-common 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-mips 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-misc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-ppc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-s390x 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-sparc 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-system-x86 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-user 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.23~cloud0
qemu-utils 1:2.11+dfsg-1ubuntu7.23~cloud0
Details
=======
openstack server show f0c5b261-73ed-43cc-bd5d-82d181680d87
+-------------------------------------+------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | hidden to protect customer |
| OS-EXT-SRV-ATTR:host | hidden to protect customer |
| OS-EXT-SRV-ATTR:hypervisor_hostname | hidden to protect customer |
| OS-EXT-SRV-ATTR:instance_name | instance-0000b9fc |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-01-28T13:16:46.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | DCS1-BO-SERVICES=172.16.15.22, xxxxxxxxxxxxxx |
VM interface 172.16.15.22
===============================
virsh domiflist instance-0000b9fc
Interface Type Source Model MAC
-------------------------------------------------------
tapba67f7c2-bc bridge br-int virtio fa:16:3e:b5:56:26
OVS stat info
=============
icmp, dns, dhcp, flows are visible
/snap/bin/ovs-stat -p results --host dcs1-clp-nod23 --tree
https://pastebin.canonical.com/p/NMZSRCqdPh/
Dead icmp
=========
ping 172.16.15.22
PING 172.16.15.22 (172.16.15.22) 56(84) bytes of data.
^C
--- 172.16.15.22 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1031ms |
|