ovs-vswitchd crashed in functional test with segmentation fault

[Ihar Hrachyshka]

2017-03-03T18:39:35.095Z|00107|connmgr|INFO|test-br368b7744<->unix: 1 flow_mods in the last 0 s (1 adds)
2017-03-03T18:39:35.144Z|00108|connmgr|INFO|br-tunb76d9d9d9<->unix: 9 flow_mods in the last 0 s (9 adds)
2017-03-03T18:39:35.148Z|00109|connmgr|INFO|br-tunb76d9d9d9<->unix: 1 flow_mods in the last 0 s (1 adds)
2017-03-03T18:39:35.255Z|00003|daemon_unix(monitor)|WARN|2 crashes: pid 7753 died, killed (Segmentation fault), waiting until 10 seconds since last restart
2017-03-03T18:39:43.255Z|00004|daemon_unix(monitor)|ERR|2 crashes: pid 7753 died, killed (Segmentation fault), restarting
2017-03-03T18:39:43.256Z|00005|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0
2017-03-03T18:39:43.256Z|00006|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU cores
2017-03-03T18:39:43.256Z|00007|memory|INFO|8172 kB peak resident set size after 694.6 seconds
2017-03-03T18:39:43.256Z|00008|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2017-03-03T18:39:43.256Z|00009|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected

http://logs.openstack.org/73/441273/1/check/gate-neutron-dsvm-functional-ubuntu-xenial/82f5446/logs/openvswitch/ovs-vswitchd.txt.gz

This triggered functional test failure.

[Ihar Hrachyshka (ihar-hrachyshka)]

From syslog:

Mar 03 18:39:35 ubuntu-xenial-osic-cloud1-disk-7689986 kernel: [7753]: segfault at 40 ip 0000000000453ee0 sp 00007ffdb9ad1708 error 4 in ovs-vswitchd[400000+1bb000]

[Ihar Hrachyshka (ihar-hrachyshka)]

In test case log, we see:

dl_dst=fa:16:3e:1b:a9:34,table=73; Stdout: ; Stderr: ovs-ofctl: /var/run/openvswitch/test-br368b7744.mgmt: failed to open socket (Connection refused)

2017-03-03 18:39:42.546 17350 ERROR neutron.agent.common.ovs_lib [req-95e5962c-4743-4dd7-bf30-5462c32a7ef9 - - - - -] Unable to execute ['ovs-ofctl', 'del-flows', 'test-br368b7744', '-']. Exception: Exit code: 1; Stdin: dl_dst=fa:16:3e:1b:a9:34,table=0
in_port=2,table=0
reg5=2
dl_dst=fa:16:3e:1b:a9:34,table=73; Stdout: ; Stderr: ovs-ofctl: /var/run/openvswitch/test-br368b7744.mgmt: failed to open socket (Connection refused)

http://logs.openstack.org/73/441273/1/check/gate-neutron-dsvm-functional-ubuntu-xenial/82f5446/logs/dsvm-functional-logs/neutron.tests.functional.agent.test_firewall.FirewallTestCaseIPv6.test_tcp_from_specific_address_OVS-Firewall-Driver,native_.txt.gz#_2017-03-03_18_39_42_546

[Ihar Hrachyshka (ihar-hrachyshka)]

The issue is not too frequent, and is in platform. We can't do much about it, so lowering importance.

[Ihar Hrachyshka (ihar-hrachyshka)]

We switched to UCA that should deliver a new openvswitch to us (2.5.2). Let's close the bug and monitor if it happens again. If it does, let's reopen.