Activity log for bug #1418097

Date Who What changed Old value New value Message
2015-02-04 16:05:45 Assaf Muller bug added bug
2015-02-04 16:06:06 Assaf Muller bug added subscriber Carl Baldwin
2015-02-04 16:06:25 Assaf Muller bug added subscriber Armando Migliaccio
2015-02-04 16:06:34 Assaf Muller bug added subscriber Terry Wilson
2015-02-04 16:19:52 Assaf Muller description Still early in triage process. Running: tox -e dsvm-functional neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_lifecycle_without_ha_with_snat_with_fips Results in a kernel panic or ovs-vswitchd crash, with this patch: https://review.openstack.org/#/c/151882/ Rebasing one patch earlier results in smooth test runs. Note: It's not necessarily that specific test that is causing issues, but it is one that reproduces the issue reliably. Running the non-DVR test case (L3AgentTestCase) works fine. So does running the OVS lib functional tests. My devstack VM is running on: * Fedora 20 * openvswitch-2.3.1-1.fc20.x86_64 * kernel-3.17.2-200.fc20.x86_64 I checked with team mates, the same test on the same code passes with: * CentOS 7 * openvswitch-2.1.2-2.el7.centos.1.x86_64 * kernel-3.10.0-123.6.3.el7.x86_64 This may happen as a result of a Neutron code change in the patch above, possibly coupled with a regression in a recent OVS version. Still early in triage process. Running: tox -e dsvm-functional neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_lifecycle_without_ha_with_snat_with_fips Results in a kernel panic or ovs-vswitchd crash, with this patch: https://review.openstack.org/#/c/151882/ Rebasing one patch earlier results in smooth test runs. If I don't get a kernel panic, then ovs-vswitchd crashes, at which point most ovs-vsctl commands fail. Restarting the openvswitch service doesn't seem to solve it, only restarting the machine. Note: It's not necessarily that specific test that is causing issues, but it is one that reproduces the issue reliably. Running the non-DVR test case (L3AgentTestCase) works fine. So does running the OVS lib functional tests. My devstack VM is running on: * Fedora 20 * openvswitch-2.3.1-1.fc20.x86_64 * kernel-3.17.2-200.fc20.x86_64 I checked with team mates, the same test on the same code passes with: * CentOS 7 * openvswitch-2.1.2-2.el7.centos.1.x86_64 * kernel-3.10.0-123.6.3.el7.x86_64 This may happen as a result of a Neutron code change in the patch above, possibly coupled with a regression in a recent OVS version.
2015-02-04 16:27:24 Assaf Muller description Still early in triage process. Running: tox -e dsvm-functional neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_lifecycle_without_ha_with_snat_with_fips Results in a kernel panic or ovs-vswitchd crash, with this patch: https://review.openstack.org/#/c/151882/ Rebasing one patch earlier results in smooth test runs. If I don't get a kernel panic, then ovs-vswitchd crashes, at which point most ovs-vsctl commands fail. Restarting the openvswitch service doesn't seem to solve it, only restarting the machine. Note: It's not necessarily that specific test that is causing issues, but it is one that reproduces the issue reliably. Running the non-DVR test case (L3AgentTestCase) works fine. So does running the OVS lib functional tests. My devstack VM is running on: * Fedora 20 * openvswitch-2.3.1-1.fc20.x86_64 * kernel-3.17.2-200.fc20.x86_64 I checked with team mates, the same test on the same code passes with: * CentOS 7 * openvswitch-2.1.2-2.el7.centos.1.x86_64 * kernel-3.10.0-123.6.3.el7.x86_64 This may happen as a result of a Neutron code change in the patch above, possibly coupled with a regression in a recent OVS version. Still early in triage process. Running: tox -e dsvm-functional neutron.tests.functional.agent.test_l3_agent.TestDvrRouter.test_dvr_router_lifecycle_without_ha_with_snat_with_fips Results in a kernel panic or ovs-vswitchd crash, with this patch: https://review.openstack.org/#/c/151882/ Rebasing one patch earlier results in smooth test runs. If I don't get a kernel panic, then ovs-vswitchd crashes, at which point most ovs-vsctl commands fail. Restarting the openvswitch service doesn't seem to solve it, only restarting the machine. Note: It's not necessarily that specific test that is causing issues, but it is one that reproduces the issue reliably. Running the non-DVR test case (L3AgentTestCase) works fine. So does running the OVS lib functional tests. My devstack VM is running on: * Fedora 20 * openvswitch-2.3.1-1.fc20.x86_64 * kernel-3.17.2-200.fc20.x86_64 I checked with team mates, the same test on the same code passes with: * CentOS 7 * openvswitch-2.1.2-2.el7.centos.1.x86_64 * kernel-3.10.0-123.6.3.el7.x86_64 This may happen as a result of a Neutron code change in the patch above, possibly coupled with a regression in a recent OVS version. I think it's clear to assume that this is not only a test-only issue, but could happen in production as well.
2015-02-04 17:00:52 Kyle Mestery bug added subscriber Kyle Mestery
2015-02-10 22:10:58 Brian Haley bug added subscriber Brian Haley
2015-02-12 20:21:18 Assaf Muller neutron: status New Confirmed
2015-02-12 23:10:30 Armando Migliaccio neutron: importance Undecided Medium
2015-02-12 23:13:54 Armando Migliaccio tags l3-dvr-backlog
2015-02-13 09:54:55 Ihar Hrachyshka bug added subscriber Ihar Hrachyshka
2015-02-13 14:31:16 Kyle Mestery neutron: importance Medium High
2015-02-13 14:31:57 Kyle Mestery tags l3-dvr-backlog l3-dvr-backlog ovs
2015-02-16 14:20:59 Assaf Muller neutron: assignee Assaf Muller (amuller)
2015-02-16 14:24:25 OpenStack Infra neutron: status Confirmed In Progress
2015-02-17 17:02:02 Kyle Mestery neutron: milestone kilo-3
2015-02-17 19:42:30 OpenStack Infra neutron: status In Progress Fix Committed
2015-03-19 16:28:16 Thierry Carrez neutron: status Fix Committed Fix Released
2015-04-30 09:44:23 Thierry Carrez neutron: milestone kilo-3 2015.1.0