neutron-tempest-ovsfw fails 100% times

[Slawek Kaplonski]

2 or 3 tests are failing in neutron-tempest-ovsfw job. Example of failed job: http://logs.openstack.org/54/537654/4/check/neutron-tempest-ovsfw/5c90b2b/logs/testr_results.html.gz

From ovs L2 agent logs it looks that there is quite many issues related to ovsfw: http://logs.openstack.org/54/537654/4/check/neutron-tempest-ovsfw/5c90b2b/logs/screen-q-agt.txt.gz?level=ERROR

Errors are like:

Feb 05 21:23:55.740789 ubuntu-xenial-inap-mtl01-0002377325 neutron-openvswitch-agent[20990]: ERROR neutron.agent.linux.openvswitch_firewall.firewall [None req-bdd502f0-2b26-4566-b0e5-ee77d4d939ae None None] Initializing unfiltered port 7f46ff5c-5c9f-462f-8348-ef8741a9194d that does not exist in ovsdb: Port 7f46ff5c-5c9f-462f-8348-ef8741a9194d is not managed by this agent..: OVSFWPortNotFound: Port 7f46ff5c-5c9f-462f-8348-ef8741a9194d is not managed by this agent.
Feb 05 21:24:09.731637 ubuntu-xenial-inap-mtl01-0002377325 neutron-openvswitch-agent[20990]: ERROR neutron.agent.linux.openvswitch_firewall.firewall [None req-bdd502f0-2b26-4566-b0e5-ee77d4d939ae None None] Initializing unfiltered port b191da42-6eca-4ba3-a12b-244b07e6fe45 that does not exist in ovsdb: Port b191da42-6eca-4ba3-a12b-244b07e6fe45 is not managed by this agent..: OVSFWPortNotFound: Port b191da42-6eca-4ba3-a12b-244b07e6fe45 is not managed by this agent.

[Slawek Kaplonski (slaweq)]

After some digging on my local devstack VM I found that problem is happend every time when I do hard reboot of VM - after hard reboot I can't ssh to it.
In attachement I paste dump of "good" OF rules - those are rules from before hard reboot when VM is accessible and "bad" OF rules - those are rules from after hard reboot when VM is not accessible anymore.
Restart of ovs L2 agent fixes issue and VM is accessible again.

[Slawek Kaplonski (slaweq)]

From git bisect it looks that https://review.openstack.org/#/c/531414/ is responsible for this issue

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/542257

[Slawek Kaplonski (slaweq)]

With proposed patch error logs like:

Feb 08 16:01:09.736995 ubuntu-xenial-inap-mtl01-0002447359 neutron-openvswitch-agent[21127]: ERROR neutron.agent.linux.openvswitch_firewall.firewall [None req-4e4c338f-3439-45ae-94ff-de4e45f74887 None None] Initializing unfiltered port 6dbb103f-2148-43ee-8196-02a84fbd9a44 that does not exist in ovsdb: Port 6dbb103f-2148-43ee-8196-02a84fbd9a44 is not managed by this agent..: OVSFWPortNotFound: Port 6dbb103f-2148-43ee-8196-02a84fbd9a44 is not managed by this agent.

are still in L2 agent logs but on my local devstack tempest tests which failed on Zuul are passing.

I will continue work on how to eliminate those error logs also....

[Slawek Kaplonski (slaweq)]

So I checked this error message from OVS agent log on one example port "4b30e9f2-7e1a-4c1d-bfd7-93782de93e58". It looks that this error message appears after deletion of port. In neutron-server log port "4b30e9f2-7e1a-4c1d-bfd7-93782de93e58" was deleted at 15:58:18 - http://logs.openstack.org/57/542257/1/check/neutron-tempest-ovsfw/2734c62/logs/screen-q-svc.txt.gz#_Feb_08_15_58_18_368247
Just after that port was deleted on L2 agent's side: http://logs.openstack.org/57/542257/1/check/neutron-tempest-ovsfw/2734c62/logs/screen-q-agt.txt.gz?#_Feb_08_15_58_19_229898 and few milliseconds later there was this error message: http://logs.openstack.org/57/542257/1/check/neutron-tempest-ovsfw/2734c62/logs/screen-q-agt.txt.gz?#_Feb_08_15_58_19_711544

So this is not so big problem IMHO but it requires more detailed check still

[Slawek Kaplonski (slaweq)]

So this bug is only about failing tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard
test and patch which should fix it is proposed in https://review.openstack.org/#/c/542257/4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/542257
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=02cc3ca30733c88003331af26fbd364d703dd552
Submitter: Zuul
Branch: master

commit 02cc3ca30733c88003331af26fbd364d703dd552
Author: Sławek Kapłoński <email address hidden>
Date: Thu Feb 8 16:07:33 2018 +0100

    ovsfw: Update SG rules even if OVSFW Port is not found

    In patch [1] ovs firewall driver was changed and update_port_filter()
    method was not trying to initialize port flows in case when
    OVSFWPortNotFound is raised.
    Without that when e.g. instance is hard rebooted and of_port
    number is changed firewall openflow rules were not initialized
    for such port and there was no connectivity to such VM.

    [1] https://review.openstack.org/#/c/531414/

    Change-Id: I6d917cbac61293e9a956a2efcd9f2b720e4cac95
    Closes-Bug: #1747709

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/543944

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543944
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=25dfaef416e6f50bd25e75d7ede334970815ffd6
Submitter: Zuul
Branch: stable/queens

commit 25dfaef416e6f50bd25e75d7ede334970815ffd6
Author: Sławek Kapłoński <email address hidden>
Date: Thu Feb 8 16:07:33 2018 +0100

    ovsfw: Update SG rules even if OVSFW Port is not found

    In patch [1] ovs firewall driver was changed and update_port_filter()
    method was not trying to initialize port flows in case when
    OVSFWPortNotFound is raised.
    Without that when e.g. instance is hard rebooted and of_port
    number is changed firewall openflow rules were not initialized
    for such port and there was no connectivity to such VM.

    [1] https://review.openstack.org/#/c/531414/

    Change-Id: I6d917cbac61293e9a956a2efcd9f2b720e4cac95
    Closes-Bug: #1747709
    (cherry picked from commit 02cc3ca30733c88003331af26fbd364d703dd552)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/644791

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/644791
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6b41b07dc3ce8df049c49a4887e7f542be04b9a8
Submitter: Zuul
Branch: stable/pike

commit 6b41b07dc3ce8df049c49a4887e7f542be04b9a8
Author: Sławek Kapłoński <email address hidden>
Date: Thu Feb 8 16:07:33 2018 +0100

    ovsfw: Update SG rules even if OVSFW Port is not found

    In patch [1] ovs firewall driver was changed and update_port_filter()
    method was not trying to initialize port flows in case when
    OVSFWPortNotFound is raised.
    Without that when e.g. instance is hard rebooted and of_port
    number is changed firewall openflow rules were not initialized
    for such port and there was no connectivity to such VM.

    [1] https://review.openstack.org/#/c/531414/

    Change-Id: I6d917cbac61293e9a956a2efcd9f2b720e4cac95
    Closes-Bug: #1747709
    (cherry picked from commit 02cc3ca30733c88003331af26fbd364d703dd552)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.