Neutron server unable to sync HA info after race between HA router creating and deleting

[LIU Yulong]

Neutron server will not be able to sync ha router data after race happened between get_ha_router_port_bindings and HA router deleting API call.

Exception:
File "neutron/db/db_base_plugin_v2.py", line 921, in _make_port_dict
    res = {"id": port["id"],
TypeError: 'NoneType' object has no attribute '__getitem__'

Trace:
http://paste.openstack.org/show/473839/

The new trace:
neutron server side:
http://paste.openstack.org/show/489511/
l3 agent side:
http://paste.openstack.org/show/489509/

[ugvddm (271025598-9)]

please get more information, log or how to reproduce it, thanks

[LIU Yulong (dragon889)]

@ugvddm (271025598-9), thank you, for more information please see here:
https://bugs.launchpad.net/neutron/+bug/1523780
and the patch is here:
https://review.openstack.org/#/c/265680/

[LIU Yulong (dragon889)]

Patch:
https://review.openstack.org/#/c/265680/

[Kevin Benton (kevinbenton)]

Please include the full stack traces in your bug reports. This by itself is not helpful to reproduce or even understand the issue.

[John Schwarz (jschwarz)]

Latest developments: this has been re-encountered in https://bugs.launchpad.net/neutron/+bug/1605546. That bug has been marked as duplicate of this one and a patch (https://review.openstack.org/#/c/265680/) is being worked on that should resolve this issue.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/348215

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/265680
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=179b8301edad50f999417f52b77092a496fb448e
Submitter: Jenkins
Branch: master

commit 179b8301edad50f999417f52b77092a496fb448e
Author: LIU Yulong <email address hidden>
Date: Mon Jan 11 11:31:36 2016 +0800

    Filter HA router without HA port bindings after race conditions

    Neutron server will not be able to sync ha router data after
    race happened between get_ha_router_port_bindings and HA router
    deleting API call. Once the ports of L3HARouterAgentPortBinding
    were deleted the _process_sync_ha_data may get a None binding
    port, and then the _process_sync_ha_data will fail to get the
    HA interface port info due to the None port. This patch will
    filter the bindings without port.

    Change-Id: Ie38baf061d678fc5d768195b25241efbad74e42f
    Closes-Bug: #1533457

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/349238

[Bernhard (b-krieger)]

After i have applied the patch i get still the exceptions

from vpn-agent.log:

2016-08-01 09:42:51.600 53157 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a9806b97-0850-4f40-8c73-c1fac666d21b', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2016-08-01 09:42:51.613 53157 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
2016-08-01 09:42:51.613 53157 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2016-08-01 09:42:51.613 53157 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2016-08-01 09:42:51.614 53157 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a9806b97-0850-4f40-8c73-c1fac666d21b', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2016-08-01 09:42:51.625 53157 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
2016-08-01 09:42:51.626 53157 DEBUG neutron.agent.l3.router_info [-] Terminating radvd daemon in router device: a9806b97-0850-4f40-8c73-c1fac666d21b disable_radvd /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:446
2016-08-01 09:42:51.626 53157 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/a9806b97-0850-4f40-8c73-c1fac666d21b.pid.radvd get_value_from_file /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:225
2016-08-01 09:42:51.627 53157 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/a9806b97-0850-4f40-8c73-c1fac666d21b.pid.radvd get_value_from_file /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:225
2016-08-01 09:42:51.627 53157 DEBUG neutron.agent.linux.external_process [-] No process started for a9806b97-0850-4f40-8c73-c1fac666d21b disable /usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py:118
2016-08-01 09:42:51.627 53157 DEBUG neutron.agent.linux.ra [-] radvd disabled for router a9806b97-0850-4f40-8c73-c1fac666d21b disable /usr/lib/python2.7/site-packages/neutron/agent/linux/ra.py:190
2016-08-01 09:42:51.627 53157 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-a9806b97-0850-4f40-8c73-c1fac666d21b', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2016-08-01 09:42:51.632 53157 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-a9806b97-0850-4f40-8c73-c1fac666d21b": No such file or dir...

[LIU Yulong (dragon889)]

@Bernhard (b-krieger)， hi, according to the log you added here, that issue was handled here:
https://bugs.launchpad.net/neutron/+bug/1607381
https://review.openstack.org/#/c/265672/

And in here:
https://review.openstack.org/#/c/265672/10//COMMIT_MSG
John and I have discussed something about that,
the l3 agent still have chance to get a HA router without ha_port.

[John Schwarz (jschwarz)]

@Bernhard, it seems like you have stumbled into https://bugs.launchpad.net/neutron/+bug/1606844 which deals with the specific issue you've encountered.

However, the issue is triggered by some error in the server-side - please look at the neutron-server side for errors at roughly that time and provide traces/warnings/errors that appear there.

John.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/349238
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=55a35196f9b9029c8e8428b75ec7f7cf0d13cff5
Submitter: Jenkins
Branch: stable/mitaka

commit 55a35196f9b9029c8e8428b75ec7f7cf0d13cff5
Author: LIU Yulong <email address hidden>
Date: Mon Jan 11 11:31:36 2016 +0800

    Filter HA router without HA port bindings after race conditions

    Neutron server will not be able to sync ha router data after
    race happened between get_ha_router_port_bindings and HA router
    deleting API call. Once the ports of L3HARouterAgentPortBinding
    were deleted the _process_sync_ha_data may get a None binding
    port, and then the _process_sync_ha_data will fail to get the
    HA interface port info due to the None port. This patch will
    filter the bindings without port.

    Change-Id: Ie38baf061d678fc5d768195b25241efbad74e42f
    Closes-Bug: #1533457
    (cherry picked from commit 179b8301edad50f999417f52b77092a496fb448e)

[Bernhard (b-krieger)]

Added this patch and patch for bug 1606844 ( as john mentioned).
Still getting into race condition when i delete a router.

http://paste.openstack.org/show/565708/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin Benton (<email address hidden>) on branch: master
Review: https://review.openstack.org/348215
Reason: This isn't necessary any more

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 8.3.0 release.