Neutron server error: failed to update port DOWN

Bug #1835731 reported by Oleg Bondarev on 2019-07-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
High
Oleg Bondarev

Bug Description

Before adding extra logging:

2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc [req-2b7602fb-e990-45ee-974f-ef3b55b41bed - - - - -] Failed to update device d75fca78-2f64-4c5a-9a94-6684c753bf3d down

After adding logging:

2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc [req-2b7602fb-e990-45ee-974f-ef3b55b41bed - - - - -] Failed to update device d75fca78-2f64-4c5a-9a94-6684c753bf3d down: 'NoneType' object has no attribute 'started_at': AttributeError: 'NoneType' object has no attribute 'started_at'2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc Traceback (most recent call last):2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/rpc.py", line 367, in update_device_list2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc **kwargs)2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/rpc.py", line 233, in update_device_down2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc n_const.PORT_STATUS_DOWN, host)2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/rpc.py", line 319, in notify_l2pop_port_wiring2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc agent_restarted = l2pop_driver.obj.agent_restarted(port_context)2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/mech_driver.py", line 253, in agent_restarted2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc if l2pop_db.get_agent_uptime(agent) < cfg.CONF.l2pop.agent_boot_time:2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/l2pop/db.py", line 51, in get_agent_uptime2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc return timeutils.delta_seconds(agent.started_at,2019-07-03 13:35:54,701.701 17220 ERROR neutron.plugins.ml2.rpc AttributeError: 'NoneType' object has no attribute 'started_at'

Fix proposed to branch: master
Review: https://review.opendev.org/669640

Changed in neutron:
status: New → In Progress
Changed in neutron:
importance: Undecided → High
tags: added: l2-pop l3-dvr-backlog

We should backport the patch up to Ocata, according to https://review.opendev.org/#/q/a5244d6d44d2b66de27dc77efa7830fa657260be (see patch description https://review.opendev.org/669640)

tags: added: rocky-backport-potential stein-backport-potential

Yes I agree that we should backport this to ocata as well.

tags: added: ocata-backport-potential

Change abandoned by Oleg Bondarev (<email address hidden>) on branch: master
Review: https://review.opendev.org/669640

Reviewed: https://review.opendev.org/671964
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c3a3031f78b3ea2d16d4eb08b181916dbc6f4c45
Submitter: Zuul
Branch: stable/stein

commit c3a3031f78b3ea2d16d4eb08b181916dbc6f4c45
Author: Oleg Bondarev <email address hidden>
Date: Mon Jul 8 15:21:18 2019 +0400

    Check for agent restarted after checking for DVR port

    This is a stable-only fix since code around the change was removed
    in master: https://review.opendev.org/#/c/641866

    Commit a5244d6d44d2b66de27dc77efa7830fa657260be changed the check
    order so regular non-dvr ports are checked for agent restarted.
    However regular ports may be unbound already, which leads to the
    error in the bug description: agent_restarted check is done against
    a 'None' agent.
    This patch fixed logic back - only check agent_restarted for dvr ports.

    This also adds some logging to have a clue why update port up/down fails.

    Change-Id: I3ad59864eeb42916d2cf15a5292d5aa9484f6e91
    Closes-Bug: #1835731

tags: added: in-stable-stein

Reviewed: https://review.opendev.org/671965
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3ba9c03c05f1cb423098b5e775df9c51f6ba48a7
Submitter: Zuul
Branch: stable/rocky

commit 3ba9c03c05f1cb423098b5e775df9c51f6ba48a7
Author: Oleg Bondarev <email address hidden>
Date: Mon Jul 8 15:21:18 2019 +0400

    Check for agent restarted after checking for DVR port

    This is a stable-only fix since code around the change was removed
    in master: https://review.opendev.org/#/c/641866

    Commit a5244d6d44d2b66de27dc77efa7830fa657260be changed the check
    order so regular non-dvr ports are checked for agent restarted.
    However regular ports may be unbound already, which leads to the
    error in the bug description: agent_restarted check is done against
    a 'None' agent.
    This patch fixed logic back - only check agent_restarted for dvr ports.

    This also adds some logging to have a clue why update port up/down fails.

    Change-Id: I3ad59864eeb42916d2cf15a5292d5aa9484f6e91
    Closes-Bug: #1835731
    (cherry picked from commit c3a3031f78b3ea2d16d4eb08b181916dbc6f4c45)

tags: added: in-stable-rocky

Reviewed: https://review.opendev.org/671966
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2318a60046c972ed1f9356e9159b583c48947eae
Submitter: Zuul
Branch: stable/queens

commit 2318a60046c972ed1f9356e9159b583c48947eae
Author: Oleg Bondarev <email address hidden>
Date: Mon Jul 8 15:21:18 2019 +0400

    Check for agent restarted after checking for DVR port

    This is a stable-only fix since code around the change was removed
    in master: https://review.opendev.org/#/c/641866

    Commit a5244d6d44d2b66de27dc77efa7830fa657260be changed the check
    order so regular non-dvr ports are checked for agent restarted.
    However regular ports may be unbound already, which leads to the
    error in the bug description: agent_restarted check is done against
    a 'None' agent.
    This patch fixed logic back - only check agent_restarted for dvr ports.

    This also adds some logging to have a clue why update port up/down fails.

    Change-Id: I3ad59864eeb42916d2cf15a5292d5aa9484f6e91
    Closes-Bug: #1835731
    (cherry picked from commit c3a3031f78b3ea2d16d4eb08b181916dbc6f4c45)

tags: added: in-stable-queens

Reviewed: https://review.opendev.org/671967
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6f595d3e01fc026aebc43dca08fb9f7f4a2d29a7
Submitter: Zuul
Branch: stable/pike

commit 6f595d3e01fc026aebc43dca08fb9f7f4a2d29a7
Author: Oleg Bondarev <email address hidden>
Date: Mon Jul 8 15:21:18 2019 +0400

    Check for agent restarted after checking for DVR port

    This is a stable-only fix since code around the change was removed
    in master: https://review.opendev.org/#/c/641866

    Commit a5244d6d44d2b66de27dc77efa7830fa657260be changed the check
    order so regular non-dvr ports are checked for agent restarted.
    However regular ports may be unbound already, which leads to the
    error in the bug description: agent_restarted check is done against
    a 'None' agent.
    This patch fixed logic back - only check agent_restarted for dvr ports.

    This also adds some logging to have a clue why update port up/down fails.

    Change-Id: I3ad59864eeb42916d2cf15a5292d5aa9484f6e91
    Closes-Bug: #1835731
    (cherry picked from commit c3a3031f78b3ea2d16d4eb08b181916dbc6f4c45)

tags: added: in-stable-pike

This issue was fixed in the openstack/neutron 14.0.3 release.

This issue was fixed in the openstack/neutron 13.0.5 release.

This issue was fixed in the openstack/neutron 12.1.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers