neutron

Restarting L3 agent when PD is used fails due to IPAddressAlreadyExists error

Bug #1892362 reported by Slawek Kaplonski
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Slawek Kaplonski

Bug Description

When there is router with subnet with enabled prefix delegation plugged and L3 agent is restarted it's failing with error like:

2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 7ef0983f-60a4-4389-909c-785eb6c9a51e: neutron.privileged.agent.linux.ip_lib.IpAddressAlreadyExists: IP address fe80::f816:3eff:fec8:b9a8 already configured on qg-351b89d8-61.
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 694, in _process_routers_if_compatible
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 540, in _process_router_if_compatible
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 581, in _process_updated_router
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent ri.process()
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 161, in call
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent self.logger(e)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent self.force_reraise()
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent raise value
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1191, in process
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent self.process_external()
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 949, in process_external
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent self._process_external_gateway(ex_gw_port)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 833, in _process_external_gateway
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent interface_name)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent return f(*args, **kwargs)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/pd.py", line 144, in add_gw_interface
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent self._add_lla(router, bind_lla_with_mask)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/pd.py", line 220, in _add_lla
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent 'link')
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/interface.py", line 208, in add_ipv6_addr
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent device.addr.add(str(net), scope)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 508, in add
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent add_broadcast)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 872, in add_ip_address
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent device, namespace, scope, broadcast)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent return input_func(*args, **kwargs)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 244, in _wrap
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent raise exc_type(*result[2])
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.IpAddressAlreadyExists: IP address fe80::f816:3eff:fec8:b9a8 already configured on qg-351b89d8-61.
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent

To workaround this problem I had to e.g. remove qrouter-XXX namespace from the node and then restart L3 agent. In that case all was configured without this error.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/747393

Changed in neutron:
assignee: nobody → Slawek Kaplonski (slaweq)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/747393
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=81d375d39ab1affe4b0a29437eaabc387fb1c570
Submitter: Zuul
Branch: master

commit 81d375d39ab1affe4b0a29437eaabc387fb1c570
Author: Slawek Kaplonski <email address hidden>
Date: Fri Aug 21 13:15:34 2020 +0200

    Handle properly existing LLA address during l3 agent restart

    In case when L3 agent is hosting routers which have got subnets
    with Prefix Delegation enabled, agent couldn't properly handle
    IpAddressAlreadyExists exception raised when pd module tries to
    configure link local IPv6 addresses.

    Now this is fixed and L3 agent can restart without problems in such
    case.

    Change-Id: Icc995f7b2b465921e41342711d17539f16ead0ce
    Closes-Bug: #1892362

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/747870

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/747871

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/747874

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/747875

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/747876

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/747871
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b934ae57f1494e14cc3f935f23a815cee6ac7c03
Submitter: Zuul
Branch: stable/train

commit b934ae57f1494e14cc3f935f23a815cee6ac7c03
Author: Slawek Kaplonski <email address hidden>
Date: Fri Aug 21 13:15:34 2020 +0200

    Handle properly existing LLA address during l3 agent restart

    In case when L3 agent is hosting routers which have got subnets
    with Prefix Delegation enabled, agent couldn't properly handle
    IpAddressAlreadyExists exception raised when pd module tries to
    configure link local IPv6 addresses.

    Now this is fixed and L3 agent can restart without problems in such
    case.

    Change-Id: Icc995f7b2b465921e41342711d17539f16ead0ce
    Closes-Bug: #1892362
    (cherry picked from commit 81d375d39ab1affe4b0a29437eaabc387fb1c570)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/747870
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b7d3a9471c4f08d0a0513a1d6e66bb22c1be8429
Submitter: Zuul
Branch: stable/ussuri

commit b7d3a9471c4f08d0a0513a1d6e66bb22c1be8429
Author: Slawek Kaplonski <email address hidden>
Date: Fri Aug 21 13:15:34 2020 +0200

    Handle properly existing LLA address during l3 agent restart

    In case when L3 agent is hosting routers which have got subnets
    with Prefix Delegation enabled, agent couldn't properly handle
    IpAddressAlreadyExists exception raised when pd module tries to
    configure link local IPv6 addresses.

    Now this is fixed and L3 agent can restart without problems in such
    case.

    Change-Id: Icc995f7b2b465921e41342711d17539f16ead0ce
    Closes-Bug: #1892362
    (cherry picked from commit 81d375d39ab1affe4b0a29437eaabc387fb1c570)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/747874
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2400a2b15bd6a5f8d1ec3ae82bab8cacd1288a0b
Submitter: Zuul
Branch: stable/stein

commit 2400a2b15bd6a5f8d1ec3ae82bab8cacd1288a0b
Author: Slawek Kaplonski <email address hidden>
Date: Fri Aug 21 13:15:34 2020 +0200

    Handle properly existing LLA address during l3 agent restart

    In case when L3 agent is hosting routers which have got subnets
    with Prefix Delegation enabled, agent couldn't properly handle
    IpAddressAlreadyExists exception raised when pd module tries to
    configure link local IPv6 addresses.

    Now this is fixed and L3 agent can restart without problems in such
    case.

    Conflicts:
        neutron/agent/linux/pd.py

    Change-Id: Icc995f7b2b465921e41342711d17539f16ead0ce
    Closes-Bug: #1892362
    (cherry picked from commit 81d375d39ab1affe4b0a29437eaabc387fb1c570)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/queens)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/747876
Reason: Don't seems like needed in Queens

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/747875
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=714fb4e3d2b33c144b25aa351f585b44cf70d20b
Submitter: Zuul
Branch: stable/rocky

commit 714fb4e3d2b33c144b25aa351f585b44cf70d20b
Author: Slawek Kaplonski <email address hidden>
Date: Fri Aug 21 13:15:34 2020 +0200

    Handle properly existing LLA address during l3 agent restart

    In case when L3 agent is hosting routers which have got subnets
    with Prefix Delegation enabled, agent couldn't properly handle
    IpAddressAlreadyExists exception raised when pd module tries to
    configure link local IPv6 addresses.

    Now this is fixed and L3 agent can restart without problems in such
    case.

    Conflicts:
        neutron/agent/linux/pd.py

    Change-Id: Icc995f7b2b465921e41342711d17539f16ead0ce
    Closes-Bug: #1892362
    (cherry picked from commit 81d375d39ab1affe4b0a29437eaabc387fb1c570)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron rocky-eol

This issue was fixed in the openstack/neutron rocky-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.