IPv6 prefix delegation does not work with DVR

[Brian Haley]

Using a recent single-node devstack install, I tried to enable IPv6 prefix delegation following the exact instructions in the advanced networking guide - http://docs.openstack.org/liberty/networking-guide/adv_config_ipv6.html

$ neutron net-create ipv6-pd
$ neutron subnet-create ipv6-pd --name ipv6-pd-1 --ip_version 6 --ipv6_ra_mode slaac --ipv6_address_mode slaac
$ neutron router-interface-add ...

The l3-agent threw an exception:

2016-02-02 10:28:07.014 23328 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-f932415c-2cbd-4bcf-a54e-d7f7ee890908', 'ip', '-6', 'addr', 'add', 'fe80::f816:3eff:fe79:188c/64', 'scope', 'link', 'dev', 'qg-e4629b84-e8'] execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
2016-02-02 10:28:07.068 23328 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "qg-e4629b84-e8"

2016-02-02 10:28:07.069 23328 DEBUG oslo_concurrency.lockutils [-] Lock "l3-agent-pd" released by "neutron.agent.linux.pd.enable_subnet" :: held 0.055s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "qg-e4629b84-e8"
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/common/utils.py", line 368, in call
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 736, in process
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 388, in _process_internal_ports
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info interface_name, p['mac_address'])
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info return f(*args, **kwargs)
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/pd.py", line 81, in enable_subnet
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info self._add_lla(router, pd_info.get_bind_lla_with_mask())
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/pd.py", line 196, in _add_lla
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info 'link')
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 194, in add_ipv6_addr
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info device.addr.add(str(net), scope)
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 532, in add
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info self._as_root([net.version], tuple(args))
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 322, in _as_root
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 140, in execute
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info raise RuntimeError(msg)
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "qg-e4629b84-e8"
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info
2016-02-02 10:28:07.069 23328 ERROR neutron.agent.l3.router_info
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'f932415c-2cbd-4bcf-a54e-d7f7ee890908'
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 497, in _process_router_update
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 436, in _process_router_if_compatible
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 450, in _process_updated_router
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent ri.process(self)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 460, in process
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process(agent)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_router_base.py", line 30, in process
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process(agent)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 371, in call
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent self.logger(e)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 368, in call
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 736, in process
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 388, in _process_internal_ports
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent interface_name, p['mac_address'])
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent return f(*args, **kwargs)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/pd.py", line 81, in enable_subnet
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent self._add_lla(router, pd_info.get_bind_lla_with_mask())
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/pd.py", line 196, in _add_lla
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent 'link')
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 194, in add_ipv6_addr
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent device.addr.add(str(net), scope)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 532, in add
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent self._as_root([net.version], tuple(args))
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 322, in _as_root
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent use_root_namespace=use_root_namespace)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent log_fail_as_error=log_fail_as_error)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 140, in execute
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent raise RuntimeError(msg)
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "qg-e4629b84-e8"
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent
2016-02-02 10:28:07.075 23328 ERROR neutron.agent.l3.agent

That interface, qg-e4629b84-e8, lives in the SNAT namespace on a network node in dvr_snat mode.

Looking into it.

[Nate Johnston (nate-johnston)]

Is this devstack-specific, or does it occur outside of devstack as well?

[John Davidge (john-davidge)]

The note after the first paragraph of the Prefix Delegation section of [1] states that DVR is not supported, so failure is currently the expected behavior. I wonder if it's worth preventing the user from doing this with an error message instead of allowing the l3 agent to throw an exception.

[1] http://docs.openstack.org/liberty/networking-guide/adv_config_ipv6.html

[Brian Haley (brian-haley)]

John - I completely missed that note, but we'll see what we can do to possibly fix it. If there's any info you can pass along I'd appreciate it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/277657

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/277657
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277657
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7b5f01c2e58cd21ab3d46676ee6a3288012fbb45
Submitter: Jenkins
Branch: master

commit 7b5f01c2e58cd21ab3d46676ee6a3288012fbb45
Author: Brian Haley <email address hidden>
Date: Tue Jan 24 13:34:58 2017 -0500

    Add IPv6 Prefix Delegation support for DVR

    Unlike Legacy routers, DVR Edge Routers have their gateway
    interfaces in the SNAT namespace as opposed to the router
    namespace. Added a new method to the router_info class,
    get_gw_ns_name(), so callers can determine which namespace
    the gateway device lives in. This can then be over-ridden
    in the DVR Edge router class.

    The Prefix Delegation code will also now listen on the
    update_router event from the l3-agent and reset the namespace
    name if it changes.

    Closes-Bug: #1541406
    Change-Id: If6ada5027d0483fac7fc3ff935fee1edfc6e2759

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.0.0rc1 release candidate.

[sean redmond (sean-redmond1)]

This impacts newton, is a backport sensible?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/463531

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/463531
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a2f07de43f250b2114c9b4b074b61b4b4d8e450f
Submitter: Jenkins
Branch: stable/newton

commit a2f07de43f250b2114c9b4b074b61b4b4d8e450f
Author: Brian Haley <email address hidden>
Date: Tue Jan 24 13:34:58 2017 -0500

    Add IPv6 Prefix Delegation support for DVR

    Unlike Legacy routers, DVR Edge Routers have their gateway
    interfaces in the SNAT namespace as opposed to the router
    namespace. Added a new method to the router_info class,
    get_gw_ns_name(), so callers can determine which namespace
    the gateway device lives in. This can then be over-ridden
    in the DVR Edge router class.

    The Prefix Delegation code will also now listen on the
    update_router event from the l3-agent and reset the namespace
    name if it changes.

    Conflicts:
        neutron/tests/unit/agent/linux/test_pd.py

    Closes-Bug: #1541406
    Change-Id: If6ada5027d0483fac7fc3ff935fee1edfc6e2759
    (cherry picked from commit 7b5f01c2e58cd21ab3d46676ee6a3288012fbb45)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.4.0 release.