neutron

prefix delegation does not work in newton

Bug #1631693 reported by Matthew Thode
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
John Davidge
neutron ocata-1 "o1"

Bug Description

Using a provider network, set up and tested working as laid out here:
  http://docs.openstack.org/newton/install-guide-ubuntu/launch-instance-networks-selfservice.html

Using the prefix delegation as defined here:
  http://docs.openstack.org/newton/networking-guide/config-ipv6.html

Neutron tries to add the prefix_delegated network directly as just '::1/64', which fails.

subnet-show ipv6-pd-1
+-------------------+--------------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------------+
| allocation_pools | {"start": "::2", "end": "::ffff:ffff:ffff:ffff"} |
| cidr | ::/64 |
| created_at | 2016-10-09T03:39:31Z |
| description | |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | ::1 |
| host_routes | |
| id | 7aca0fbc-2700-4956-9ecf-25eb40923377 |
| ip_version | 6 |
| ipv6_address_mode | slaac |
| ipv6_ra_mode | slaac |
| name | ipv6-pd-1 |
| network_id | 642b292f-3dee-4444-be52-04520eaed9d1 |
| project_id | e88db48bc6f0406593d0fcb8b648babe |
| revision_number | 2 |
| service_types | |
| subnetpool_id | prefix_delegation |
| tenant_id | e88db48bc6f0406593d0fcb8b648babe |
| updated_at | 2016-10-09T03:39:31Z |
+-------------------+--------------------------------------------------+

2016-10-08 22:46:48.389 19498 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-85e10e09-a9a5-4568-ada3-534bd38f66b4', 'ip', '-6', 'addr', 'add', '::1/64', 'scope', 'global', 'dev', 'qr-76a5d305-56'] create_process /usr/lib64/python2.7/site-packages/neutron/agent/linux/utils.py:83

2016-10-08 22:46:48.969 19498 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: ebe5ebeb036842258fddf3a9958928de __call__ /usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-10-08 22:46:48.974 19498 DEBUG neutron.agent.l3.agent [req-66fe2ee5-2cbf-4d0f-892d-e48c50a1d562 user project - - -] Got routers updated notification :[u'85e10e09-a9a5-4568-ada3-534bd38f66b4'] routers_updated /usr/lib64/python2.7/site-packages/neutron/agent/l3/agent.py:400

2016-10-08 22:46:49.441 19498 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address

2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/common/utils.py", line 239, in call
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1060, in process
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 506, in _process_internal_ports
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info self.internal_network_added(p)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 429, in internal_network_added
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info mtu=port.get('mtu'))
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 407, in _internal_network_added
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info interface_name, ip_cidrs, namespace=ns_name)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/interface.py", line 160, in init_router_port
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info clean_connections=clean_connections)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/interface.py", line 129, in init_l3
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info device.addr.add(ip_cidr)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 577, in add
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info self._as_root([net.version], tuple(args))
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 364, in _as_root
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info raise RuntimeError(msg)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 85e10e09-a9a5-4568-ada3-534bd38f66b4
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/agent.py", line 508, in _process_router_update
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/agent.py", line 445, in _process_router_if_compatible
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/agent.py", line 460, in _process_updated_router
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent ri.process(self)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/common/utils.py", line 242, in call
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self.logger(e)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self.force_reraise()
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/common/utils.py", line 239, in call
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1060, in process
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 506, in _process_internal_ports
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self.internal_network_added(p)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 429, in internal_network_added
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent mtu=port.get('mtu'))
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 407, in _internal_network_added
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent interface_name, ip_cidrs, namespace=ns_name)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/interface.py", line 160, in init_router_port
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent clean_connections=clean_connections)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/interface.py", line 129, in init_l3
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent device.addr.add(ip_cidr)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 577, in add
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self._as_root([net.version], tuple(args))
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 364, in _as_root
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent use_root_namespace=use_root_namespace)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent log_fail_as_error=log_fail_as_error)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent raise RuntimeError(msg)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent

John Davidge (john-davidge)
Changed in neutron:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → John Davidge (john-davidge)
John Davidge (john-davidge)
tags: added: ipv6 l3-ipam-dhcp
Revision history for this message
John Davidge (john-davidge) wrote :

Looks like this was caused by the switch to pluggable IPAM. A PD-specific check in the non-pluggable backend is missing from the pluggable version. Fix incoming, thanks for the report.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/384577

Changed in neutron:
status: Confirmed → In Progress
Brian Haley (brian-haley)
tags: added: newton-backport-potential
Revision history for this message
Shannon McFarland (shmcfarl) wrote :
Download full text (10.1 KiB)

I cherry picked that fix and I still have no joy:
2016-10-13 13:23:52.288 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-e7f5ae6e-268c-473c-a6a2-7489be4866c9', 'ip', '-6', 'addr', 'add', '::1/64', 'scope', 'global', 'dev', 'qr-501c4487-47'] from (pid=381) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:99
2016-10-13 13:23:52.319 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address

2016-10-13 13:23:52.320 ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address

2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/common/utils.py", line 239, in call
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info return func(*args, **kwargs)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1060, in process
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 506, in _process_internal_ports
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info self.internal_network_added(p)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 429, in internal_network_added
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info mtu=port.get('mtu'))
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 407, in _internal_network_added
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info interface_name, ip_cidrs, namespace=ns_name)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 160, in init_router_port
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info clean_connections=clean_connections)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 129, in init_l3
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info device.addr.add(ip_cidr)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 577, in add
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info self._as_root([net.version], tuple(args))
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 364, in _as_root
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info log_fail_as_error=self...

Revision history for this message
Matthew Thode (prometheanfire) wrote :

That's odd, it worked for me, but I did fully recreate the network, subnet and router.

Revision history for this message
John Davidge (john-davidge) wrote :

Hey Shannon, are you sure you restarted everything? It shouldn't still be trying to create the port with a "::1/64" address.

Changed in neutron:
milestone: none → ocata-1
Revision history for this message
Shannon McFarland (shmcfarl) wrote :

I unstacked/stacked again, created the network, subnet, router and then attached the router to the public network and the new network.

I see this error which I don't think I have seen before:
2016-10-14 09:25:32.544 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 38b7fea1-63fd-4f37-bc22-533e9167ee54

Revision history for this message
Shannon McFarland (shmcfarl) wrote :

John, I unstacked, rebooted, cherry-picked the patch, stacked (verified the patch was there) and created the network/subnet, router and attached the interfaces. This is a mostly default Devstack test. I did notice that you were testing the Ubuntu install steps with provider networks which is different than my setup but I am certainly seeing the issue in my environment.

If you have a specific test setup that I should follow, then please send that over (you can send it on the email thread we have going) and I will replicate it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/384577
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=14ffea4444985eb041181d1bdef9af53ccb4d9f9
Submitter: Jenkins
Branch: master

commit 14ffea4444985eb041181d1bdef9af53ccb4d9f9
Author: John Davidge <email address hidden>
Date: Mon Oct 10 14:05:03 2016 +0100

    Fix IPv6 PD with pluggable IPAM

    A PD-sepcific check that was present in the non-pluggable IPAM
    backend was missing from the pluggable version, causing router
    interfaces to be created with a '::1' IP (equal to the gateway on
    PD subnets). The resulting command:

    ip netns exec <router_namespace> ip -6 addr add ::1/64

    would then fail. This patch restores the check to ensure that
    '::1' is not used for router interface ports, and introduces a
    test to protect against future regressions.

    Change-Id: I6a2e7cd60dd42d92b900a57bd8f3ffbc5909908e
    Closes-Bug: 1631693

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
Shannon McFarland (shmcfarl) wrote :

I got this working. Thanks for the support!

Revision history for this message
John Davidge (john-davidge) wrote :

Hey Shannon, glad you sorted this over the weekend! What did you have to change to get it working?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/388600

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/388600
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=93828c7e33ba46628f47cabb22610fc3f2db8096
Submitter: Jenkins
Branch: stable/newton

commit 93828c7e33ba46628f47cabb22610fc3f2db8096
Author: John Davidge <email address hidden>
Date: Mon Oct 10 14:05:03 2016 +0100

    Fix IPv6 PD with pluggable IPAM

    A PD-sepcific check that was present in the non-pluggable IPAM
    backend was missing from the pluggable version, causing router
    interfaces to be created with a '::1' IP (equal to the gateway on
    PD subnets). The resulting command:

    ip netns exec <router_namespace> ip -6 addr add ::1/64

    would then fail. This patch restores the check to ensure that
    '::1' is not used for router interface ports, and introduces a
    test to protect against future regressions.

    Change-Id: I6a2e7cd60dd42d92b900a57bd8f3ffbc5909908e
    Closes-Bug: 1631693
    (cherry picked from commit 14ffea4444985eb041181d1bdef9af53ccb4d9f9)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.1.0

This issue was fixed in the openstack/neutron 9.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.0.0b1

This issue was fixed in the openstack/neutron 10.0.0.0b1 development milestone.

Ihar Hrachyshka (ihar-hrachyshka)
tags: added: neutron-proactive-backport-potential
Ihar Hrachyshka (ihar-hrachyshka)
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421990

Ihar Hrachyshka (ihar-hrachyshka)
tags: removed: newton-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/mitaka)

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/421990
Reason: Mitaka is CVE only.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.