prefix delegation does not work in newton

Bug #1631693 reported by Matthew Thode
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
High
John Davidge

Bug Description

Using a provider network, set up and tested working as laid out here:
  http://docs.openstack.org/newton/install-guide-ubuntu/launch-instance-networks-selfservice.html

Using the prefix delegation as defined here:
  http://docs.openstack.org/newton/networking-guide/config-ipv6.html

Neutron tries to add the prefix_delegated network directly as just '::1/64', which fails.

subnet-show ipv6-pd-1
+-------------------+--------------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------------+
| allocation_pools | {"start": "::2", "end": "::ffff:ffff:ffff:ffff"} |
| cidr | ::/64 |
| created_at | 2016-10-09T03:39:31Z |
| description | |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | ::1 |
| host_routes | |
| id | 7aca0fbc-2700-4956-9ecf-25eb40923377 |
| ip_version | 6 |
| ipv6_address_mode | slaac |
| ipv6_ra_mode | slaac |
| name | ipv6-pd-1 |
| network_id | 642b292f-3dee-4444-be52-04520eaed9d1 |
| project_id | e88db48bc6f0406593d0fcb8b648babe |
| revision_number | 2 |
| service_types | |
| subnetpool_id | prefix_delegation |
| tenant_id | e88db48bc6f0406593d0fcb8b648babe |
| updated_at | 2016-10-09T03:39:31Z |
+-------------------+--------------------------------------------------+

2016-10-08 22:46:48.389 19498 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-85e10e09-a9a5-4568-ada3-534bd38f66b4', 'ip', '-6', 'addr', 'add', '::1/64', 'scope', 'global', 'dev', 'qr-76a5d305-56'] create_process /usr/lib64/python2.7/site-packages/neutron/agent/linux/utils.py:83

2016-10-08 22:46:48.969 19498 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: ebe5ebeb036842258fddf3a9958928de __call__ /usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:196
2016-10-08 22:46:48.974 19498 DEBUG neutron.agent.l3.agent [req-66fe2ee5-2cbf-4d0f-892d-e48c50a1d562 user project - - -] Got routers updated notification :[u'85e10e09-a9a5-4568-ada3-534bd38f66b4'] routers_updated /usr/lib64/python2.7/site-packages/neutron/agent/l3/agent.py:400

2016-10-08 22:46:49.441 19498 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address

2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/common/utils.py", line 239, in call
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1060, in process
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 506, in _process_internal_ports
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info self.internal_network_added(p)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 429, in internal_network_added
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info mtu=port.get('mtu'))
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 407, in _internal_network_added
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info interface_name, ip_cidrs, namespace=ns_name)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/interface.py", line 160, in init_router_port
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info clean_connections=clean_connections)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/interface.py", line 129, in init_l3
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info device.addr.add(ip_cidr)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 577, in add
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info self._as_root([net.version], tuple(args))
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 364, in _as_root
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info raise RuntimeError(msg)
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info
2016-10-08 22:46:49.442 19498 ERROR neutron.agent.l3.router_info
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 85e10e09-a9a5-4568-ada3-534bd38f66b4
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/agent.py", line 508, in _process_router_update
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/agent.py", line 445, in _process_router_if_compatible
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/agent.py", line 460, in _process_updated_router
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent ri.process(self)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/common/utils.py", line 242, in call
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self.logger(e)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self.force_reraise()
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/common/utils.py", line 239, in call
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1060, in process
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 506, in _process_internal_ports
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self.internal_network_added(p)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 429, in internal_network_added
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent mtu=port.get('mtu'))
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/l3/router_info.py", line 407, in _internal_network_added
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent interface_name, ip_cidrs, namespace=ns_name)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/interface.py", line 160, in init_router_port
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent clean_connections=clean_connections)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/interface.py", line 129, in init_l3
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent device.addr.add(ip_cidr)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 577, in add
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent self._as_root([net.version], tuple(args))
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 364, in _as_root
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent use_root_namespace=use_root_namespace)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent log_fail_as_error=log_fail_as_error)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent File "/usr/lib64/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent raise RuntimeError(msg)
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent
2016-10-08 22:46:49.446 19498 ERROR neutron.agent.l3.agent

Changed in neutron:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → John Davidge (john-davidge)
tags: added: ipv6 l3-ipam-dhcp
Revision history for this message
John Davidge (john-davidge) wrote :

Looks like this was caused by the switch to pluggable IPAM. A PD-specific check in the non-pluggable backend is missing from the pluggable version. Fix incoming, thanks for the report.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/384577

Changed in neutron:
status: Confirmed → In Progress
tags: added: newton-backport-potential
Revision history for this message
Shannon McFarland (shmcfarl) wrote :
Download full text (10.1 KiB)

I cherry picked that fix and I still have no joy:
2016-10-13 13:23:52.288 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-e7f5ae6e-268c-473c-a6a2-7489be4866c9', 'ip', '-6', 'addr', 'add', '::1/64', 'scope', 'global', 'dev', 'qr-501c4487-47'] from (pid=381) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:99
2016-10-13 13:23:52.319 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address

2016-10-13 13:23:52.320 ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Cannot assign requested address

2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/common/utils.py", line 239, in call
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info return func(*args, **kwargs)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1060, in process
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 506, in _process_internal_ports
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info self.internal_network_added(p)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 429, in internal_network_added
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info mtu=port.get('mtu'))
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 407, in _internal_network_added
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info interface_name, ip_cidrs, namespace=ns_name)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 160, in init_router_port
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info clean_connections=clean_connections)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 129, in init_l3
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info device.addr.add(ip_cidr)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 577, in add
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info self._as_root([net.version], tuple(args))
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 364, in _as_root
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-10-13 13:23:52.320 TRACE neutron.agent.l3.router_info log_fail_as_error=self...

Revision history for this message
Matthew Thode (prometheanfire) wrote :

That's odd, it worked for me, but I did fully recreate the network, subnet and router.

Revision history for this message
John Davidge (john-davidge) wrote :

Hey Shannon, are you sure you restarted everything? It shouldn't still be trying to create the port with a "::1/64" address.

Changed in neutron:
milestone: none → ocata-1
Revision history for this message
Shannon McFarland (shmcfarl) wrote :

I unstacked/stacked again, created the network, subnet, router and then attached the router to the public network and the new network.

I see this error which I don't think I have seen before:
2016-10-14 09:25:32.544 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 38b7fea1-63fd-4f37-bc22-533e9167ee54

Revision history for this message
Shannon McFarland (shmcfarl) wrote :

John, I unstacked, rebooted, cherry-picked the patch, stacked (verified the patch was there) and created the network/subnet, router and attached the interfaces. This is a mostly default Devstack test. I did notice that you were testing the Ubuntu install steps with provider networks which is different than my setup but I am certainly seeing the issue in my environment.

If you have a specific test setup that I should follow, then please send that over (you can send it on the email thread we have going) and I will replicate it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/384577
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=14ffea4444985eb041181d1bdef9af53ccb4d9f9
Submitter: Jenkins
Branch: master

commit 14ffea4444985eb041181d1bdef9af53ccb4d9f9
Author: John Davidge <email address hidden>
Date: Mon Oct 10 14:05:03 2016 +0100

    Fix IPv6 PD with pluggable IPAM

    A PD-sepcific check that was present in the non-pluggable IPAM
    backend was missing from the pluggable version, causing router
    interfaces to be created with a '::1' IP (equal to the gateway on
    PD subnets). The resulting command:

    ip netns exec <router_namespace> ip -6 addr add ::1/64

    would then fail. This patch restores the check to ensure that
    '::1' is not used for router interface ports, and introduces a
    test to protect against future regressions.

    Change-Id: I6a2e7cd60dd42d92b900a57bd8f3ffbc5909908e
    Closes-Bug: 1631693

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
Shannon McFarland (shmcfarl) wrote :

I got this working. Thanks for the support!

Revision history for this message
John Davidge (john-davidge) wrote :

Hey Shannon, glad you sorted this over the weekend! What did you have to change to get it working?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/388600

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/388600
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=93828c7e33ba46628f47cabb22610fc3f2db8096
Submitter: Jenkins
Branch: stable/newton

commit 93828c7e33ba46628f47cabb22610fc3f2db8096
Author: John Davidge <email address hidden>
Date: Mon Oct 10 14:05:03 2016 +0100

    Fix IPv6 PD with pluggable IPAM

    A PD-sepcific check that was present in the non-pluggable IPAM
    backend was missing from the pluggable version, causing router
    interfaces to be created with a '::1' IP (equal to the gateway on
    PD subnets). The resulting command:

    ip netns exec <router_namespace> ip -6 addr add ::1/64

    would then fail. This patch restores the check to ensure that
    '::1' is not used for router interface ports, and introduces a
    test to protect against future regressions.

    Change-Id: I6a2e7cd60dd42d92b900a57bd8f3ffbc5909908e
    Closes-Bug: 1631693
    (cherry picked from commit 14ffea4444985eb041181d1bdef9af53ccb4d9f9)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.1.0

This issue was fixed in the openstack/neutron 9.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 10.0.0.0b1

This issue was fixed in the openstack/neutron 10.0.0.0b1 development milestone.

tags: added: neutron-proactive-backport-potential
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421990

tags: removed: newton-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/mitaka)

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/421990
Reason: Mitaka is CVE only.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers