Traces seen in l3 agent with prefix delegation enabled subnet

Bug #1639033 reported by Baodong (Robert) Li
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Undecided
Baodong (Robert) Li

Bug Description

2016-11-03 11:22:41.381 [00;32mDEBUG neutron.agent.linux.utils [[00;36m-[00;32m] [01;35m[00;32mRunning command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-9f1e22fe-eac6-4032-87f2-315f028076c8', 'ip', '-6', 'addr', 'add', '::8/64', 'scope', 'global', 'dev', 'qr-352a870a-7d'][00m [00;33mfrom (pid=28627) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100[00m
2016-11-03 11:22:41.444 [01;31mERROR neutron.agent.linux.utils [[00;36m-[01;31m] [01;35m[01;31mExit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
[00m
2016-11-03 11:22:41.445 [01;31mERROR neutron.agent.l3.router_info [[00;36m-[01;31m] [01;35m[01;31mExit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
[00m
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00mTraceback (most recent call last):
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/common/utils.py", line 216, in call
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m return func(*args, **kwargs)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1064, in process
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m self._process_internal_ports(agent.pd)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 555, in _process_internal_ports
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m updated_cidrs)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 394, in _internal_network_updated
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m self.ns_name)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 183, in add_ipv6_addr
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m device.addr.add(str(net), scope)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 580, in add
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m self._as_root([net.version], tuple(args))
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 367, in _as_root
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m use_root_namespace=use_root_namespace)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 96, in _as_root
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m log_fail_as_error=self.log_fail_as_error)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 105, in _execute
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m log_fail_as_error=log_fail_as_error)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 139, in execute
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m raise RuntimeError(msg)
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00mRuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m
[01;31m2016-11-03 11:22:41.445 TRACE neutron.agent.l3.router_info [01;35m[00m
2016-11-03 11:22:41.448 [01;31mERROR neutron.agent.l3.agent [[00;36m-[01;31m] [01;35m[01;31mFailed to process compatible router: 9f1e22fe-eac6-4032-87f2-315f028076c8[00m
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00mTraceback (most recent call last):
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 508, in _process_router_update
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m self._process_router_if_compatible(router)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 445, in _process_router_if_compatible
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m self._process_updated_router(router)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 460, in _process_updated_router
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m ri.process(self)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/common/utils.py", line 219, in call
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m self.logger(e)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m self.force_reraise()
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m six.reraise(self.type_, self.value, self.tb)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/common/utils.py", line 216, in call
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m return func(*args, **kwargs)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1064, in process
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m self._process_internal_ports(agent.pd)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 555, in _process_internal_ports
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m updated_cidrs)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 394, in _internal_network_updated
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m self.ns_name)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 183, in add_ipv6_addr
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m device.addr.add(str(net), scope)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 580, in add
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m self._as_root([net.version], tuple(args))
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 367, in _as_root
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m use_root_namespace=use_root_namespace)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 96, in _as_root
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m log_fail_as_error=self.log_fail_as_error)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 105, in _execute
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m log_fail_as_error=log_fail_as_error)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 139, in execute
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m raise RuntimeError(msg)
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00mRuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m
[01;31m2016-11-03 11:22:41.448 TRACE neutron.agent.l3.agent [01;35m[00m
2016-11-03 11:22:41.450 [00;32mDEBUG neutron.agent.l3.agent [[00;36m-[00;32m] [01;35m[00;32mPayload:
{
     "enable_snat": true,
     "gw_port": {
          "allowed_address_pairs": [],
          "extra_dhcp_opts": [],
          "updated_at": "2016-11-03T15:22:34Z",
          "device_owner": "network:router_gateway",
          "revision_number": 8,
          "port_security_enabled": false,
          "binding:profile": {},
          "binding:vnic_type": "normal",
          "fixed_ips": [
               {
                    "subnet_id": "e8cbec96-2ea4-43d5-a717-93b03231d393",
                    "prefixlen": 24,
                    "ip_address": "172.24.4.5"
               },
               {
                    "subnet_id": "68fc0073-619a-43ca-be62-569842366e48",
                    "prefixlen": 64,
                    "ip_address": "2001:db8::7"
               }
          ],
          "id": "8ffa9d13-f66d-4cfe-b16d-f71c315c53a9",
          "security_groups": [],
          "binding:vif_details": {
               "port_filter": true,
               "ovs_hybrid_plug": true
          },
          "address_scopes": {
               "4": null,
               "6": null
          },
          "binding:vif_type": "ovs",
          "mac_address": "fa:16:3e:eb:cb:05",
          "project_id": "",
          "status": "ACTIVE",
          "subnets": [
               {
                    "dns_nameservers": [],
                    "ipv6_ra_mode": null,
                    "gateway_ip": "2001:db8::2",
                    "cidr": "2001:db8::/64",
                    "id": "68fc0073-619a-43ca-be62-569842366e48",
                    "subnetpool_id": null
               },
               {
                    "dns_nameservers": [],
                    "ipv6_ra_mode": null,
                    "gateway_ip": "172.24.4.1",
                    "cidr": "172.24.4.0/24",
                    "id": "e8cbec96-2ea4-43d5-a717-93b03231d393",
                    "subnetpool_id": null
               }
          ],
          "binding:host_id": "ip6-devstack-43",
          "description": "",
          "device_id": "9f1e22fe-eac6-4032-87f2-315f028076c8",
          "name": "",
          "admin_state_up": true,
          "network_id": "8da945b9-8ac2-4d1b-840f-094dbd794500",
          "tenant_id": "",
          "created_at": "2016-11-03T15:22:30Z",
          "mtu": 1500,
          "extra_subnets": []
     },
     "updated_at": "2016-11-03T15:22:31Z",
     "revision_number": 6,
     "id": "9f1e22fe-eac6-4032-87f2-315f028076c8",
     "_interfaces": [
          {
               "allowed_address_pairs": [],
               "extra_dhcp_opts": [],
               "updated_at": "2016-11-03T15:22:30Z",
               "device_owner": "network:router_interface",
               "revision_number": 8,
               "port_security_enabled": false,
               "binding:profile": {},
               "binding:vnic_type": "normal",
               "fixed_ips": [
                    {
                         "subnet_id": "3b227ecc-c0ac-4b9e-bc29-9458e0f35633",
                         "prefixlen": 64,
                         "ip_address": "::8"
                    }
               ],
               "id": "352a870a-7d20-4964-b739-7febbefa8687",
               "security_groups": [],
               "binding:vif_details": {
                    "port_filter": true,
                    "ovs_hybrid_plug": true
               },
               "address_scopes": {
                    "4": null,
                    "6": null
               },
               "binding:vif_type": "ovs",
               "mac_address": "fa:16:3e:3d:7b:0d",
               "project_id": "59485ad25c25488cae7711528c3a3914",
               "status": "ACTIVE",
               "subnets": [
                    {
                         "dns_nameservers": [],
                         "ipv6_ra_mode": "slaac",
                         "gateway_ip": "2001:420:2c50:2024::1",
                         "cidr": "2001:420:2c50:2024::/64",
                         "id": "3b227ecc-c0ac-4b9e-bc29-9458e0f35633",
                         "subnetpool_id": "prefix_delegation"
                    }
               ],
               "binding:host_id": "ip6-devstack-43",
               "description": "",
               "device_id": "9f1e22fe-eac6-4032-87f2-315f028076c8",
               "name": "",
               "admin_state_up": true,
               "network_id": "243184e5-72d4-43eb-9295-b10ae8b44c7a",
               "tenant_id": "59485ad25c25488cae7711528c3a3914",
               "created_at": "2016-11-03T15:22:26Z",
               "mtu": 1500,
               "extra_subnets": []
          }
     ],
     "availability_zone_hints": [],
     "availability_zones": [
          "nova"
     ],
     "distributed": false,
     "project_id": "59485ad25c25488cae7711528c3a3914",
     "status": "ACTIVE",
     "ha_vr_id": 0,
     "description": "",
     "ha": false,
     "gw_port_host": "ip6-devstack-43",
     "external_gateway_info": {
          "network_id": "8da945b9-8ac2-4d1b-840f-094dbd794500",
          "enable_snat": true,
          "external_fixed_ips": [
               {
                    "subnet_id": "e8cbec96-2ea4-43d5-a717-93b03231d393",
                    "ip_address": "172.24.4.5"
               },
               {
                    "subnet_id": "68fc0073-619a-43ca-be62-569842366e48",
                    "ip_address": "2001:db8::7"
               }
          ]
     },
     "name": "ha-router",
     "gw_port_id": "8ffa9d13-f66d-4cfe-b16d-f71c315c53a9",
     "admin_state_up": true,
     "tenant_id": "59485ad25c25488cae7711528c3a3914",
     "created_at": "2016-11-03T15:22:22Z",
     "flavor_id": null,
     "routes": []
}[00m [00;33mfrom (pid=28627) log_verbose_exc /opt/stack/neutron/neutron/agent/l3/agent.py:76[00m

Noticed that in the above payload, "gateway_ip": "2001:420:2c50:2024::1", but fixed ip doesn't get updated on the gateway port.

The traceback caused resync, and subsequent router update has the correct fixed up.

Changed in neutron:
assignee: nobody → Baodong (Robert) Li (baoli)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/399662

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Miguel Lavalle (<email address hidden>) on branch: master
Review: https://review.openstack.org/399662

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers