Gateway clear generates a TRACE - AttributeError in get_int_device_name in DVR routers

[Swaminathan Vasudevan]

A recent change in the agent code have introduced this problem.

When a Gateway is cleared from the router, even though there are no existing floating IPs, the "external_gateway_removed" function in "agent.py" is calling "process_floatingips".
That may be the reason for this failure.

Stderr: RTNETLINK answers: No such process

2015-02-11 23:12:15.307 2809 ERROR neutron.agent.l3.dvr [-] DVR: removed snat failed
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr Traceback (most recent call last):
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr File "/opt/stack/new/neutron/neutron/agent/l3/dvr.py", line 197, in _snat_redirect_remove
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr ns_ipd.route.delete_gateway(table=snat_idx)
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 415, in delete_gateway
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr self._as_root(*args)
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 253, in _as_root
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr kwargs.get('use_root_namespace', False))
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 83, in _as_root
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr log_fail_as_error=self.log_fail_as_error)
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 95, in _execute
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr log_fail_as_error=log_fail_as_error)
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 83, in execute
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr raise RuntimeError(m)
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr RuntimeError:
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-1cfe7654-a669-4f73-a21d-d5110d7c0297', 'ip', 'route', 'del', 'default', 'dev', 'qr-467e8832-93', 'table', '547711270']
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr Exit code: 2
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr Stdout:
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr Stderr: RTNETLINK answers: No such process
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr
2015-02-11 23:12:15.307 2809 TRACE neutron.agent.l3.dvr
2015-02-11 23:12:18.846 2809 ERROR neutron.agent.l3.agent [-] 'NoneType' object has no attribute 'get_int_device_name'
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/common/utils.py", line 342, in call
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 602, in process_router
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent self._process_external(ri)
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 565, in _process_external
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent self._process_external_gateway(ri)
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 503, in _process_external_gateway
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent self.external_gateway_removed(ri, ri.ex_gw_port, interface_name)
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 905, in external_gateway_removed
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent ri, ex_gw_port)
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 694, in _get_external_device_interface_name
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent fip_int = ri.fip_ns.get_int_device_name(ri.router_id)
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent AttributeError: 'NoneType' object has no attribute 'get_int_device_name'
2015-02-11 23:12:18.846 2809 TRACE neutron.agent.l3.agent
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    func(*args, **kwargs)
  File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 1137, in _process_router_update
    self._router_removed(update.id)
  File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 409, in _router_removed
    self.process_router(ri)
  File "/opt/stack/new/neutron/neutron/common/utils.py", line 345, in call
    self.logger(e)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/new/neutron/neutron/common/utils.py", line 342, in call
    return func(*args, **kwargs)
  File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 602, in process_router
    self._process_external(ri)
  File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 565, in _process_external
    self._process_external_gateway(ri)
  File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 503, in _process_external_gateway
    self.external_gateway_removed(ri, ri.ex_gw_port, interface_name)
  File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 905, in external_gateway_removed
    ri, ex_gw_port)
  File "/opt/stack/new/neutron/neutron/agent/l3/agent.py", line 694, in _get_external_device_interface_name
    fip_int = ri.fip_ns.get_int_device_name(ri.router_id)
AttributeError: 'NoneType' object has no attribute 'get_int_device_name'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/156357

[Carl Baldwin (carl-baldwin)]

Could we use something like paste bin for the full stack trace? It is very hard to read a trace the way that launchpad adds line breaks.

[Rajeev Grover (rajeev-grover)]

Here's one:
http://paste.openstack.org/show/177074/

[Rajeev Grover (rajeev-grover)]

@amuller

Looks like traces started appearing with this patch https://review.openstack.org/#/c/152342/9/neutron/agent/l3/dvr_router.py .
Doesn’t look like it is the culprit but it resumed deletion of the fip namespace causing the failure to show up.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fef04147bd68bc101c74c93d4e94f5f5b4643d6f
Submitter: Jenkins
Branch: master

commit fef04147bd68bc101c74c93d4e94f5f5b4643d6f
Author: rajeev <email address hidden>
Date: Fri Feb 13 16:58:53 2015 -0500

    fix for _get_external_device_interface_name trace

    On removal of external gateway from DVR the code path
    external_gateway_removed(...) was trying to access the
    agent gateway port interface even when no fip namespace exists.
    This change checks for existence of namespace before
    accessing the agent gateway port interface through
    _get_external_device_interface_name(...) or looking for
    floating ips that may have been configured on the
    port.

    Change-Id: Idcf28ff93f16f1d692fe7acb678fb90aabe5af5e
    Closes-Bug: #1421497