Iteration for metering fails because metering-agent tried to operate chain in namespace of deleted router

[Kengo Hobo]

Metering-agent cannot recognize namespace for router is deleted.
Thus, metering-agent sometimes fails to operate chain for metering with following trace
because namespace for router had already deleted, especially when synchronizing information with neutron-server.

I assume that we should simply catch the error and continue the iteration.

How to reproduce
=========================
1. neutron meter-label-create test
2. neutron meter-label-rule-create <UUID of meter-label in procedure 1.>
3. neutron router-create test
4. neutron router-gateway-set <UUID of router in procedure 3.>
5. Check that iptables chain for the router is generated.
6. neutron router-delete test
7. wait a minute.(until next _sync_routers_task is executed.)

trace in metering-agent.log
=========================
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent [req-4798393c-417b-4af6-bb16-15c7b8cb4b17 - - - - -] Driver neutron.services.meterin
g.drivers.iptables.iptables_driver.IptablesMeteringDriver:update_routers runtime error
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent Traceback (most recent call last):
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/services/metering/agents/metering_agent.py", line
 166, in _invoke_driver
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent return getattr(self.metering_driver, func_name)(context, meterings)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/usr/local/lib/python2.7/dist-packages/oslo_log/helpers.py", line 48, in wra
pper
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent return method(*args, **kwargs)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
r.py", line 109, in update_routers
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent self._process_disassociate_metering_label(rm.router)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
r.py", line 257, in _process_disassociate_metering_label
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent del rm.metering_labels[label_id]
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/services/metering/drivers/iptables/iptables_drive
r.py", line 58, in __exit__
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent self.im.apply()
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 437, in apply
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent return self._apply()
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 445, in _apply
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent first = self._apply_synchronized()
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 480, in _apply_synchronized
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent save_output = self.execute(args, run_as_root=True)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 138, in execute
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent raise RuntimeError(msg)
2016-09-16 00:20:35.118 16578 ERROR neutron.services.metering.agents.metering_agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-15e41fb9-44f7-4233-8ff9-12f74f73ade2": No such file or directory
==========================

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/371407

[Hirofumi Ichihara (ichihara-hirofumi)]

Have you already found why it happens? Could you elaborate a way of reproduce if you haven't? I'd try to find it.

[Kengo Hobo (hobo-kengo)]

Hi,
I have added how to reproduce the bug in description of this bug report.

[Hirofumi Ichihara (ichihara-hirofumi)]

Thank you. I can reproduce.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/371407
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Bug closed due to lack of activity, please feel free to reopen if needed.