Analyzed latest log stash entry. Found the following traceback:
2015-09-01 12:27:53.906 | 2015-09-01 12:27:53.883 | Traceback (most recent call last): 2015-09-01 12:27:53.916 | 2015-09-01 12:27:53.894 | File "neutron/tests/functional/agent/test_l3_agent.py", line 930, in test_access_to_metadata_proxy 2015-09-01 12:27:53.927 | 2015-09-01 12:27:53.904 | router = self.manage_router(self.agent, router_info) 2015-09-01 12:27:53.937 | 2015-09-01 12:27:53.915 | File "neutron/tests/functional/agent/test_l3_agent.py", line 129, in manage_router 2015-09-01 12:27:53.948 | 2015-09-01 12:27:53.925 | agent._process_added_router(router) 2015-09-01 12:27:53.959 | 2015-09-01 12:27:53.936 | File "neutron/agent/l3/agent.py", line 439, in _process_added_router 2015-09-01 12:27:53.969 | 2015-09-01 12:27:53.946 | self._router_added(router['id'], router) 2015-09-01 12:27:53.979 | 2015-09-01 12:27:53.956 | File "neutron/agent/l3/agent.py", line 328, in _router_added 2015-09-01 12:27:53.988 | 2015-09-01 12:27:53.966 | ri.initialize(self.process_monitor) 2015-09-01 12:27:53.998 | 2015-09-01 12:27:53.975 | File "neutron/agent/l3/router_info.py", line 85, in initialize 2015-09-01 12:27:54.008 | 2015-09-01 12:27:53.985 | self.router_namespace.create() 2015-09-01 12:27:54.018 | 2015-09-01 12:27:53.995 | File "neutron/agent/l3/namespaces.py", line 71, in create 2015-09-01 12:27:54.032 | 2015-09-01 12:27:54.009 | ip_wrapper = self.ip_wrapper_root.ensure_namespace(self.name) 2015-09-01 12:27:54.043 | 2015-09-01 12:27:54.020 | File "neutron/agent/linux/ip_lib.py", line 161, in ensure_namespace 2015-09-01 12:27:54.056 | 2015-09-01 12:27:54.031 | if not self.netns.exists(name): 2015-09-01 12:27:54.066 | 2015-09-01 12:27:54.044 | File "neutron/agent/linux/ip_lib.py", line 737, in exists 2015-09-01 12:27:54.076 | 2015-09-01 12:27:54.054 | run_as_root=cfg.CONF.AGENT.use_helper_for_ns_read) 2015-09-01 12:27:54.086 | 2015-09-01 12:27:54.063 | File "neutron/agent/linux/ip_lib.py", line 88, in _execute 2015-09-01 12:27:54.097 | 2015-09-01 12:27:54.074 | log_fail_as_error=log_fail_as_error) 2015-09-01 12:27:54.107 | 2015-09-01 12:27:54.085 | File "neutron/agent/linux/utils.py", line 156, in execute 2015-09-01 12:27:54.118 | 2015-09-01 12:27:54.095 | raise RuntimeError(m) 2015-09-01 12:27:54.128 | 2015-09-01 12:27:54.105 | RuntimeError: 2015-09-01 12:27:54.152 | 2015-09-01 12:27:54.116 | Command: ['ip', '-o', 'netns', 'list'] 2015-09-01 12:27:54.152 | 2015-09-01 12:27:54.128 | Exit code: 126 2015-09-01 12:27:54.162 | 2015-09-01 12:27:54.139 | Stdin: 2015-09-01 12:27:54.173 | 2015-09-01 12:27:54.149 | Stdout: {"data":[[65534]],"headings":["ofport"]} 2015-09-01 12:27:54.187 | 2015-09-01 12:27:54.161 | 2015-09-01 12:27:54.199 | 2015-09-01 12:27:54.176 | Stderr:
Logstash indicates that this happened with change 196149. Investigating in Gerrit, found https://review.openstack.org/#/c/196149, which creates on purpose the failure shown in the traceback above. Investigating with the patchset author
Analyzed latest log stash entry. Found the following traceback:
2015-09-01 12:27:53.906 | 2015-09-01 12:27:53.883 | Traceback (most recent call last): tests/functiona l/agent/ test_l3_ agent.py" , line 930, in test_access_ to_metadata_ proxy router( self.agent, router_info) tests/functiona l/agent/ test_l3_ agent.py" , line 129, in manage_router process_ added_router( router) agent/l3/ agent.py" , line 439, in _process_ added_router added(router[ 'id'], router) agent/l3/ agent.py" , line 328, in _router_added self.process_ monitor) agent/l3/ router_ info.py" , line 85, in initialize namespace. create( ) agent/l3/ namespaces. py", line 71, in create wrapper_ root.ensure_ namespace( self.name) agent/linux/ ip_lib. py", line 161, in ensure_namespace exists( name): agent/linux/ ip_lib. py", line 737, in exists root=cfg. CONF.AGENT. use_helper_ for_ns_ read) agent/linux/ ip_lib. py", line 88, in _execute as_error= log_fail_ as_error) agent/linux/ utils.py" , line 156, in execute :[[65534] ],"headings" :["ofport" ]}
2015-09-01 12:27:53.916 | 2015-09-01 12:27:53.894 | File "neutron/
2015-09-01 12:27:53.927 | 2015-09-01 12:27:53.904 | router = self.manage_
2015-09-01 12:27:53.937 | 2015-09-01 12:27:53.915 | File "neutron/
2015-09-01 12:27:53.948 | 2015-09-01 12:27:53.925 | agent._
2015-09-01 12:27:53.959 | 2015-09-01 12:27:53.936 | File "neutron/
2015-09-01 12:27:53.969 | 2015-09-01 12:27:53.946 | self._router_
2015-09-01 12:27:53.979 | 2015-09-01 12:27:53.956 | File "neutron/
2015-09-01 12:27:53.988 | 2015-09-01 12:27:53.966 | ri.initialize(
2015-09-01 12:27:53.998 | 2015-09-01 12:27:53.975 | File "neutron/
2015-09-01 12:27:54.008 | 2015-09-01 12:27:53.985 | self.router_
2015-09-01 12:27:54.018 | 2015-09-01 12:27:53.995 | File "neutron/
2015-09-01 12:27:54.032 | 2015-09-01 12:27:54.009 | ip_wrapper = self.ip_
2015-09-01 12:27:54.043 | 2015-09-01 12:27:54.020 | File "neutron/
2015-09-01 12:27:54.056 | 2015-09-01 12:27:54.031 | if not self.netns.
2015-09-01 12:27:54.066 | 2015-09-01 12:27:54.044 | File "neutron/
2015-09-01 12:27:54.076 | 2015-09-01 12:27:54.054 | run_as_
2015-09-01 12:27:54.086 | 2015-09-01 12:27:54.063 | File "neutron/
2015-09-01 12:27:54.097 | 2015-09-01 12:27:54.074 | log_fail_
2015-09-01 12:27:54.107 | 2015-09-01 12:27:54.085 | File "neutron/
2015-09-01 12:27:54.118 | 2015-09-01 12:27:54.095 | raise RuntimeError(m)
2015-09-01 12:27:54.128 | 2015-09-01 12:27:54.105 | RuntimeError:
2015-09-01 12:27:54.152 | 2015-09-01 12:27:54.116 | Command: ['ip', '-o', 'netns', 'list']
2015-09-01 12:27:54.152 | 2015-09-01 12:27:54.128 | Exit code: 126
2015-09-01 12:27:54.162 | 2015-09-01 12:27:54.139 | Stdin:
2015-09-01 12:27:54.173 | 2015-09-01 12:27:54.149 | Stdout: {"data"
2015-09-01 12:27:54.187 | 2015-09-01 12:27:54.161 |
2015-09-01 12:27:54.199 | 2015-09-01 12:27:54.176 | Stderr:
Logstash indicates that this happened with change 196149. Investigating in Gerrit, found https:/ /review. openstack. org/#/c/ 196149, which creates on purpose the failure shown in the traceback above. Investigating with the patchset author