Gate failing on test_set_external_network_empty: ('Connection aborted.', gaierror(-2, 'Name or service not known'))

Bug #1363159 reported by Julie Pichon
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
Fix Released
Critical
Julie Pichon

Bug Description

All patches are failing with the following error. david-lyle and fungi debugged on #openstack-infra that this was due to requests being bumped to 2.4.0.

2014-08-29 15:16:30.485 | ======================================================================
2014-08-29 15:16:30.485 | ERROR: test_set_external_network_empty (openstack_dashboard.dashboards.admin.routers.tests.RouterTests)
2014-08-29 15:16:30.523 | ----------------------------------------------------------------------
2014-08-29 15:16:30.523 | Traceback (most recent call last):
2014-08-29 15:16:30.523 | File "/home/jenkins/workspace/gate-horizon-python27/openstack_dashboard/test/helpers.py", line 77, in instance_stub_out
2014-08-29 15:16:30.524 | return fn(self, *args, **kwargs)
2014-08-29 15:16:30.524 | File "/home/jenkins/workspace/gate-horizon-python27/openstack_dashboard/dashboards/admin/routers/tests.py", line 72, in test_set_external_network_empty
2014-08-29 15:16:30.524 | res = self.client.get(self.INDEX_URL)
2014-08-29 15:16:30.524 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/django/test/client.py", line 473, in get
2014-08-29 15:16:30.524 | response = super(Client, self).get(path, data=data, **extra)
2014-08-29 15:16:30.524 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/django/test/client.py", line 280, in get
2014-08-29 15:16:30.524 | return self.request(**r)
2014-08-29 15:16:30.524 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/django/test/client.py", line 444, in request
2014-08-29 15:16:30.524 | six.reraise(*exc_info)
2014-08-29 15:16:30.525 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
2014-08-29 15:16:30.525 | response = wrapped_callback(request, *callback_args, **callback_kwargs)
2014-08-29 15:16:30.525 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/decorators.py", line 36, in dec
2014-08-29 15:16:30.525 | return view_func(request, *args, **kwargs)
2014-08-29 15:16:30.525 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/decorators.py", line 84, in dec
2014-08-29 15:16:30.525 | return view_func(request, *args, **kwargs)
2014-08-29 15:16:30.525 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/decorators.py", line 52, in dec
2014-08-29 15:16:30.525 | return view_func(request, *args, **kwargs)
2014-08-29 15:16:30.525 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/decorators.py", line 36, in dec
2014-08-29 15:16:30.525 | return view_func(request, *args, **kwargs)
2014-08-29 15:16:30.526 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/decorators.py", line 84, in dec
2014-08-29 15:16:30.526 | return view_func(request, *args, **kwargs)
2014-08-29 15:16:30.526 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
2014-08-29 15:16:30.526 | return self.dispatch(request, *args, **kwargs)
2014-08-29 15:16:30.526 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
2014-08-29 15:16:30.526 | return handler(request, *args, **kwargs)
2014-08-29 15:16:30.526 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/tables/views.py", line 157, in get
2014-08-29 15:16:30.526 | handled = self.construct_tables()
2014-08-29 15:16:30.526 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/tables/views.py", line 148, in construct_tables
2014-08-29 15:16:30.526 | handled = self.handle_table(table)
2014-08-29 15:16:30.527 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/tables/views.py", line 120, in handle_table
2014-08-29 15:16:30.527 | data = self._get_data_dict()
2014-08-29 15:16:30.527 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/tables/views.py", line 185, in _get_data_dict
2014-08-29 15:16:30.527 | self._data = {self.table_class._meta.name: self.get_data()}
2014-08-29 15:16:30.527 | File "/home/jenkins/workspace/gate-horizon-python27/openstack_dashboard/dashboards/admin/routers/views.py", line 56, in get_data
2014-08-29 15:16:30.527 | routers = self._get_routers()
2014-08-29 15:16:30.527 | File "/home/jenkins/workspace/gate-horizon-python27/openstack_dashboard/dashboards/admin/routers/views.py", line 43, in _get_routers
2014-08-29 15:16:30.527 | tenant_dict = self._get_tenant_list()
2014-08-29 15:16:30.527 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/utils/memoized.py", line 90, in wrapped
2014-08-29 15:16:30.528 | value = cache[key] = func(*args, **kwargs)
2014-08-29 15:16:30.528 | File "/home/jenkins/workspace/gate-horizon-python27/openstack_dashboard/dashboards/admin/networks/views.py", line 50, in _get_tenant_list
2014-08-29 15:16:30.528 | exceptions.handle(self.request, msg)
2014-08-29 15:16:30.528 | File "/home/jenkins/workspace/gate-horizon-python27/horizon/exceptions.py", line 334, in handle
2014-08-29 15:16:30.528 | six.reraise(exc_type, exc_value, exc_traceback)
2014-08-29 15:16:30.528 | File "/home/jenkins/workspace/gate-horizon-python27/openstack_dashboard/dashboards/admin/networks/views.py", line 46, in _get_tenant_list
2014-08-29 15:16:30.528 | tenants, has_more = api.keystone.tenant_list(self.request)
2014-08-29 15:16:30.528 | File "/home/jenkins/workspace/gate-horizon-python27/openstack_dashboard/api/keystone.py", line 271, in tenant_list
2014-08-29 15:16:30.528 | tenants = manager.list(domain=domain, user=user)
2014-08-29 15:16:30.528 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2014-08-29 15:16:30.529 | return func(*args, **kwargs)
2014-08-29 15:16:30.529 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/v3/projects.py", line 79, in list
2014-08-29 15:16:30.529 | **kwargs)
2014-08-29 15:16:30.529 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/base.py", line 71, in func
2014-08-29 15:16:30.529 | return f(*args, **new_kwargs)
2014-08-29 15:16:30.529 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/base.py", line 352, in list
2014-08-29 15:16:30.529 | self.collection_key)
2014-08-29 15:16:30.529 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/base.py", line 112, in _list
2014-08-29 15:16:30.529 | resp, body = self.client.get(url, **kwargs)
2014-08-29 15:16:30.530 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 611, in get
2014-08-29 15:16:30.530 | return self._cs_request(url, 'GET', **kwargs)
2014-08-29 15:16:30.530 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 603, in _cs_request
2014-08-29 15:16:30.530 | return self.request(url, method, **kwargs)
2014-08-29 15:16:30.530 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 578, in request
2014-08-29 15:16:30.530 | resp = super(HTTPClient, self).request(url, method, **kwargs)
2014-08-29 15:16:30.530 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/baseclient.py", line 21, in request
2014-08-29 15:16:30.530 | return self.session.request(url, method, **kwargs)
2014-08-29 15:16:30.530 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2014-08-29 15:16:30.530 | return func(*args, **kwargs)
2014-08-29 15:16:30.531 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/session.py", line 324, in request
2014-08-29 15:16:30.531 | resp = self._send_request(url, method, redirect, log, **kwargs)
2014-08-29 15:16:30.531 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/session.py", line 350, in _send_request
2014-08-29 15:16:30.531 | resp = self.session.request(method, url, **kwargs)
2014-08-29 15:16:30.531 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/keystoneclient/session.py", line 65, in request
2014-08-29 15:16:30.531 | return requests.request(*args, **kwargs)
2014-08-29 15:16:30.531 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
2014-08-29 15:16:30.531 | return session.request(method=method, url=url, **kwargs)
2014-08-29 15:16:30.531 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/sessions.py", line 448, in request
2014-08-29 15:16:30.531 | resp = self.send(prep, **send_kwargs)
2014-08-29 15:16:30.532 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/sessions.py", line 554, in send
2014-08-29 15:16:30.532 | r = adapter.send(request, **kwargs)
2014-08-29 15:16:30.532 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/adapters.py", line 359, in send
2014-08-29 15:16:30.532 | timeout=timeout
2014-08-29 15:16:30.532 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
2014-08-29 15:16:30.532 | _pool=self, _stacktrace=stacktrace)
2014-08-29 15:16:30.532 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/util/retry.py", line 245, in increment
2014-08-29 15:16:30.532 | raise six.reraise(type(error), error, _stacktrace)
2014-08-29 15:16:30.532 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 516, in urlopen
2014-08-29 15:16:30.532 | body=body, headers=headers)
2014-08-29 15:16:30.533 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 308, in _make_request
2014-08-29 15:16:30.533 | conn.request(method, url, **httplib_request_kw)
2014-08-29 15:16:30.533 | File "/usr/lib/python2.7/httplib.py", line 973, in request
2014-08-29 15:16:30.533 | self._send_request(method, url, body, headers)
2014-08-29 15:16:30.533 | File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
2014-08-29 15:16:30.533 | self.endheaders(body)
2014-08-29 15:16:30.533 | File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
2014-08-29 15:16:30.534 | self._send_output(message_body)
2014-08-29 15:16:30.534 | File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
2014-08-29 15:16:30.534 | self.send(msg)
2014-08-29 15:16:30.534 | File "/usr/lib/python2.7/httplib.py", line 791, in send
2014-08-29 15:16:30.534 | self.connect()
2014-08-29 15:16:30.534 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 146, in connect
2014-08-29 15:16:30.534 | conn = self._new_conn()
2014-08-29 15:16:30.534 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 125, in _new_conn
2014-08-29 15:16:30.534 | (self.host, self.port), self.timeout, **extra_kw)
2014-08-29 15:16:30.534 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/requests/packages/urllib3/util/connection.py", line 64, in create_connection
2014-08-29 15:16:30.534 | for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
2014-08-29 15:16:30.534 | ProtocolError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))
2014-08-29 15:16:30.535 |
2014-08-29 15:16:30.535 | ======================================================================
2014-08-29 15:16:30.535 | ERROR: test_set_external_network_empty (openstack_dashboard.dashboards.admin.routers.tests.RouterTests)
2014-08-29 15:16:30.536 | ----------------------------------------------------------------------
2014-08-29 15:16:30.538 | Traceback (most recent call last):
2014-08-29 15:16:30.538 | File "/home/jenkins/workspace/gate-horizon-python27/openstack_dashboard/test/helpers.py", line 155, in tearDown
2014-08-29 15:16:30.538 | self.mox.VerifyAll()
2014-08-29 15:16:30.538 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/mox.py", line 286, in VerifyAll
2014-08-29 15:16:30.538 | mock_obj._Verify()
2014-08-29 15:16:30.538 | File "/home/jenkins/workspace/gate-horizon-python27/.tox/py27/local/lib/python2.7/site-packages/mox.py", line 506, in _Verify
2014-08-29 15:16:30.538 | raise ExpectedMethodCallsError(self._expected_calls_queue)
2014-08-29 15:16:30.538 | ExpectedMethodCallsError: Verify: Expected methods never called:
2014-08-29 15:16:30.539 | 0. network_list.__call__(<class 'django.http.request.HttpRequest'>, router:external=True) -> [<Network: {'status': 'ACTIVE', 'subnets': [<Subnet: {'allocation_pools': [{'start': '172.24.4.226.', 'end': '172.24.4.238'}], 'host_routes': [], 'cidr': '172.24.4.0/28', 'id': 'd6bdc71c-7566-4d32-b3ff-36441ce746e8', 'name': 'ext_subnet', 'enable_dhcp': False, 'network_id': '9b466b94-213a-4cda-badf-72c102a874da', 'tenant_id': '3', 'dns_nameservers': [], 'gateway_ip': '172.24.4.225', 'ip_version': 4, 'ipver_str': 'IPv4'}>], 'name': 'ext_net', 'admin_state_up': True, 'router__external': True, 'router:external': True, 'admin_state': 'UP', 'tenant_id': '3', 'shared': False, 'id': '9b466b94-213a-4cda-badf-72c102a874da'}>]

Tags: gate
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to horizon (master)

Fix proposed to branch: master
Review: https://review.openstack.org/117831

Changed in horizon:
status: New → In Progress
Revision history for this message
Akihiro Motoki (amotoki) wrote :

The detail observation is as follows

When api.keystone.tenant_list raises ConnectionRefused, the unit test admin.routers.tests.RouterTests.test_set_external_network_empty succeeds.
keystoneclient.openstack.common.apiclient.exceptions.ConnectionRefused is aliased to keystoneclient.exceptions.ConnectionError, and ConnectionError is one of openstack_dashboard.exceptions RECOVERABLE exceptions. Thus it is caught and the unit test didn't fail.

On the other hand, if ProtocolError (perhaps from eventlet.websocket) is raised, it is not a part of RECOVERABLE error, so the unit test fails.

I am not sure this kind of exception should be included in RECOVERABLE.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to horizon (master)

Reviewed: https://review.openstack.org/117831
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=f3da51632be4b5db7f411a640d906c7e7f6fbaa4
Submitter: Jenkins
Branch: master

commit f3da51632be4b5db7f411a640d906c7e7f6fbaa4
Author: Julie Pichon <email address hidden>
Date: Fri Aug 29 17:28:55 2014 +0100

    Add missing mock to test_set_external_network_empty

    A missing mock causes test failures when using requests 2.4.0 (I'm
    uncertain as to why). Add the mock, and change the "expected error
    messages" assertion down to 1 since only one failure should occur (I
    assume previously the failure on get_tenant_list caused a 2nd error
    message to pop up).

    Change-Id: Icb0110c5937c3d4e437cc708ce8dbb8d0edaae74
    Closes-Bug: #1363159

Changed in horizon:
status: In Progress → Fix Committed
Revision history for this message
Julie Pichon (jpichon) wrote :

Thanks for looking into the details Akihiro! It would make sense to me to add ProtocolError to the list of recoverable errors, since the resulting behaviour is similar to connection errors and we know how to handle these.

Revision history for this message
Julie Pichon (jpichon) wrote :

Or maybe it's not necessary, looks like this will be fixed in the next version of requests to send a ConnectionError again (cf. http://lists.openstack.org/pipermail/openstack-dev/2014-September/044793.html ), and there's a requirements.txt update to block 2.4.0 in the meantime.

Thierry Carrez (ttx)
Changed in horizon:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in horizon:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.