gnocchi container fails with "No server is available to handle this request"

Bug #1658818 reported by Sean M. Collins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Undecided
Unassigned

Bug Description

http://logs.openstack.org/83/424283/1/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/f27b6d4/console.html#_2017-01-23_19_43_08_752953

2017-01-23 19:42:15.786624 | FAILED - RETRYING: TASK: os_gnocchi : Ensure Archive Policies (5 retries left).
2017-01-23 19:42:26.372443 | FAILED - RETRYING: TASK: os_gnocchi : Ensure Archive Policies (4 retries left).
2017-01-23 19:42:36.953738 | FAILED - RETRYING: TASK: os_gnocchi : Ensure Archive Policies (3 retries left).
2017-01-23 19:42:47.600447 | FAILED - RETRYING: TASK: os_gnocchi : Ensure Archive Policies (2 retries left).
2017-01-23 19:42:58.191433 | FAILED - RETRYING: TASK: os_gnocchi : Ensure Archive Policies (1 retries left).
2017-01-23 19:43:08.752225 | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: (HTTP 503)
2017-01-23 19:43:08.752953 | failed: [aio1_gnocchi_container-a96d72dd] (item={u'definition': [{u'points': 86400, u'granularity': u'1s'}, {u'points': 43200, u'granularity': u'1m'}, {u'points': 8760, u'granularity': u'1h'}], u'name': u'high'}) => {"attempts": 5, "failed": true, "item": {"definition": [{"granularity": "1s", "points": 86400}, {"granularity": "1m", "points": 43200}, {"granularity": "1h", "points": 8760}], "name": "high"}, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_j9EKvM/ansible_module_gnocchi.py\", line 311, in <module>\n main()\n File \"/tmp/ansible_j9EKvM/ansible_module_gnocchi.py\", line 308, in main\n mg.route()\n File \"/tmp/ansible_j9EKvM/ansible_module_gnocchi.py\", line 210, in route\n resource = action(vars=action_command['vars'\n File \"/tmp/ansible_j9EKvM/ansible_module_gnocchi.py\", line 241, in ensure_archive_policy\n ap = policy_mgr.get(name=variables.get('name', None))\n File \"/usr/local/lib/python2.7/dist-packages/gnocchiclient/v1/archive_policy.py\", line 35, in get\n return self._get(self.url + name).json()\n File \"/usr/local/lib/python2.7/dist-packages/gnocchiclient/v1/base.py\", line 37, in _get\n return self.client.api.get(*args, **kwargs)\n File \"/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py\", line 216, in get\n return self.request(url, 'GET', **kwargs)\n File \"/usr/local/lib/python2.7/dist-packages/gnocchiclient/client.py\", line 38, in request\n raise exceptions.from_response(resp, method)\ngnocchiclient.exceptions.ClientException: <html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n (HTTP 503)\n", "module_stdout": "", "msg": "MODULE FAILURE"}
2017-01-23 19:43:08.752974 |
2017-01-23 19:43:08.753000 | NO MORE HOSTS LEFT *************************************************************
2017-01-23 19:43:08.753008 |
2017-01-23 19:43:08.753031 | NO MORE HOSTS LEFT *************************************************************
2017-01-23 19:43:08.753641 |
2017-01-23 19:43:08.753684 | PLAY RECAP *********************************************************************
2017-01-23 19:43:08.753762 | aio1_gnocchi_container-a96d72dd : ok=77 changed=1 unreachable=0 failed=1
2017-01-23 19:43:08.753860 |
2017-01-23 19:43:08.765107 | TASK: os_gnocchi : Ensure Archive Policies ---------------------------- 161.51s
2017-01-23 19:43:08.765150 | TASK: os_gnocchi : Wait for Gnocchi API to come up --------------------- 10.38s
2017-01-23 19:43:08.765178 | TASK: os_gnocchi : Install required pip packages ------------------------ 3.49s
2017-01-23 19:43:08.765203 | TASK: os_gnocchi : Perform a Gnocchi DB sync ---------------------------- 2.88s
2017-01-23 19:43:08.765227 | TASK: pip_install : Install PIP ----------------------------------------- 2.16s
2017-01-23 19:43:08.771681 | TASK: os_gnocchi : Retrieve and config_template upstream files ---------- 1.23s
2017-01-23 19:43:08.771723 | TASK: galera_client : Find old sources ---------------------------------- 1.18s
2017-01-23 19:43:08.771756 | TASK: os_gnocchi : Ensure Gnocchi service ------------------------------- 1.03s
2017-01-23 19:43:08.771780 | TASK: pip_install : Install apt packages -------------------------------- 1.03s
2017-01-23 19:43:08.771804 | TASK: pip_install : Install pip packages -------------------------------- 1.00s
2017-01-23 19:43:08.954772 |
2017-01-23 19:43:08.954860 | Hit maximum number of retries, giving up...

Revision history for this message
Sean M. Collins (scollins) wrote :

This appears to be affecting the gate for openstack-ansible currently. It seems to be non-deterministic since we have some patches merging still.

summary: - gnocci container fails with "No server is available to handle this
+ gnocchi container fails with "No server is available to handle this
request"
description: updated
Revision history for this message
Sean M. Collins (scollins) wrote :
Revision history for this message
Sean M. Collins (scollins) wrote :

Error log inside the container:

http://logs.openstack.org/83/424283/1/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-xenial/f27b6d4/logs/openstack/aio1_gnocchi_container-a96d72dd/gnocchi/gnocchi-apache-error.log.txt.gz#_2017-01-23_19_42_20_038385

2017-01-23 19:42:20.037615 2017-01-23 19:42:20.037 2580 INFO gnocchi.rest.app [-] WSGI config used: /etc/gnocchi/api-paste.ini
2017-01-23 19:42:20.038166 mod_wsgi (pid=2580): Target WSGI script '/var/www/cgi-bin/gnocchi/gnocchi-api' cannot be loaded as Python module.
2017-01-23 19:42:20.038175 mod_wsgi (pid=2580): Exception occurred processing WSGI script '/var/www/cgi-bin/gnocchi/gnocchi-api'.
2017-01-23 19:42:20.038188 Traceback (most recent call last):
2017-01-23 19:42:20.038199 File "/var/www/cgi-bin/gnocchi/gnocchi-api", line 22, in <module>
2017-01-23 19:42:20.038223 application = app.build_wsgi_app()
2017-01-23 19:42:20.038230 File "/openstack/venvs/gnocchi-master/lib/python2.7/site-packages/gnocchi/rest/app.py", line 130, in build_wsgi_app
2017-01-23 19:42:20.038249 return load_app(service.prepare_service())
2017-01-23 19:42:20.038254 File "/openstack/venvs/gnocchi-master/lib/python2.7/site-packages/gnocchi/rest/app.py", line 106, in load_app
2017-01-23 19:42:20.038265 global_conf={'configkey': configkey})
2017-01-23 19:42:20.038270 File "/openstack/venvs/gnocchi-master/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2017-01-23 19:42:20.038283 return loadobj(APP, uri, name=name, **kw)
2017-01-23 19:42:20.038288 File "/openstack/venvs/gnocchi-master/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
2017-01-23 19:42:20.038298 global_conf=global_conf)
2017-01-23 19:42:20.038303 File "/openstack/venvs/gnocchi-master/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
2017-01-23 19:42:20.038329 global_conf=global_conf)
2017-01-23 19:42:20.038335 File "/openstack/venvs/gnocchi-master/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
2017-01-23 19:42:20.038344 return loader.get_context(object_type, name, global_conf)
2017-01-23 19:42:20.038349 File "/openstack/venvs/gnocchi-master/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 408, in get_context
2017-01-23 19:42:20.038358 object_type, name=name)
2017-01-23 19:42:20.038363 File "/openstack/venvs/gnocchi-master/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 587, in find_config_section
2017-01-23 19:42:20.038372 self.filename))
2017-01-23 19:42:20.038385 LookupError: No section 'main' (prefixed by 'app' or 'application' or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config /etc/gnocchi/api-paste.ini

Then it loops, starts up again and then loops over and over

Changed in openstack-ansible:
status: New → Confirmed
Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

I believe that Gnocchi is being deployed properly as of today.

Please feel free to raise a new bug or bump this one if issue is still present on any of currently supported releases.

Changed in openstack-ansible:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.