tripleo ocata ceilometer-db-upgrade fails, as gnocchi cant reach swift

[wes hayutin]

"deploy_stdout": "\u001b[mNotice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend\u001b[0m\n\u001b[mNotice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.\u001b[0m\n\u001b[mNotice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 34.40 seconds

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-ocata-oooq/3f84978/logs/undercloud/home/jenkins/failed_deployments.log.txt.gz

[Pradeep Kilambi (pkilambi)]

Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]

in gnocchi wsgi logs:

[Mon Sep 25 08:22:20.311402 2017] [:error] [pid 59207] [remote 172.17.0.12:112] mod_wsgi (pid=59207): Exception occurred processing WSGI script '/var/www/cgi-bin/gnocchi/app'.
[Mon Sep 25 08:22:20.311441 2017] [:error] [pid 59207] [remote 172.17.0.12:112] Traceback (most recent call last):
[Mon Sep 25 08:22:20.311475 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/var/www/cgi-bin/gnocchi/app", line 22, in <module>
[Mon Sep 25 08:22:20.311524 2017] [:error] [pid 59207] [remote 172.17.0.12:112] application = app.build_wsgi_app()
[Mon Sep 25 08:22:20.311541 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/usr/lib/python2.7/site-packages/gnocchi/rest/app.py", line 136, in build_wsgi_app
[Mon Sep 25 08:22:20.311571 2017] [:error] [pid 59207] [remote 172.17.0.12:112] return load_app(service.prepare_service())
[Mon Sep 25 08:22:20.311586 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/usr/lib/python2.7/site-packages/gnocchi/rest/app.py", line 91, in load_app
[Mon Sep 25 08:22:20.311611 2017] [:error] [pid 59207] [remote 172.17.0.12:112] storage = gnocchi_storage.get_driver(conf)
[Mon Sep 25 08:22:20.311626 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 144, in get_driver
[Mon Sep 25 08:22:20.312047 2017] [:error] [pid 59207] [remote 172.17.0.12:112] conf.incoming)
[Mon Sep 25 08:22:20.312068 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/usr/lib/python2.7/site-packages/gnocchi/storage/incoming/swift.py", line 36, in __init__
[Mon Sep 25 08:22:20.312331 2017] [:error] [pid 59207] [remote 172.17.0.12:112] self.swift.put_container(self.MEASURE_PREFIX)
[Mon Sep 25 08:22:20.312356 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1755, in put_container
[Mon Sep 25 08:22:20.313262 2017] [:error] [pid 59207] [remote 172.17.0.12:112] query_string=query_string)
[Mon Sep 25 08:22:20.313290 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1673, in _retry
[Mon Sep 25 08:22:20.313323 2017] [:error] [pid 59207] [remote 172.17.0.12:112] service_token=self.service_token, **kwargs)
[Mon Sep 25 08:22:20.313339 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1014, in put_container
[Mon Sep 25 08:22:20.313363 2017] [:error] [pid 59207] [remote 172.17.0.12:112] conn.request(method, path, '', headers)
[Mon Sep 25 08:22:20.313378 2017] [:error] [pid 59207] [remote 172.17.0.12:112] File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 436, in request
[Mon Sep 25 08:22:20.313402 2017] [:error] [pid 59207] [remote 172.17.0.12:112] files=files, **self.requests_args)...

[Pradeep Kilambi (pkilambi)]

Swift cant be reached and from what i can tell its not running. Which in turn causes gnocchi to fail to start properly and ceilometer upgrade fails. The issue seems to be with swift. But we need better logs to confirm.

[wes hayutin (weshayutin)]

related bug: https://bugs.launchpad.net/tripleo/+bug/1719421

[Christian Schwede (cschwede)]

Indeed, looks like the Swift services on the controller nodes are all in state failed.

Not sure why this happens, there are no Swift logfiles and /var/log/messages is empty?

[Alfredo Moralejo (amoralej)]

I think root cause for this is the same as https://bugs.launchpad.net/tripleo/+bug/1719421

[wes hayutin (weshayutin)]

We should have the logs in place in tonights run, now that 1719421 is closed

[Alfredo Moralejo (amoralej)]

Perodic jobs have passed, swift problem has been fixed by https://review.openstack.org/#/c/507495/