mojo Build #23040 specs/full_stack/next_designate_ha xenial-pike: upgrade action failed

[Alex Kavanagh]

11:44:56 #############################################################################
11:44:56 Setup ceilometer
11:44:56 #############################################################################
11:44:56
11:44:56
11:44:56 2020-07-06 11:44:56 [INFO] Pulling secrets from /tmp/tmp.Q5zfnXUUye/LOCAL/mojo-openstack-specs/specs/full_stack/next_designate_ha/pike to /tmp/tmp.Q5zfnXUUye/mojo-openstack-specs/xenial/osci-mojo/local
11:44:56 2020-07-06 11:44:56 [WARNING] Automatic secrets phase ran but secrets directory /tmp/tmp.Q5zfnXUUye/LOCAL/mojo-openstack-specs/specs/full_stack/next_designate_ha/pike does not exist!
11:44:56 2020-07-06 11:44:56 [INFO] Running script ceilometer_setup.py
11:45:16 2020-07-06 11:45:16 [WARNING] No debug log matching debug-logs found. Using default.
11:45:19 2020-07-06 11:45:19 [ERROR] 2020-07-06 11:44:57 [INFO] Setting up ceilometer schema
11:45:19 Traceback (most recent call last):
11:45:19 File "/tmp/tmp.Q5zfnXUUye/mojo-openstack-specs/xenial/osci-mojo/spec/specs/full_stack/next_designate_ha/pike/ceilometer_setup.py", line 23, in <module>
11:45:19 sys.exit(main(sys.argv))
11:45:19 File "/tmp/tmp.Q5zfnXUUye/mojo-openstack-specs/xenial/osci-mojo/spec/specs/full_stack/next_designate_ha/pike/ceilometer_setup.py", line 19, in main
11:45:19 action.status))
11:45:19 Exception: ceilometer-upgrade action returned failed status

[Aurelien Lourot (aurelien-lourot)]

Happened again here: http://osci:8080/view/MojoMatrix/job/mojo_runner/23243/console

[Aurelien Lourot (aurelien-lourot)]

Happened again here: http://osci:8080/job/mojo_runner/23871/console

2020-09-30 05:02:17 DEBUG ceilometer-upgrade none
2020-09-30 05:02:17 DEBUG ceilometer-upgrade none
2020-09-30 05:02:18 DEBUG juju-log Running ceilometer-upgrade: ceilometer-upgrade --debug
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.541 11628 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.543 11628 DEBUG ceilometer.cmd.storage [-] Upgrading metering database upgrade /usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py:48
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.543 11628 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/dist-packages/ceilometer/storage/__init__.py:87
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.588 11628 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('172.17.107.28', 27017)]
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.656 11628 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py:57
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 CRITICAL ceilometer [-] Unhandled error: ClientException: {"message": "The server is currently unavailable. Please try again at a later time.<br /><br />\n\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer Traceback (most recent call last):
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer sys.exit(upgrade())
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py", line 59, in upgrade
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer gnocchi_client.upgrade_resource_types(conf)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/gnocchi_client.py", line 194, in upgrade_resource_types
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer gnocchi.resource_type.get(name=name)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/v1/resource_type.py", line 44, in get
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer headers={'Content-Type': "application/json"}).json()
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File ...

[Aurelien Lourot (aurelien-lourot)]

On gnocchi's side we see

2020-09-30 05:03:05.210 30713 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://172.17.107.232:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",)): ConnectFailure: Unable to establish connection to http://172.17.107.232:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

172.17.107.232 being keystone's VIP. I wonder if we should go for a tenacity/retry mechanism somewhere?

[Alex Kavanagh (ajkavanagh)]

So it looks like keystone was 'away' when these occurred. I can't tell from the logs; is this an "unstable-test" or causing gnocchi to die due to a charm bug in gnocchi?

[Aurelien Lourot (aurelien-lourot)]

Seen on xenial-queens as well: http://osci:8080/job/mojo_runner/24121/console