Description of problem:
Randomly deployment fails in overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0 with this error (the error is visible in "openstack stack failures list"):
Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]
If you re-run the deployment command it will resume from where it failed and finish successfully. It appears that Keystone is not running when it's needed by ceilometer. ceilo-upgrade tries to authenticate with keystone to create resource types in gnocchi. But keystone throws a 503:
10.35.191.20 - - [02/Jul/2017:14:48:06 +0000] "GET
/v1/resource_type/instance_disk HTTP/1.1" 503 170 "-"
"ceilometer-upgrade keystoneauth1/2.18.0 python-requests/2.11.1
CPython/2.7.5"
2017-07-02 14:48:11.800 116449 ERROR keystonemiddleware.auth_token [-]
Bad response code while validating token: 503
2017-07-02 14:48:11.800 116449 WARNING keystonemiddleware.auth_token
[-] Identity response: <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
and ceilometer-upgrade fails with:
2017-07-02 14:48:11.803 123807 CRITICAL ceilometer [-]
ClientException: {"message": "The server is currently unavailable.
Please try again at a later time.<br /><br />\n\n\n", "code": "503
Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
How reproducible:
randomly
Steps to Reproduce:
1. Deploy the default plan. I deployed a topology of 3 controllers + 1 compute + 3 ceph with this command: openstack overcloud deploy --templates --ntp-server clock.redhat.com -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e counters.yaml
2. If you hit this error you re-run the command to resume it from where it failed, and it passes the 2nd time.
Expected results:
Deployment should pass the first time. You shouldn't hit errors due to keystone not being up and running on time.
Pradeep, I think we have the same issue here
https:/