CI: ovb OOOQ container job fails

[Sagi (Sergey) Shnaidman]

It's difficult to understand why it fails from logs:

http://logs.openstack.org/45/470745/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/07199a0/logs/oooq/undercloud/home/jenkins/failed_deployment_list.log.txt.gz

The problem started approx. from 02 Jun 2017, 16:30 UTC

[Bogdan Dobrelya (bogdando)]

The root cause seems is:
Jun 4 13:56:59 ./logs/oooq/overcloud-controller-0/var/log/messages.txt.gz localhost os-collect-config: [2017-06-04 13:56:48,634] (heat-config) [ERROR] Error running ['docker', 'exec', 'keystone', u'/usr/bin/bootstrap_host_exec', u'keystone', u'keystone-manage', u'bootstrap', u'--bootstrap-password', u'RGyBmv76JE9RGVRDBF73updur']. [1]

[Bogdan Dobrelya (bogdando)]

Yeah, here it is http://logs.openstack.org/45/470745/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/07199a0/logs/oooq/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz#_2017-06-04_13_56_48_548

[Numan Siddique (numansiddique)]

I am seeing the same problem.

Jun 5 08:05:05 localhost os-collect-config: [2017-06-05 08:04:51,620] (heat-config) [DEBUG] Running container: keystone_bootstrap
Jun 5 08:05:05 localhost os-collect-config: [2017-06-05 08:04:51,620] (heat-config) [DEBUG] docker ps -a --filter label=container_name=keystone --filter label=config_id=71368667-669a-404a-91f3-62616b6710f2 --format {{.Names}}
Jun 5 08:05:05 localhost os-collect-config: [2017-06-05 08:04:51,636] (heat-config) [DEBUG] keystone
Jun 5 08:05:05 localhost os-collect-config: [2017-06-05 08:04:51,636] (heat-config) [DEBUG]
Jun 5 08:05:05 localhost os-collect-config: [2017-06-05 08:04:51,636] (heat-config) [DEBUG] docker exec keystone /usr/bin/bootstrap_host_exec keystone keystone-manage --debug bootstrap --bootstrap-password aW6JfdJbQPycPmVc4ecceAKCC
Jun 5 08:05:05 localhost os-collect-config: [2017-06-05 08:04:54,088] (heat-config) [DEBUG]
Jun 5 08:05:05 localhost os-collect-config: [2017-06-05 08:04:54,088] (heat-config) [DEBUG]
Jun 5 08:05:05 localhost os-collect-config: [2017-06-05 08:04:54,088] (heat-config) [ERROR] Error running ['docker', 'exec', 'keystone', u'/usr/bin/bootstrap_host_exec', u'keystone', u'keystone-manage', u'bootstrap', u'--bootstrap-password', u'aW6JfdJbQPycPmVc4ecceAKCC']. [1]

[Jiří Stránský (jistr)]

Just a data point: this is not specific to OVB, the multinode CI fails for the same reason.

[Jiří Stránský (jistr)]

Here's a clue: keystone.log contains a critical error about missing bcrypt library:

http://logs.openstack.org/47/470147/4/check/gate-tripleo-ci-centos-7-containers-multinode-nv/edf5f4e/logs/subnode-2/var/log/containers/keystone/keystone.log.txt.gz#_2017-06-05_13_11_39_870

MissingBackendError: bcrypt: no backends available -- recommend you install one (e.g. 'pip install bcrypt')

While an earlier successful job doesn't have that error:

http://logs.openstack.org/70/470370/1/check/gate-tripleo-ci-centos-7-containers-multinode-nv/05ae68e/logs/subnode-2/var/log/containers/keystone/keystone.log.txt.gz

[Jiří Stránský (jistr)]

This is related to

https://review.rdoproject.org/r/6959
https://github.com/rdo-packages/keystone-distgit/commit/dfe598c305004870d66bbecc19237233e5a497b5

We need to rebuild the keystone container image.

[Jiří Stránský (jistr)]

I've rebuilt the keystone image, got past step 3 in deployment (still running but i don't have time to watch it till completion on my rather slow env). Pushed the image to dockerhub:

https://hub.docker.com/r/tripleoupstream/centos-binary-keystone/

[John Trowbridge (trown)]

This was fixed by the keystone container rebuild.