tripleo-ci-centos-8-standalone job failing with keystone resource creation async task

[Takashi Kajinami]

The error is observed in master several times.

https://b711d185688da3b864bc-5593d50c131879f6a486eeedbad80e3c.ssl.cf2.rackcdn.com/710643/1/check/tripleo-ci-centos-8-standalone/91eca12/logs/undercloud/home/zuul/standalone_deploy.log

2020-03-01 23:53:35 | TASK [tripleo_keystone_resources : Async creation of Keystone public endpoint] ***
2020-03-01 23:53:35 | Sunday 01 March 2020 23:53:35 +0000 (0:00:00.137) 0:26:55.974 **********
2020-03-01 23:53:35 | changed: [undercloud] => (item=None)
2020-03-01 23:53:36 | changed: [undercloud] => (item=None)
2020-03-01 23:53:36 | changed: [undercloud] => (item=None)
2020-03-01 23:53:36 | changed: [undercloud] => (item=None)
2020-03-01 23:53:37 | changed: [undercloud] => (item=None)
2020-03-01 23:53:37 | changed: [undercloud] => (item=None)
2020-03-01 23:53:37 | changed: [undercloud] => (item=None)
2020-03-01 23:53:37 | changed: [undercloud]
2020-03-01 23:53:37 |
2020-03-01 23:53:37 | TASK [tripleo_keystone_resources : Check Keystone public endpoint status] ******
2020-03-01 23:53:37 | Sunday 01 March 2020 23:53:37 +0000 (0:00:02.471) 0:26:58.446 **********
2020-03-01 23:53:38 | FAILED - RETRYING: Check Keystone public endpoint status (30 retries left).
2020-03-01 23:53:43 | FAILED - RETRYING: Check Keystone public endpoint status (29 retries left).
2020-03-01 23:53:48 | FAILED - RETRYING: Check Keystone public endpoint status (28 retries left).
2020-03-01 23:53:53 | FAILED - RETRYING: Check Keystone public endpoint status (27 retries left).
2020-03-01 23:53:59 | FAILED - RETRYING: Check Keystone public endpoint status (26 retries left).
2020-03-01 23:54:05 | FAILED - RETRYING: Check Keystone public endpoint status (25 retries left).
2020-03-01 23:54:11 | changed: [undercloud] => (item=None)
2020-03-01 23:54:11 | FAILED - RETRYING: Check Keystone public endpoint status (30 retries left).
2020-03-01 23:54:16 | changed: [undercloud] => (item=None)
2020-03-01 23:54:17 | changed: [undercloud] => (item=None)
2020-03-01 23:54:17 | changed: [undercloud] => (item=None)
2020-03-01 23:54:17 | changed: [undercloud] => (item=None)
2020-03-01 23:54:17 | FAILED - RETRYING: Check Keystone public endpoint status (30 retries left).
2020-03-01 23:54:22 | FAILED - RETRYING: Check Keystone public endpoint status (29 retries left).
2020-03-01 23:54:28 | FAILED - RETRYING: Check Keystone public endpoint status (28 retries left).
2020-03-01 23:54:33 | FAILED - RETRYING: Check Keystone public endpoint status (27 retries left).
2020-03-01 23:54:38 | FAILED - RETRYING: Check Keystone public endpoint status (26 retries left).
2020-03-01 23:54:43 | FAILED - RETRYING: Check Keystone public endpoint status (25 retries left).
2020-03-01 23:54:49 | FAILED - RETRYING: Check Keystone public endpoint status (24 retries left).
2020-03-01 23:54:54 | FAILED - RETRYING: Check Keystone public endpoint status (23 retries left).
2020-03-01 23:54:59 | FAILED - RETRYING: Check Keystone public endpoint status (22 retries left).
2020-03-01 23:55:04 | FAILED - RETRYING: Check Keystone public endpoint status (21 retries left).
2020-03-01 23:55:10 | FAILED - RETRYING: Check Keystone public endpoint status (20 retries left).
2020-03-01 23:55:15 | FAILED - RETRYING: Check Keystone public endpoint status (19 retries left).
2020-03-01 23:55:20 | FAILED - RETRYING: Check Keystone public endpoint status (18 retries left).
2020-03-01 23:55:25 | FAILED - RETRYING: Check Keystone public endpoint status (17 retries left).
2020-03-01 23:55:30 | FAILED - RETRYING: Check Keystone public endpoint status (16 retries left).
2020-03-01 23:55:36 | FAILED - RETRYING: Check Keystone public endpoint status (15 retries left).
2020-03-01 23:55:41 | FAILED - RETRYING: Check Keystone public endpoint status (14 retries left).
2020-03-01 23:55:46 | FAILED - RETRYING: Check Keystone public endpoint status (13 retries left).
2020-03-01 23:55:51 | FAILED - RETRYING: Check Keystone public endpoint status (12 retries left).
2020-03-01 23:55:57 | FAILED - RETRYING: Check Keystone public endpoint status (11 retries left).
2020-03-01 23:56:02 | FAILED - RETRYING: Check Keystone public endpoint status (10 retries left).
2020-03-01 23:56:07 | FAILED - RETRYING: Check Keystone public endpoint status (9 retries left).
2020-03-01 23:56:12 | FAILED - RETRYING: Check Keystone public endpoint status (8 retries left).
2020-03-01 23:56:17 | FAILED - RETRYING: Check Keystone public endpoint status (7 retries left).
2020-03-01 23:56:23 | FAILED - RETRYING: Check Keystone public endpoint status (6 retries left).
2020-03-01 23:56:28 | FAILED - RETRYING: Check Keystone public endpoint status (5 retries left).
2020-03-01 23:56:33 | FAILED - RETRYING: Check Keystone public endpoint status (4 retries left).
2020-03-01 23:56:38 | FAILED - RETRYING: Check Keystone public endpoint status (3 retries left).
2020-03-01 23:56:44 | FAILED - RETRYING: Check Keystone public endpoint status (2 retries left).
2020-03-01 23:56:49 | FAILED - RETRYING: Check Keystone public endpoint status (1 retries left).
2020-03-01 23:56:54 | failed: [undercloud] (item=None) => changed=false
2020-03-01 23:56:54 | attempts: 30
2020-03-01 23:56:54 | censored: 'the output has been hidden due to the fact that ''no_log: true'' was specified for this result'
2020-03-01 23:56:54 | FAILED - RETRYING: Check Keystone public endpoint status (30 retries left).
2020-03-01 23:57:00 | FAILED - RETRYING: Check Keystone public endpoint status (29 retries left).
2020-03-01 23:57:05 | FAILED - RETRYING: Check Keystone public endpoint status (28 retries left).
2020-03-01 23:57:10 | FAILED - RETRYING: Check Keystone public endpoint status (27 retries left).
2020-03-01 23:57:15 | FAILED - RETRYING: Check Keystone public endpoint status (26 retries left).
2020-03-01 23:57:20 | FAILED - RETRYING: Check Keystone public endpoint status (25 retries left).
2020-03-01 23:57:26 | FAILED - RETRYING: Check Keystone public endpoint status (24 retries left).
2020-03-01 23:57:31 | FAILED - RETRYING: Check Keystone public endpoint status (23 retries left).
2020-03-01 23:57:36 | FAILED - RETRYING: Check Keystone public endpoint status (22 retries left).
2020-03-01 23:57:41 | FAILED - RETRYING: Check Keystone public endpoint status (21 retries left).
2020-03-01 23:57:47 | FAILED - RETRYING: Check Keystone public endpoint status (20 retries left).
2020-03-01 23:57:52 | FAILED - RETRYING: Check Keystone public endpoint status (19 retries left).
2020-03-01 23:57:57 | FAILED - RETRYING: Check Keystone public endpoint status (18 retries left).
2020-03-01 23:58:02 | FAILED - RETRYING: Check Keystone public endpoint status (17 retries left).
2020-03-01 23:58:07 | FAILED - RETRYING: Check Keystone public endpoint status (16 retries left).
2020-03-01 23:58:13 | FAILED - RETRYING: Check Keystone public endpoint status (15 retries left).
2020-03-01 23:58:18 | FAILED - RETRYING: Check Keystone public endpoint status (14 retries left).
2020-03-01 23:58:23 | FAILED - RETRYING: Check Keystone public endpoint status (13 retries left).
2020-03-01 23:58:28 | FAILED - RETRYING: Check Keystone public endpoint status (12 retries left).
2020-03-01 23:58:34 | FAILED - RETRYING: Check Keystone public endpoint status (11 retries left).
2020-03-01 23:58:39 | FAILED - RETRYING: Check Keystone public endpoint status (10 retries left).
2020-03-01 23:58:44 | FAILED - RETRYING: Check Keystone public endpoint status (9 retries left).
2020-03-01 23:58:49 | FAILED - RETRYING: Check Keystone public endpoint status (8 retries left).
2020-03-01 23:58:54 | FAILED - RETRYING: Check Keystone public endpoint status (7 retries left).
2020-03-01 23:59:00 | FAILED - RETRYING: Check Keystone public endpoint status (6 retries left).
2020-03-01 23:59:05 | FAILED - RETRYING: Check Keystone public endpoint status (5 retries left).
2020-03-01 23:59:10 | FAILED - RETRYING: Check Keystone public endpoint status (4 retries left).
2020-03-01 23:59:16 | FAILED - RETRYING: Check Keystone public endpoint status (3 retries left).
2020-03-01 23:59:21 | FAILED - RETRYING: Check Keystone public endpoint status (2 retries left).
2020-03-01 23:59:26 | FAILED - RETRYING: Check Keystone public endpoint status (1 retries left).
2020-03-01 23:59:31 | failed: [undercloud] (item=None) => changed=false
2020-03-01 23:59:31 | attempts: 30
2020-03-01 23:59:31 | censored: 'the output has been hidden due to the fact that ''no_log: true'' was specified for this result'
2020-03-01 23:59:31 | fatal: [undercloud]: FAILED! => changed=true
2020-03-01 23:59:31 | censored: 'the output has been hidden due to the fact that ''no_log: true'' was specified for this result'
2020-03-01 23:59:31 |

It seems that tripleo-ansible timed out because it took too long time to create keystone endpoint resource.

https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_keystone_resources/tasks/endpoints.yml#L42

We have another issue in tripleo-ci-centos-8-standalone in database initialization, and it seems that both failures are caused by slow database in gate.

https://bugs.launchpad.net/tripleo/+bug/1865473

[wes hayutin (weshayutin)]

This is I think an inconsistent issue..
I'll look it up but I don't see this happening often enough for a code change.
http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone

If you feel otherwise please respond.

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]