OVB - overcloud-ssl fail with KeyError - breaks legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master and featureset035

[Harald Jensås]

This change: https://review.openstack.org/602171 changed the TLS environment files used by quickstart.

We now get the following error in OVB jobs:
fatal: [undercloud]: FAILED! => {"changed": false, "module_stderr": "Warning: Permanently added '127.0.0.2' (ECDSA) to the list of known hosts.\r\nTraceback (most recent call last):\n File \"/tmp/ansible_hWpsIF/ansible_module_tls_tht.py\", line 156, in <module>\n main()\n File \"/tmp/ansible_hWpsIF/ansible_module_tls_tht.py\", line 144, in main\n module.params[\"tht_release\"])\n File \"/tmp/ansible_hWpsIF/ansible_module_tls_tht.py\", line 88, in create_enable_file\n \"{}/puppet/extraconfig/tls/tls-cert-inject.yaml\".format(source_dir)\nKeyError: 'resource_registry'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}

https://logs.rdoproject.org/71/608471/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/c73d7da/job-output.txt.gz#_2018-10-07_23_38_46_924968

(added promotion blocker as this is affecting ovb jobs featureset001 and 35)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/608588

[OpenStack Infra (hudson-openstack)]

Change abandoned by Harald Jensås (<email address hidden>) on branch: master
Review: https://review.openstack.org/608588
Reason: There is another fix proposed here: https://review.openstack.org/608589

[Marios Andreou (marios-b)]

The legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master (and also featureset035-master) are failing for on overcloud-ssl task " : fetch template from single remote host" trace like:

  2018-10-08 01:14:16.878209 | tripleo-ovb-centos-7 | TASK [overcloud-ssl : Generate SSL certificates] *******************************
  2018-10-08 01:14:16.891635 | tripleo-ovb-centos-7 | Monday 08 October 2018 01:14:16 +0000 (0:00:03.136) 0:58:16.958 ********
  2018-10-08 01:14:18.642427 | tripleo-ovb-centos-7 | changed: [undercloud]
  2018-10-08 01:14:18.674930 | tripleo-ovb-centos-7 |
  2018-10-08 01:14:18.675058 | tripleo-ovb-centos-7 | TASK [overcloud-ssl : fetch template from single remote host] ******************
  2018-10-08 01:14:18.689770 | tripleo-ovb-centos-7 | Monday 08 October 2018 01:14:18 +0000 (0:00:01.798) 0:58:18.756 ********
  2018-10-08 01:14:19.875586 | tripleo-ovb-centos-7 | fatal: [undercloud]: FAILED! => {"changed": false, "module_stderr": "Warning: Permanently added '127.0.0.2' (ECDSA) to the list of known hosts.\r\nTraceback (most recent call last):\n File \"/tmp/ansible_sM7Y3s/ansible_module_tls_tht.py\", line 156, in <module>\n main()\n File \"/tmp/ansible_sM7Y3s/ansible_module_tls_tht.py\", line 144, in main\n module.params[\"tht_release\"])\n File \"/tmp/ansible_sM7Y3s/ansible_module_tls_tht.py\", line 88, in create_enable_file\n \"{}/puppet/extraconfig/tls/tls-cert-inject.yaml\".format(source_dir)\nKeyError: 'resource_registry'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
  2018-10-08 01:14:19.893709 | tripleo-ovb-centos-7 |
  2018-10-08 01:14:19.893835 | tripleo-ovb-centos-7 | PLAY RECAP *********************************************************************
  2018-10-08 01:14:19.893942 | tripleo-ovb-centos-7 | 127.0.0.2 : ok=0 changed=0 unreachable=0 failed=0
  2018-10-08 01:14:19.894044 | tripleo-ovb-centos-7 | localhost : ok=19 changed=7 unreachable=0 failed=0
  2018-10-08 01:14:19.894177 | tripleo-ovb-centos-7 | undercloud : ok=165 changed=94 unreachable=0 failed=1
  2018-10-08 01:14:19.894196 | tripleo-ovb-centos-7 |

Many examples like [1-4] below.

[1] http://logs.rdoproject.org/98/604298/27/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/d9bb00b/job-output.txt.gz
[2] http://logs.rdoproject.org/80/582180/15/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/d3b9046/job-output.txt.gz
[3] http://logs.rdoproject.org/81/582181/15/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/1883da8/job-output.txt.gz
[4] http://logs.rdoproject.org/98/604298/27/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/dc3b08c/job-output.txt.gz

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.openstack.org/608589
Reason: job has failed in the gate, killing then restarting :)

Failed due to infra hiccup on fetching mirrored dlrn repos
http://logs.openstack.org/89/608589/5/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5d46fce/job-output.txt.gz#_2018-10-08_14_00_22_167020

[wes hayutin (weshayutin)]

AFAICT...

https://review.openstack.org/#/c/602171/ should not have been approved and merged. I know it's difficult to go through results and costly however it's still important. :)

https://logs.rdoproject.org/71/602171/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/36a57f6/job-output.txt.gz#_2018-10-07_12_00_16_354414

[Marios Andreou (marios-b)]

as it didn't report here, the fix by bandini is at

https://review.openstack.org/#/c/608589/ Fix TLS resource_registry key error I53851edbb8bb562dc4194fb99d6ade259227d2f9 t-q-e

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.openstack.org/608589
Reason: We need this patch, https://review.openstack.org/#/c/609289/

restarting the job

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/608589
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=00ea9d56233757279c3b9419a2541d13433a089f
Submitter: Zuul
Branch: master

commit 00ea9d56233757279c3b9419a2541d13433a089f
Author: Michele Baldessari <email address hidden>
Date: Mon Oct 8 08:55:42 2018 +0200

    Fix TLS resource_registry key error

    Via I1bfdb6d064f3b10b269dedafd36ca367139fe1df we moved to using
    environments/ssl/enable-tls.yaml. The problem is that
    the code in roles/overcloud-ssl/library/tls_tht.py assumes that
    the parsed yaml file already has the 'resource_registry' key in the
    dictionary.

    That was true with environments/enable-tls.yaml but is not
    true any longer for environments/ssl/enable-tls.yaml.

    Since Iaf7386207e5bd8b336759f51e4405fe15114123a in rocky
    NodeTLSData is not used anymore, so let's just skip the whole assignment
    starting with rocky.

    Closes-Bug: #1796626

    Depends-On: Ibee6ba188585f80f0f7d136c81146096cb4432c2
    Change-Id: I53851edbb8bb562dc4194fb99d6ade259227d2f9

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.