gate fail (THT): Unable to establish connection to https://192.168.24.2:13808/v1/AUTH_9bf3500126a448e8a7e691c26cf33190: ('Connection aborted.', BadStatusLine(\"''\",))

[Matt Young]

Failure just spotted in the gates for the following patch:

- https://review.openstack.org/#/c/555007 Deploy Designate in scenario003

It's failing on the tripleo-ci-centos-7-undercloud-containers job, during undercloud-sanity-check.sh:

- http://logs.openstack.org/07/555007/7/gate/tripleo-ci-centos-7-undercloud-containers/c597c83/job-output.txt.gz#_2018-05-09_15_08_54_332754

- http://logs.openstack.org/07/555007/7/gate/tripleo-ci-centos-7-undercloud-containers/c597c83/logs/undercloud/home/zuul/undercloud_sanity_check.log.txt.gz#_2018-05-09_15_09_07

/home/zuul/undercloud-sanity-check.sh: line 51: /home/zuul/stackrc: No such file or directory

[Alex Schultz (alex-schultz)]

The undercloud-install failed, so i'm not sure why it didn't get picked up in CI until the sanity check task.

It failed on create_swift_temp_url_key during the install.

http://logs.openstack.org/07/555007/7/gate/tripleo-ci-centos-7-undercloud-containers/c597c83/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-05-09_15_08_44

2018-05-09 15:08:44 | "Error running ['docker', 'run', '--name', 'create_swift_temp_url_key', '--label', 'config_id=tripleo_step4', '--label', 'container_name=create_swift_temp_url_key', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 70, \"command\": \"/usr/bin/bootstrap_host_exec ironic_conductor /create_swift_temp_url_key.sh\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/puppet-generated/ironic/etc/ironic:/etc/ironic:ro\", \"/var/lib/docker-config-scripts/create_swift_temp_url_key.sh:/create_swift_temp_url_key.sh:ro\"], \"image\": \"docker.io/tripleomaster/centos-binary-ironic-conductor:81c23c047e8e0fc03b54164921f49fdb4103202c_b333f915\", \"detach\": false, \"net\": \"host\"}', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/puppet-generated/ironic/etc/ironic:/etc/ironic:ro', '--volume=/var/lib/docker-config-scripts/create_swift_temp_url_key.sh:/create_swift_temp_url_key.sh:ro', 'docker.io/tripleomaster/centos-binary-ironic-conductor:81c23c047e8e0fc03b54164921f49fdb4103202c_b333f915', '/usr/bin/bootstrap_host_exec', 'ironic_conductor', '/create_swift_temp_url_key.sh']. [1]",
2018-05-09 15:08:44 | "stdout: Check if a temporary URL key already exists",
2018-05-09 15:08:44 | "Creating a new temporary URL for project service",
2018-05-09 15:08:44 | "stderr: Unable to establish connection to https://192.168.24.2:13808/v1/AUTH_9bf3500126a448e8a7e691c26cf33190: ('Connection aborted.', BadStatusLine(\"''\",))",

[Matt Young (halcyondude)]

We should have failed and exited the ansible run here (I think) when the following task was executed:

http://logs.openstack.org/07/555007/7/gate/tripleo-ci-centos-7-undercloud-containers/c597c83/logs/ara_oooq/file/3ff0587b-326e-404b-bfa4-c3c6f11e574a/#line-6

Which launched the UC install script:

http://logs.openstack.org/07/555007/7/gate/tripleo-ci-centos-7-undercloud-containers/c597c83/logs/undercloud/home/zuul/undercloud-install.sh.txt.gz

Like this:

http://logs.openstack.org/07/555007/7/gate/tripleo-ci-centos-7-undercloud-containers/c597c83/logs/ara_oooq/result/e919f051-efd8-43aa-b0d0-5a16465b2d10

Investigating...

[Matt Young (halcyondude)]

Perhaps we should split this into a CI issue and the actual UC install failure.

Regarding the "why didn't the UC install task fail" question, upon first look the non-zero status of the UC install script should cause undercloud-install.sh to fail (set -e), and the pipefail in the ansible task's shell block should fail the task and cause playbook execution to stop at UC install (vs. continuing on to fail the sanity check later).

I'll open a different issue to track that, I still don't fully understand why we didn't fail as expected at UC install

[Alex Schultz (alex-schultz)]

I found why the task didn't fail. we lost tripleoclient returning non-zero for failures in the containerized undercloud deployment in some of the recent refactoring. I'm fixing that. This bug can be fore the swift isue.

[Matt Young (halcyondude)]

(tripleo-ci triage)

rechecked the patches (and it's dependency) where this was found initially. waiting on results.

[Matt Young (halcyondude)]

https://review.openstack.org/#/c/555007
https://review.openstack.org/#/c/564049 (parent)

It depends on on :

are both thru check jobs and into gates

[Alex Schultz (alex-schultz)]

This error showed up because we lost the error code from the install. https://review.openstack.org/#/c/567384/ closing this bug out.