ci: OVB based jobs are not collecting logs from OC nodes

[Matt Young]

As of {unknown_date} we do not have logs being collected and/or persisted for OC nodes from OVB jobs.

Example:

- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/1107bd8

We do have logs for the UC, but not OC nodes. In this specific case as well OC deploy was successful:

- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/1107bd8/console.txt.gz#_2018-03-14_11_41_28_692

[Matt Young (halcyondude)]

Looks like OC nodes are unreachable at the point in time when collect logs is invoked. root cause unknown (so far)

---

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/1107bd8/console.txt.gz#_2018-03-14_10_01_33_327

+(./toci_quickstart.sh:66): QUICKSTART_COLLECTLOGS_CMD='

/home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/.quickstart/bin/ansible-playbook
/home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/.quickstart/playbooks/collect-logs.yml
-vv
--extra-vars @/home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/.quickstart/config/release/tripleo-ci/promotion-testing-hash-queens.yml
--extra-vars @/home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/.quickstart/config/general_config/featureset020.yml
--extra-vars @/opt/stack/new/tripleo-ci/toci-quickstart/config/testenv/ovb.yml --extra-vars @/opt/stack/new/tripleo-ci/toci-quickstart/config/testenv/ovb-rdocloud.yml
--extra-vars local_working_dir=/home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/.quickstart
--extra-vars virthost=undercloud
--inventory /home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/.quickstart/hosts
--extra-vars tripleo_root=/opt/stack/new
--extra-vars working_dir=/home/jenkins
--extra-vars validation_args='\''--validation-errors-nonfatal'\''
--extra-vars @/opt/stack/new/tripleo-ci/toci-quickstart/config/collect-logs.yml
--extra-vars artcl_collect_dir=/home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/logs
--tags all
--skip-tags teardown-all '

---

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/1107bd8/quickstart_collect_logs.log

fatal: [overcloud-controller-foo-0]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '127.0.0.2' (ECDSA) to the list of known hosts.
channel 0: open failed: connect failed: No route to host
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host
", "unreachable": true}
fatal: [overcloud-novacompute-bar-0]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '127.0.0.2' (ECDSA) to the list of known hosts.
channel 0: open failed: connect failed: No route to host
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host
", "unreachable": true}

[Matt Young (halcyondude)]

Last change to where collect logs is invoked landed 8 days ago

- https://github.com/openstack-infra/tripleo-ci/commit/765389d1782fe7e05eac106e8a27554f876bf1f9

This was merged 5-march:

- https://github.com/openstack-infra/tripleo-ci/commit/dc4a553f3a55492bf09d1b801fe699a0a44af97b
- https://review.rdoproject.org/r/#/c/12442

there's no ref'd bug / details other than commit message

---

Run logs collection after job finished in RDO CI

we run logs collection in post playbook in upstream infra CI,
do the same in third-party CI whith OVB and multinode jobs.
Collect logs fter job is finished.

---

Looking at the job history (in this case for queens)

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens

17b22fe/ 2018-03-08 03:50
5234057/ 2018-03-07 19:08
88c8758/ 2018-03-07 13:02
b949520/ 2018-03-07 04:24

# b949520 is the first missing OC logs
# f24ab30 is the last job with OC logs

f24ab30/ 2018-03-06 19:08
3a7dcbe/ 2018-03-06 12:56
e724251/ 2018-03-06 07:38
0d4c7d0/ 2018-03-05 22:43
b86511d/ 2018-03-05 13:08

[Matt Young (halcyondude)]

cross-ref'ing with master periodic job logs...the same point in time is when things started failing:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/?C=M;O=D

6b8323e 2018-03-08 05:04
8263cc3 2018-03-07 19:55
02f4bc0 2018-03-07 13:48
b9b0389 2018-03-07 05:11

# b9b0389 is the first missing OC logs
# 4c5a355 is the last job with OC logs

4c5a355 2018-03-06 20:07
a049ff7 2018-03-06 13:11
70df410 2018-03-06 07:41
43182cd 2018-03-05 22:47
bf3f34e 2018-03-05 12:58

[Matt Young (halcyondude)]

This could be fallout from recent tripleo-ci squad work:

- https://trello.com/c/2e1WPhLn/581-add-logs-collection-after-job-in-rdo-jobs

However is just a working theory, investigating

[Matt Young (halcyondude)]

discussion in IRC about this from #oooq

http://paste.openstack.org/show/700998

[yatin (yatinkarel)]

Proposed revert of work that created the issue:-
https://review.openstack.org/#/c/553420
https://review.rdoproject.org/r/#/c/12932

[John Trowbridge (trown)]

https://review.openstack.org/#/c/553420 does not seem to have helped:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/6ab55f8/quickstart_collect_logs.log

[yatin (yatinkarel)]

<<< https://review.openstack.org/#/c/553420 does not seem to have helped:

This helped, as for other featureset's i can see overcloud logs are collected.

<<< https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/6ab55f8/quickstart_collect_logs.log

Overcloud is deleted before collecting logs:- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/6ab55f8/console.txt.gz#_2018-03-19_14_45_28_527

Because step_delete_overcloud: true for featureset002 https://github.com/openstack/tripleo-quickstart/blob/7fa967f57663216d67f6fbc062e8c99ed5b3256e/config/general_config/featureset002.yml#L13-L14

[Alex Schultz (alex-schultz)]

logs are being captured, closing this out. feel free to reopen if it happens again.

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/9a4960c/