CI: Most/all legacy-tripleo jobs failing on gate

[Alex Schultz]

Zuul v3 cutover and the jobs are currently failing on gate_hook: not found.

See:

http://logs.openstack.org/89/507889/2/gate/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-puppet/03a4588/job-output.txt.gz#_2017-09-28_19_07_03_200440

http://logs.openstack.org/89/507889/2/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a4877fe/job-output.txt.gz#_2017-09-28_19_10_49_835208

[David Moreau Simard (dmsimard)]

Multi-node jobs were not migrated with multinode nodesets, review here: https://review.openstack.org/#/c/508292/

[David Moreau Simard (dmsimard)]

OVB jobs were not migrated with the tripleo-centos-7 nodesets, review here: https://review.openstack.org/508301

[wes hayutin (weshayutin)]

Both reviews have merged.

[Michele Baldessari (michele)]

I am still seeing this via recheck done a few mins ago http://logs.openstack.org/66/497766/3/check/legacy-tripleo-ci-centos-7-undercloud-oooq/318a3b6/job-output.txt.gz#_2017-09-29_07_38_21_027901

[David Moreau Simard (dmsimard)]

We're past the sudoers error as well as the nodeset errors now.
New error is:

2017-09-29 14:01:35.457 | ++ /home/zuul/workspace/devstack-gate/functions.sh:setup_workspace:L414: basename openstack/ceilometer
2017-09-29 14:01:35.459 | + /home/zuul/workspace/devstack-gate/functions.sh:setup_workspace:L414: rsync -a '/home/zuul/src/*/openstack/ceilometer/' ceilometer
2017-09-29 14:01:35.461 | rsync: change_dir "/home/zuul/src/*/openstack/ceilometer" failed: No such file or directory (2)

http://logs.openstack.org/89/507889/2/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/603ef1f/logs/devstack-gate-setup-workspace-new.txt

[David Moreau Simard (dmsimard)]

The following stack of patches should get us to an improved state:
- https://review.openstack.org/#/c/508548/
- https://review.openstack.org/#/c/508510/
- https://review.openstack.org/#/c/508511/

[David Moreau Simard (dmsimard)]

It looks like the next issue is the following:

/opt/stack/new/tripleo-ci/toci_gate_test.sh: line 154: NODEPOOL_PROVIDER: unbound variable
http://logs.openstack.org/89/507889/2/check/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/3f7277c/ara/result/c4c9c0cc-fb0b-45e4-b185-0126dd8a6360/

[wes hayutin (weshayutin)]

I opened this [1] bug over the weekend, is it a dupe of this one 1720220?

[1] https://bugs.launchpad.net/tripleo/+bug/1720556

[David Moreau Simard (dmsimard)]

Fix for nodepool provider: https://review.openstack.org/#/c/508714/

[David Moreau Simard (dmsimard)]

I suspect we might need patches like these [1] for tripleo, let's wait and see what 508714 gives.

[1]: https://review.openstack.org/#/q/topic:fix-multinode

[Sagi (Sergey) Shnaidman (sshnaidm)]

just FYI, I'm working on this too in https://review.openstack.org/#/c/508660/

See also bug for all OVB jobs: https://bugs.launchpad.net/tripleo/+bug/1720721
"CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud"

[Sagi (Sergey) Shnaidman (sshnaidm)]

Currently the main problem is hardlinks failure, the patch that should solve it[1] doesn't work for all jobs consistently.
My patch also fixes from "jenkins" user to "zuul", patch for cirros image, moving "sub_nodes" to "sub_nodes_private" other things: https://review.openstack.org/#/c/508660/
It depends also on few patches in quickstart and extras.

[1] https://review.openstack.org/#/c/508772/

[Alex Schultz (alex-schultz)]

Upstream switched back to zuul v2. Removing alert tag, adding zuulv3.

[wes hayutin (weshayutin)]

zuul v2 jobs seem stable, can we close this?

[Alex Schultz (alex-schultz)]

Closing this for now as we're on v2 and v3 is getting worked on