tripleo upgrade jobs timeout on stable/ocata

[Emilien Macchi]

Alex and I noticed that TripleO upgrade jobs timeout quite often in stable/ocata (and also probably on master).
We found out that it's when running on RAX public cloud (iad and ord AZs).

Here are some numbers:
https://www.diffchecker.com/14g29Xoh

We can see that it's not only the steps where packages are installed (in that case we could say it's networking related) but also when doing basic workflows such as setting up SSH keys (see SshHostPubKey). We think it's probably an overcommittion on the hypervisor that would create bad performances on the VMs.

Any help from RAX folks is welcome to figure out if we can improve the situation.

[Emilien Macchi (emilienm)]

Example of job that timeouts: http://logs.openstack.org/87/481587/2/check/gate-tripleo-ci-centos-7-multinode-upgrades/798ee4b/

[Alex Schultz (alex-schultz)]

It also happens on ovh http://logs.openstack.org/37/477637/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-upgrades/2ecaf7c/

[Tom Barron (tpb)]

gate-tripleo-ci-centos-7-multinode-upgrades consistently fails in https://review.openstack.org/#/c/499111 with 143 timeout issue, e.g. http://logs.openstack.org/11/499111/2/check/gate-tripleo-ci-centos-7-multinode-upgrades/e8c5dec/console.html#_2017-09-09_14_11_06_389943

[Sagi (Sergey) Shnaidman (sshnaidm)]

We need to use tripleo upgrade role for upgrades, then we can continue to work on this.

[Tom Barron (tpb)]

Is there an ETA for tripleo upgrade role here, if that blocks the fix for this one? I'm asking b/c I have an important backport for manila ( https://bugs.launchpad.net/tripleo/+bug/1712842 ) that is stuck in stable/ocata on timeout failures on gate-tripleo-ci-centos-7-multinode-upgrades job.

[Tom Barron (tpb)]

Got lucky on a recheck of fix for LP:1712842 so it's no longer blocked.

[wes hayutin (weshayutin)]

starting to migrate these jobs to rdo sf https://review.rdoproject.org/r/#/c/9831/

[Alex Schultz (alex-schultz)]

So we dropped the upgrade jobs from upstream infra and moved them to rdo cloud. Closing this bug as it's no longer an issue.