Comment 1 for bug 1840616

Revision history for this message
yatin (yatinkarel) wrote : Re: Master check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy

So noticed it's taking time for network operations(example: docker pull) on an overcloud node.
So for some images it takes time > 10 minutes(ideally should take few seconds as it's local network).
so on affected node all docker pull takes time.

Good to check if it's master only or affects all releases to properly isolate the issue.

Example 1(14 minute 21 second):- https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/edd7af2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-08-14_03_45_18
2019-08-14 03:45:18 | changed: [overcloud-controller-0] => {
2019-08-14 03:45:18 |
2019-08-14 03:45:18 | "changed": true,
2019-08-14 03:45:18 | "cmd": "docker pull 192.168.24.1:8787/tripleomaster/centos-binary-cinder-volume:dff0f7b2c1ce50d9929321f65f1ab290f2d042a9_e8dbf8af-updated-20190814013900",
2019-08-14 03:45:18 | "delta": "0:14:21.982103",
2019-08-14 03:45:18 | "end": "2019-08-14 03:39:27.913156",
2019-08-14 03:45:18 | "rc": 0,
2019-08-14 03:45:18 | "start": "2019-08-14 03:25:05.931053"
2019-08-14 03:45:18 | }

Example 2(10 minute 32 second):- https://logs.rdoproject.org/64/676364/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/19bb5b1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-08-15_16_38_15

2019-08-15 16:38:15 | changed: [overcloud-controller-2] => {
2019-08-15 16:38:15 |
2019-08-15 16:38:15 | "changed": true,
2019-08-15 16:38:15 | "cmd": "docker pull 192.168.24.1:8787/tripleomaster/centos-binary-cinder-volume:6cd9e55d69e90ea73219abced4e9a3f2372b204e_10e135ca-updated-20190815144647",
2019-08-15 16:38:15 | "delta": "0:10:32.870128",
2019-08-15 16:38:15 | "end": "2019-08-15 16:37:34.438094",
2019-08-15 16:38:15 | "rc": 0,
2019-08-15 16:38:15 | "start": "2019-08-15 16:27:01.567966"
2019-08-15 16:38:15 | }