periodic- centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master fails to download tempest cirros image

[Marios Andreou]

This blocks the baremetal component promotions

at [1][2] the periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master fails after successful deployment during tempest specifically trying to download the cirros image trace looks like:

         config_tempest.constants [-] Downloading 'https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img' and saving as 'etc/cirros-0.4.0-x86_64-disk.img'
        2020-03-16 02:11:08.206121 | primary | 2020-03-16 02:11:07.984 235883 CRITICAL tempest [-] Unhandled error: urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>
        2020-03-16 02:11:08.206131 | primary | 2020-03-16 02:11:07.984 235883 ERROR tempest Traceback (most recent call last):
        2020-03-16 02:11:08.206137 | primary | 2020-03-16 02:11:07.984 235883 ERROR tempest File "/usr/lib64/python3.6/urllib/request.py", line 1318, in do_open
        ...
        2020-03-16 02:11:08.206249 | primary | 2020-03-16 02:11:07.984 235883 ERROR tempest raise err
        2020-03-16 02:11:08.206254 | primary | 2020-03-16 02:11:07.984 235883 ERROR tempest File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
        2020-03-16 02:11:08.206260 | primary | 2020-03-16 02:11:07.984 235883 ERROR tempest sock.connect(sa)
        2020-03-16 02:11:08.206265 | primary | 2020-03-16 02:11:07.984 235883 ERROR tempest OSError: [Errno 101] Network is unreachable

[1] https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/54a83b8/job-output.txt
[2] https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/61ab8a3/job-output.txt

[Ronelle Landy (rlandy)]

Note: possible workaround:

you can set here for example: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset001.yml#L190
image.image_path: 'https://download.another.cirros.repo'

[wes hayutin (weshayutin)]

put up a backup..
http://images.rdoproject.org/cirros/cirros-0.4.0-x86_64-disk.img

the image is not blocked from https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
but the downloads have failing even manually.. so...

we're testing w/ that

[Marios Andreou (marios-b)]

fix there

        * add backup tempest cirros url
        * Change-Id: Ibe5ff806539479fb94d4a61aff9ac06a883a6895
        * https://review.opendev.org/#/c/713505/3

bug also seen in cloudops scen2 today

        * https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-cloudops-master/21e4ed0/job-output.txt
        * 2020-03-18 06:20:11.467767 | primary | 2020-03-18 06:11:08.176 229109 INFO config_tempest.constants [-] Downloading 'https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img' and saving as 'etc/cirros-0.4.0-x86_64-disk.img'
2020-03-18 06:20:11.467778 | primary | 2020-03-18 06:13:17.477 229109 WARNING config_tempest.constants [-] <urlopen error [Errno 101] Network is unreachable>, Retrying in 3 seconds.: urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

[wes hayutin (weshayutin)]

patch is merging .. and was transient anyway

[Marios Andreou (marios-b)]

also hitting that in cinder standalone4 today

        * https://logserver.rdoproject.org/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-cinder-master/6eeb8cc/job-output.txt
        * 2020-03-24 02:34:56.572451 | primary | 2020-03-24 02:25:50.947 155994 INFO config_tempest.constants [-] Downloading 'https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img' and saving as 'etc/cirros-0.4.0-x86_64-disk.img'
2020-03-24 02:34:56.572456 | primary | 2020-03-24 02:28:02.709 155994 WARNING config_tempest.constants [-] <urlopen error [Errno 101] Network is unreachable>, Retrying in 3 seconds.: urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

After some digging, we *do* have that in the featureset 052 [1] - however we are using a featureset_override to set the same tempest_tempest_conf_overrides [2] - AFAICS from the job [3] the ones on the job win.

Going to post something in a sec to add this to the job definition then we should remove from the featureset

[1] https://github.com/openstack/tripleo-quickstart/blob/16beb60b8f6f6dc7233dd3a8e2e839ec48ee8a64/config/general_config/featureset052.yml#L67
[2] https://github.com/rdo-infra/rdo-jobs/blob/f968638e42e4edce27a4ffdad09929ab0a462346/zuul.d/standalone-jobs.yaml#L267-L282
[3] https://logserver.rdoproject.org/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-cinder-master/6eeb8cc/logs/undercloud/home/zuul/tempest/etc/profile.yaml.txt.gz

[Marios Andreou (marios-b)]

following comment #5 i posted that rdo-jobs master: Add tempest image_path override into centos-8 standalone-4 https://review.rdoproject.org/r/26076

[Marios Andreou (marios-b)]

sorry hit return for comment #6 too quickly...
we discussed this on today's calls... chkumar has an alternative fix is os-tempest itself https://review.opendev.org/#/q/topic:tripleocirros+(status:open+OR+status:merged

however the underlying issue of, when we have a featureset var and override that in the featureset_override of the job definition, the job definition wins.

[Marios Andreou (marios-b)]

filed a new bug for the issue described in comment #5 about the tempest_tempest_conf_overrides being dropped there https://bugs.launchpad.net/tripleo/+bug/1868938

[chandan kumar (chkumar246)]

In order to avoid the usage of override and duplicating the cirros image in job, We are making it default in os_tempest role and tempest.yaml playbook.
https://review.opendev.org/#/c/714601/ in os_tempest - Added tempest_tempestconf_image var for tempestconf image

and https://review.opendev.org/714609 Set tempest_tempestconf_image path for periodic in TQE tempest playbook will fix the issue.