tripleo-tox-molecule consistently failing with failed: [localhost] (item=https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/ironic-python-agent.tar)

[Pooja Jadhav]

tripleo-tox-molecule consistently failing with failed: [localhost] (item=https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/ironic-python-agent.tar) with below traceback :

2022-08-10 12:45:20.330202 | primary | TASK [validate image urls] *****************************************************
2022-08-10 12:45:20.330214 | primary | Wednesday 10 August 2022 12:45:18 +0000 (0:00:00.024) 0:00:00.024 ******
2022-08-10 12:45:20.330227 | primary | failed: [localhost] (item=https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/ironic-python-agent.tar) => {
2022-08-10 12:45:20.330239 | primary |  "ansible_loop_var": "item",
2022-08-10 12:45:20.330253 | primary |  "changed": false,
2022-08-10 12:45:20.330265 | primary |  "connection": "close",
2022-08-10 12:45:20.330278 | primary |  "content_type": "text/html; charset=iso-8859-1",
2022-08-10 12:45:20.330289 | primary |  "date": "Wed, 10 Aug 2022 12:45:19 GMT",
2022-08-10 12:45:20.330308 | primary |  "elapsed": 0,
2022-08-10 12:45:20.330320 | primary |  "item": "https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/ironic-python-agent.tar",
2022-08-10 12:45:20.330333 | primary |  "redirected": false,
2022-08-10 12:45:20.330344 | primary |  "server": "Apache/2.4.37 (centos) OpenSSL/1.1.1k",
2022-08-10 12:45:20.330356 | primary |  "status": 404,
2022-08-10 12:45:20.330368 | primary |  "url": "https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/ironic-python-agent.tar"
2022-08-10 12:45:20.330380 | primary | }
2022-08-10 12:45:20.330392 | primary | 
2022-08-10 12:45:20.330404 | primary | MSG:
2022-08-10 12:45:20.330416 | primary | 
2022-08-10 12:45:20.330428 | primary | Status code was 404 and not [200]: HTTP Error 404: Not Found
2022-08-10 12:45:20.330440 | primary | ok: [localhost] => (item=https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/overcloud-full.tar) => {
2022-08-10 12:45:20.330452 | primary |  "accept_ranges": "bytes",
2022-08-10 12:45:20.330464 | primary |  "ansible_loop_var": "item",
2022-08-10 12:45:20.330476 | primary |  "changed": false,
2022-08-10 12:45:20.330488 | primary |  "connection": "close",
2022-08-10 12:45:20.330500 | primary |  "content_length": "1551831040",
2022-08-10 12:45:20.330512 | primary |  "content_type": "application/x-tar",
2022-08-10 12:45:20.330524 | primary |  "cookies": {},
2022-08-10 12:45:20.330535 | primary |  "cookies_string": "",
2022-08-10 12:45:20.330547 | primary |  "date": "Wed, 10 Aug 2022 12:45:20 GMT",
2022-08-10 12:45:20.330559 | primary |  "elapsed": 0,
2022-08-10 12:45:20.330571 | primary |  "etag": "\"5c7f1000-5e5cbb7e0b9f5\"",
2022-08-10 12:45:20.330583 | primary |  "item": "https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/overcloud-full.tar",
2022-08-10 12:45:20.330595 | primary |  "last_modified": "Tue, 09 Aug 2022 09:40:09 GMT",
2022-08-10 12:45:20.330607 | primary |  "redirected": false,
2022-08-10 12:45:20.330618 | primary |  "server": "Apache/2.4.37 (centos) OpenSSL/1.1.1k",
2022-08-10 12:45:20.330664 | primary |  "status": 200,
2022-08-10 12:45:20.330690 | primary |  "url": "https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/overcloud-full.tar"
2022-08-10 12:45:20.330711 | primary | }
2022-08-10 12:45:20.330725 | primary | 
2022-08-10 12:45:20.330738 | primary | MSG:
2022-08-10 12:45:20.330750 | primary | 
2022-08-10 12:45:20.330762 | primary | OK (1551831040 bytes)

looks like ipa image is not present here: https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/

Detailed Logs :

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_081/852731/1/check/tripleo-tox-molecule/08135c2/job-output.txt

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ae5/852733/1/check/tripleo-tox-molecule/ae51d1e/job-output.txt

https://1725e75d82166188a24a-f734581470945d848d2e1e75c1fa1768.ssl.cf5.rackcdn.com/852594/3/check/tripleo-tox-molecule/9503765/job-output.txt

[Marios Andreou (marios-b)]

k lets try re-pushing the image for current-tripleo e04837c814032381ae4e9d817c276a22 (https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5)

something must have happened during the promotion

running https://review.rdoproject.org/r/c/testproject/+/44504

[Marios Andreou (marios-b)]

actually not sure that is going to work. that will just build that image.

we need to get it onto the image server (if not with promotion then manually)

so we need both...
the image is missing in https://images.rdoproject.org/centos8/train/rdo_trunk/e04837c814032381ae4e9d817c276a22/

so after https://review.rdoproject.org/r/c/testproject/+/44504 it should push the image there and dpawlik will help us put it in to the current-tripleo path

[Marios Andreou (marios-b)]

after https://review.rdoproject.org/r/c/testproject/+/44504

looks like image is back OK @ https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/

lets wait for a green run to confirm then we can fix-released this

[Marios Andreou (marios-b)]

k green runs starting again in builds [1] - first example there https://zuul.opendev.org/t/openstack/build/b262ad7df18640bba8fdeb607e812e9b

[1] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-tox-molecule&skip=0