cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby fails with "Error running container image prepare"

[Soniya Murlidhar Vyas]

cs8 octavia wallaby fails with "Error running container image prepare"

Following is the traceback observed:-

[WARNING]: ('undercloud -> localhost', 'fa163e1f-b116-711c-f8a6-0000000000a0')
missing from stats
2022-06-09 13:55:29.266349 | fa163e1f-b116-711c-f8a6-0000000000a1 | TIMING | include_tasks | undercloud | 0:03:06.258828 | 0.02s
2022-06-09 13:55:29.289252 | 422c840a-0d1e-435b-bea6-a317c700977e | INCLUDED | /home/zuul/tripleo-deploy/standalone-ansible-uevowtdd/external_deploy_steps_tasks_step1.yaml | undercloud
2022-06-09 13:55:29.305694 | fa163e1f-b116-711c-f8a6-000000001448 | TASK | Container image prepare
2022-06-09 13:56:30.988720 | fa163e1f-b116-711c-f8a6-000000001448 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "401 Client Error: UNAUTHORIZED for url: https://quay.rdoproject.org/v2/tripleowallaby/openstack-rsyslog/manifests/376a92910b42e0e0f03dd304dabb7a60", "msg": "Error running container image prepare: 401 Client Error: UNAUTHORIZED for url: https://quay.rdoproject.org/v2/tripleowallaby/openstack-rsyslog/manifests/376a92910b42e0e0f03dd304dabb7a60", "params": {}, "success": false}
2022-06-09 13:56:30.992629 | fa163e1f-b116-711c-f8a6-000000001448 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:04:07.985113 | 61.69s

PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
standalone : ok=248 changed=137 unreachable=0 failed=0 skipped=79 rescued=0 ignored=0
undercloud : ok=18 changed=6 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

Please refer following links for more details:

1. https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/0fbf8ef/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
2. https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/4219fe6/logs/undercloud/home/zuul/ansible.log.txt.gz
3. https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/12d1999/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[Ronelle Landy (rlandy)]

https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/0fbf8ef/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

[Soniya Murlidhar Vyas (svyas)]

Following is the traceback observed in [1]:

2022-06-09 13:56:05,169 67973 WARNING tripleo_common.image.image_uploader [ ] Re-authenticating: id af32f8507626bcfa8e489a7cd1826321188d4919, count 10
2022-06-09 13:56:05,172 67973 DEBUG urllib3.connectionpool [ ] Resetting dropped connection: quay.rdoproject.org
2022-06-09 13:56:05,191 67973 DEBUG urllib3.connectionpool [ ] https://quay.rdoproject.org:443 "GET /v2/ HTTP/1.1" 401 4
2022-06-09 13:56:05,191 67973 DEBUG tripleo_common.image.image_uploader [ ] https://quay.rdoproject.org/v2/ status code 401
2022-06-09 13:56:05,192 67973 DEBUG tripleo_common.image.image_uploader [ ] Using bearer token auth
2022-06-09 13:56:05,194 67973 DEBUG urllib3.connectionpool [ ] Resetting dropped connection: quay.rdoproject.org
2022-06-09 13:56:05,225 67973 DEBUG urllib3.connectionpool [ ] https://quay.rdoproject.org:443 "GET /v2/auth?service=quay.rdoproject.org&scope=repository%3Atripleowallaby%2Fopenstack-neutron-metadata-agent-ovn%3Apull HTTP/1.1" 200 937
2022-06-09 13:56:05,227 67973 DEBUG tripleo_common.image.image_uploader [ ] Session authenticated: id b1e1aba07925810b12391e9cfc601c7482eefc6b
2022-06-09 13:56:05,227 67973 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleowallaby/openstack-neutron-metadata-agent-ovn:376a92910b42e0e0f03dd304dabb7a60-updated-20220609134036] Failed uploading the target image
2022-06-09 13:56:05,231 67973 INFO tripleo_common.image.image_uploader [ ] [quay.rdoproject.org/tripleowallaby/openstack-swift-account:376a92910b42e0e0f03dd304dabb7a60] Starting upload image process
2022-06-09 13:56:05,232 67973 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2022-06-09 13:56:05,234 67973 DEBUG urllib3.connectionpool [ ] Starting new HTTP connection (1): 192.168.24.1:8787
2022-06-09 13:56:05,235 67973 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
2022-06-09 13:56:05,236 67973 DEBUG tripleo_common.image.image_uploader [ ] http://192.168.24.1:8787/v2/ status code 200
2022-06-09 13:56:05,238 67973 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/tripleowallaby/openstack-swift-account/tags/list HTTP/1.1" 404 196
2022-06-09 13:56:05,238 67973 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 2bee4f764201101ad89d85b0beaf3149be7af142, status 404, reason Not Found, text

From the above traceback we can see that image has failed to get uploaded.

[1] https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/0fbf8ef/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-octavia-wallaby/4219fe6/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

[Rabi Mishra (rabi)]

> 401 Client Error: UNAUTHORIZED for url: https://quay.rdoproject.org/v2/tripleowallaby/openstack-glance-api/manifests/376a92910b42e0e0f03dd304dabb7a60

Namespace looks incorrect , should be https://quay.rdoproject.org/v2/tripleowallabycentos8 I guess, probably need to update[1].

[1] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/centosci/CentOS-8/wallaby-current-tripleo.yml#L12

[Soniya Murlidhar Vyas (svyas)]

this issue is also seen centos-8 standalone cloudops wallaby - https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-cloudops-wallaby/eabc877/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-cloudops-wallaby/3df7283/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

[Soniya Murlidhar Vyas (svyas)]

same is happening with c8 standalone tempest wallaby and fs01 tripleo wallaby -

1. https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-wallaby/eb4645d/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

2. https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-wallaby/d85d060/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

3. https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/f4cc084/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

[Ronelle Landy (rlandy)]

https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/ may fix this. Patch is in gates right now

[Ronelle Landy (rlandy)]

Trying testproject https://review.rdoproject.org/r/c/testproject/+/40101 with depends-on: https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/

[Dariusz Smigiel (smigiel-dariusz)]

After testing it with depends-on: https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/ it's still failing.

Breaking change: https://github.com/openstack/tripleo-ci/commit/a70a36db14e73008f5cfef17178170f2fbbc9b33

[Ronelle Landy (rlandy)]

@ananya: component jobs are trying to pull containers from the RDO quay registry with the namespace from the quay.io registry

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568

[Ananya Banerjee (frenzyfriday)]

Looks like wallaby c8 component jobs pass if we change this https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568

But isnt https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568/1/config/release/tripleo-ci/CentOS-8/wallaby.yml#26 supposed to be the namespace in quay.io, as we are still using quay.io and not rdoproj for the check jobs https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568/1/config/release/tripleo-ci/CentOS-8/wallaby.yml#24 ?

test proj: https://review.rdoproject.org/r/c/testproject/+/40101

[Soniya Murlidhar Vyas (svyas)]

On wallaby/Cloudops/tempest, we are green[1],

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/845568
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/12fc4167c70531da80403746c1f172ec56570679
Submitter: "Zuul (22348)"
Branch: master

commit 12fc4167c70531da80403746c1f172ec56570679
Author: frenzyfriday <email address hidden>
Date: Mon Jun 13 10:40:41 2022 +0200

    Changing the namespace for wallabyc8 to tripleowallabycentos8 only for component jobs

    The file wallaby.yml is used both by component and check jobs. We need to change the namespace to tripleowallabycentos8 only for the component jobs as they use registry quay.rdoproj [1] [2]

    [1] https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/multinode-vexxhost.yml#L53
    [2] https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/ovb-vexxhost.yml#L39

    Related-Bug: 1978319

    Testproj: https://review.rdoproject.org/r/c/testproject/+/43594

    Change-Id: I2fc2bea2b27ea688f68d3c96eb3f7e5d29405e33