Property error: resources.NovaCompute.properties.image: Error validating value 'overcloud-full': No images matching {'name': 'overcloud-full'}

[Bhagyashri Shewale]

2021-07-14 20:54:54 | Stack overcloud/4130e97e-0a7b-47f3-8aa9-003541f731c6 CREATE_FAILED
2021-07-14 20:54:54 |
2021-07-14 20:54:54 | overcloud.Compute.0:
2021-07-14 20:54:54 | resource_type: OS::TripleO::Compute
2021-07-14 20:54:54 | physical_resource_id:
2021-07-14 20:54:54 | status: CREATE_FAILED
2021-07-14 20:54:54 | status_reason: |
2021-07-14 20:54:54 | resources[0]: Property error: resources.NovaCompute.properties.image: Error validating value 'overcloud-full': No images matching {'name': 'overcloud-full'}.
2021-07-14 20:54:54 | overcloud.Compute.1:
2021-07-14 20:54:54 | resource_type: OS::TripleO::Compute
2021-07-14 20:54:54 | physical_resource_id:
2021-07-14 20:54:54 | status: CREATE_FAILED
2021-07-14 20:54:54 | status_reason: |
2021-07-14 20:54:54 | resources[1]: Property error: resources.NovaCompute.properties.image: Error validating value 'overcloud-full': No images matching {'name': 'overcloud-full'}.
2021-07-14 20:54:55 | + status_code=1
2021-07-14 20:54:55 | + openstack stack list
2021-07-14 20:54:55 | + grep -q overcloud
2021-07-14 20:54:59 | + openstack stack list
2021-07-14 20:54:59 | + grep -Eq '(CREATE|UPDATE)_COMPLETE'
2021-07-14 20:55:02 | + openstack stack failures list overcloud --long
2021-07-14 20:55:06 | ++ openstack stack resource list --nested-depth 5 overcloud
2021-07-14 20:55:06 | ++ grep 'StructuredDeployment '
2021-07-14 20:55:06 | ++ grep FAILED
2021-07-14 20:55:06 | ++ cut -d '|' -f3
2021-07-14 20:55:20 | + exit 1

Affected jobs:

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train
periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train
periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train

#links:

[1]: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/ab8385f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[2]: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/ab8385f/logs/undercloud/var/log/extra/errors.txt.gz
[3]: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/ab8385f/job-output.txt

[Rabi Mishra (rabi)]

Looks like it's intermittent with some successful job runs.

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train

image seems to be present in glance and the request from heat returns 200 OK[1]. May be something to do with the cache getting invalidated[2].

If the issue persists we probably need to investigate further.

[1]
2021-07-14 20:47:32.281 16 INFO eventlet.wsgi.server [req-5e8bb47b-af3d-43fa-87dd-48796463f1e7 7486475e02714119b0cd9559853ac033 0e9d978eb0e14af2bc58aa398ce60eca - default default] 192.168.24.1 - - [14/Jul/2021 20:47:32] "GET /v2/images?name=overcloud-full&limit=20 HTTP/1.1" 200 272 0.238217

[2]
2021-07-14 20:51:37.412 7 DEBUG dogpile.cache.region [req-164d15ec-d225-4360-b960-6581143fb3e4 - admin - default default] No value present for key: 'heat.engine.clients.os.glance:_find_image_id|0e9d978eb0e14af2bc58aa398ce60eca overcloud-full' _is_cache_miss /usr/lib/python3.6/site-packages/dogpile/cache/region.py:798
2021-07-14 20:51:37.412 7 DEBUG dogpile.lock [req-164d15ec-d225-4360-b960-6581143fb3e4 - admin - default default] Calling creation function for not-yet-present value _enter_create /usr/lib/python3.6/site-packages/dogpile/lock.py:178
2021-07-14 20:51:37.440 7 DEBUG heat.engine.scheduler [req-164d15ec-d225-4360-b960-6581143fb3e4 - admin - default default] Task create from TemplateResource "MySQLClient" Stack "overcloud-ControllerServiceChain-3ughhnhbzqkx-ServiceChain-hsjwdkdcuo32-13-lzgyp3njglax" [7c6d8f47-a087-4ee7-a15c-c305e7982bd6] starting start /usr/lib/python3.6/site-packages/heat/engine/scheduler.py:177

[Ronelle Landy (rlandy)]

We are picking up this same error in downstream on 16.2

https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2/278026a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-16.2/96855a5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

It's also taking out real baremetal jobs:

https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envB-3ctlr_1comp-featureset001-baremetal-rhos-16.2/2e78add/logs/undercloud/home/zuul/overcloud_deploy.log

[Ronelle Landy (rlandy)]

https://github.com/openstack/tripleo-quickstart-extras/commit/9c6667d9156f7228bc91871db820823485ba94ed changed image upload.
Trying to put train back to use legacy

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/800965

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/800965
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/d91c7f6ee4ca09356de97da5024db4a9cdef3194
Submitter: "Zuul (22348)"
Branch: master

commit d91c7f6ee4ca09356de97da5024db4a9cdef3194
Author: Ronelle Landy <email address hidden>
Date: Thu Jul 15 11:53:39 2021 -0400

    Use legacy image upload for train

    Commit 9c6667d9156f7228bc91871db820823485ba94ed
    switched to use operators for uploading images on all
    releases from train onwards.

    Train jobs are no failing deploy, unable to find
    the overcloud-full image.

    This patch adds train to the relesase that use
    the legacy upload script.

    Depends-On: https://review.opendev.org/c/openstack/python-tripleoclient/+/801019

    Change-Id: Iabcc33deaa494449799cf5ffe9dca44dcabed52e
    Closes-Bug: #1936287

[chandan kumar (chkumar246)]

I am seeing similar issue on Ussuri during overcloud deploy
https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-ussuri/499052a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

```
overcloud.Compute.0:
2021-07-20 14:52:05 | resource_type: OS::TripleO::Compute
2021-07-20 14:52:05 | physical_resource_id:
2021-07-20 14:52:05 | status: CREATE_FAILED
2021-07-20 14:52:05 | status_reason: |
2021-07-20 14:52:05 | resources[0]: Property error: resources.NovaCompute.properties.image: Error validating value 'overcloud-full': No images matching {'name': 'overcloud-full'}.
2021-07-20 14:52:05 | + status_code=1
2021-07-20 14:52:05 | + openstack stack list
2021-07-20 14:52:05 | + grep -q overcloud
2021-07-20 14:52:07 | + openstack stack list
2021-07-20 14:52:07 | + grep -Eq '(CREATE|UPDATE)_COMPLETE'
2021-07-20 14:52:10 | + openstack stack failures list overcloud --long
2021-07-20 14:52:14 | ++ openstack stack resource list --nested-depth 5 overcloud
2021-07-20 14:52:14 | ++ grep 'StructuredDeployment '
2021-07-20 14:52:14 | ++ grep FAILED
2021-07-20 14:52:14 | ++ cut -d '|' -f3
2021-07-20 14:52:26 | + exit 1
```

Do we need to enable legacy mode for ussuri also?

[Rabi Mishra (rabi)]

I think https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/801192 would fix it. We use nova in ussuri in some scenarios.

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/c/testproject/+/24995 fix the

periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-ussuri SUCCESS 3h 17m 12s the job.

I think we can close it once https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/801192 merges.

[Alex Schultz (alex-schultz)]

The original train issue was because the image upload was using --local which is not used unless using metalsmith. For train nova should be used so it shouldn't have --local

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/ab8385f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz