[stein] Image validation does not work for unofficial images from Docker Hub

[Radosław Piliszek]

EDIT: changed topic to reflect the real issue

OLD DESCRIPTION (read the thread for real issue):
gbraad/openstack-client:alpine image is no longer available from Docker Hub. The default cloud shell does not start.

[hongbin (hongbin034)]

It looks the image is still there https://hub.docker.com/r/gbraad/openstack-client ?

[Radosław Piliszek (yoctozepto)]

Yeah, sorry. It seems my installation does not allow for any unofficial image via Zun. So images named like ubuntu work but not e.g. gbraad/openstack-client. Docker itself happily accepts them but not Zun. Any idea whether it's a config issue or bug?

[hongbin (hongbin034)]

I ran this command:

  $ zun run gbraad/openstack-client:alpine

It looks Zun does accepts it. I also opened the cloud shell with default image, and the cloud shell seems to work correctly.

It looks I couldn't reproduce the error you saw. Could you elaborate the issue you were hitting? and how to reproduce it?

[Radosław Piliszek (yoctozepto)]

$ zun run gbraad/openstack-client:alpine
ERROR: Image gbraad/openstack-client:alpine could not be found. (HTTP 400) (Request-ID: req-c1a94210-3ae0-44f8-a72f-c5465df9c184)

Unfortunately nothing relevant in any logs (no mention of this req).

Docker directly works fine and Zun works fine with e.g.:
$ zun run alpine
[long output with table]

I will test this in kolla's CI. So far no idea what I could have done wrong in there.
If it passes in CI, I will probably be more puzzled but no better idea how to approach this.

[Radosław Piliszek (yoctozepto)]

I moved to Zun because bypassing Zun-UI has the same result. Better track it here.

[Radosław Piliszek (yoctozepto)]

CI test: https://review.opendev.org/677663

[Radosław Piliszek (yoctozepto)]

Aaand it worked (un)fortunately.

[Radosław Piliszek (yoctozepto)]

I turned on debug locally but it's not helpful either:
2019-08-21 18:44:10.559 16 DEBUG zun.objects.container [req-8dccbe67-31f1-4b6a-b699-7d9b4deb247b bc48f489fae54c0a85962bc7b0b94d95 0916103ff8de4b088d1be43db1f30c38 default - -] Lazy-loading 'registry' on Container uuid 74dc9360-921d-4fb0-af86-17858bb13fa2 obj_load_attr /var/lib/kolla/venv/lib/python2.7/site-packages/zun/objects/container.py:312
2019-08-21 18:44:11.488 16 DEBUG zun.common.exception [req-8dccbe67-31f1-4b6a-b699-7d9b4deb247b bc48f489fae54c0a85962bc7b0b94d95 0916103ff8de4b088d1be43db1f30c38 default - -] Image gbraad/openstack-client:alpine could not be found. wrapped /var/lib/kolla/venv/lib/python2.7/site-packages/zun/common/exception.py:128

[Radosław Piliszek (yoctozepto)]

I caught some weirdness as well. Req IDs do not match:

client:
RESP: [400] Content-Length: 236 Content-Type: application/json Date: Wed, 21 Aug 2019 17:37:13 GMT OpenStack-API-Maximum-Version: container 1.32 OpenStack-API-Minimum-Version: container 1.1 OpenStack-API-Version: container 1.30 Server: Apache vary: OpenStack-API-Version x-openstack-request-id: req-280bba8e-fc9a-4467-9b98-1f49f706eee5
RESP BODY: {"errors": [{"status": 400, "code": "container.image-not-found", "links": [], "title": "Image gbraad/openstack-client:alpine could not be found.", "detail": "Image gbraad/openstack-client:alpine could not be found.", "request_id": ""}]}
POST call to container for http://cloud.internal.uco.uwb.edu.pl:9517/v1/containers?run=true used request id req-280bba8e-fc9a-4467-9b98-1f49f706eee5
Image gbraad/openstack-client:alpine could not be found. (HTTP 400) (Request-ID: req-280bba8e-fc9a-4467-9b98-1f49f706eee5)

API:
2019-08-21 19:37:17.216 19 DEBUG zun.common.exception [req-29ee84cd-d329-4c35-9075-99c643ff2aab bc48f489fae54c0a85962bc7b0b94d95 0916103ff8de4b088d1be43db1f30c38 default - -] Image gbraad/openstack-client:alpine could not be found. wrapped /var/lib/kolla/venv/lib/python2.7/site-packages/zun/common/exception.py:128

[hongbin (hongbin034)]

@Radosław,

A few questions:

* Which version of Zun you were using (i.e. master, stable/stein, stable/rocky, etc.)
* How did you install Zun (i.e. Kolla, Devstack, or manual install)

Also, it would be helpful if you can paste me the full log (with sensitive information removed).

[Radosław Piliszek (yoctozepto)]

@Hongbin,

1) Latest released (for Stein): 3.0.0
2) Kolla-Ansible

Full API logs: http://paste.openstack.org/show/FanzVkEZA1FMsUJb2DCF/

Thanks for handling.
I will try stable/stein too. Forgot I was on release actually.

[Radosław Piliszek (yoctozepto)]

Yeah, debugged that now:
in stable/stein the request works and the logs show:

client:
RESP: [202] Content-Length: 1015 Content-Type: application/json Date: Thu, 22 Aug 2019 08:07:40 GMT Location: http://cloud.internal.uco.uwb.edu.pl:9517/v1/containers/65392f0b-1494-4b81-be7e-e455daa5623a OpenStack-API-Maximum-Version: container 1.32 OpenStack-API-Minimum-Version: container 1.1 OpenStack-API-Version: container 1.30 Server: Apache vary: OpenStack-API-Version x-openstack-request-id: req-fe692e08-a8dc-4de3-8c7f-36dd8c01231f

API:
2019-08-22 10:07:44.485 18 WARNING zun.compute.api [req-fe692e08-a8dc-4de3-8c7f-36dd8c01231f bc48f489fae54c0a85962bc7b0b94d95 0916103ff8de4b088d1be43db1f30c38 default - -] Skip validation since image search failed with unexpected exception: Image gbraad/openstack-client:alpine could not be found.: ImageNotFound: Image gbraad/openstack-client:alpine could not be found.

So the unmatched req id issue is gone as well.

I found a related fix: https://review.opendev.org/#/c/657516/1/zun/compute/api.py

So it turns out the real problem is that "image validation" does work only with official images, not the rest.

[hongbin (hongbin034)]

Right,

Thanks for digging into the issue. The "image validation" feature relies on docker search, which has several limitation. I recommend to turn it off as well.

[hongbin (hongbin034)]

The image validation feature basically gives users a pre-flight check on provided image, but it turned to be broken in several scenario. In later release, this feature will be turned off by default.

We plan to deprecate this feature, so I am going to mark this bug as "not fix".

[Radosław Piliszek (yoctozepto)]

Fine by me. Thanks for insights.