Zun

[stein] Image validation does not work for unofficial images from Docker Hub

Bug #1840629 reported by Radosław Piliszek on 2019-08-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zun
Undecided
Unassigned

Bug Description

EDIT: changed topic to reflect the real issue

OLD DESCRIPTION (read the thread for real issue):
gbraad/openstack-client:alpine image is no longer available from Docker Hub. The default cloud shell does not start.

hongbin (hongbin034) wrote :

It looks the image is still there https://hub.docker.com/r/gbraad/openstack-client ?

Radosław Piliszek (yoctozepto) wrote :

Yeah, sorry. It seems my installation does not allow for any unofficial image via Zun. So images named like ubuntu work but not e.g. gbraad/openstack-client. Docker itself happily accepts them but not Zun. Any idea whether it's a config issue or bug?

hongbin (hongbin034) wrote :

I ran this command:

  $ zun run gbraad/openstack-client:alpine

It looks Zun does accepts it. I also opened the cloud shell with default image, and the cloud shell seems to work correctly.

It looks I couldn't reproduce the error you saw. Could you elaborate the issue you were hitting? and how to reproduce it?

Radosław Piliszek (yoctozepto) wrote :

$ zun run gbraad/openstack-client:alpine
ERROR: Image gbraad/openstack-client:alpine could not be found. (HTTP 400) (Request-ID: req-c1a94210-3ae0-44f8-a72f-c5465df9c184)

Unfortunately nothing relevant in any logs (no mention of this req).

Docker directly works fine and Zun works fine with e.g.:
$ zun run alpine
[long output with table]

I will test this in kolla's CI. So far no idea what I could have done wrong in there.
If it passes in CI, I will probably be more puzzled but no better idea how to approach this.

Radosław Piliszek (yoctozepto) wrote :

I moved to Zun because bypassing Zun-UI has the same result. Better track it here.

affects: zun-ui → zun
Radosław Piliszek (yoctozepto) wrote :

Aaand it worked (un)fortunately.

Radosław Piliszek (yoctozepto) wrote :

I turned on debug locally but it's not helpful either:
2019-08-21 18:44:10.559 16 DEBUG zun.objects.container [req-8dccbe67-31f1-4b6a-b699-7d9b4deb247b bc48f489fae54c0a85962bc7b0b94d95 0916103ff8de4b088d1be43db1f30c38 default - -] Lazy-loading 'registry' on Container uuid 74dc9360-921d-4fb0-af86-17858bb13fa2 obj_load_attr /var/lib/kolla/venv/lib/python2.7/site-packages/zun/objects/container.py:312
2019-08-21 18:44:11.488 16 DEBUG zun.common.exception [req-8dccbe67-31f1-4b6a-b699-7d9b4deb247b bc48f489fae54c0a85962bc7b0b94d95 0916103ff8de4b088d1be43db1f30c38 default - -] Image gbraad/openstack-client:alpine could not be found. wrapped /var/lib/kolla/venv/lib/python2.7/site-packages/zun/common/exception.py:128

Radosław Piliszek (yoctozepto) wrote :

I caught some weirdness as well. Req IDs do not match:

client:
RESP: [400] Content-Length: 236 Content-Type: application/json Date: Wed, 21 Aug 2019 17:37:13 GMT OpenStack-API-Maximum-Version: container 1.32 OpenStack-API-Minimum-Version: container 1.1 OpenStack-API-Version: container 1.30 Server: Apache vary: OpenStack-API-Version x-openstack-request-id: req-280bba8e-fc9a-4467-9b98-1f49f706eee5
RESP BODY: {"errors": [{"status": 400, "code": "container.image-not-found", "links": [], "title": "Image gbraad/openstack-client:alpine could not be found.", "detail": "Image gbraad/openstack-client:alpine could not be found.", "request_id": ""}]}
POST call to container for http://cloud.internal.uco.uwb.edu.pl:9517/v1/containers?run=true used request id req-280bba8e-fc9a-4467-9b98-1f49f706eee5
Image gbraad/openstack-client:alpine could not be found. (HTTP 400) (Request-ID: req-280bba8e-fc9a-4467-9b98-1f49f706eee5)

API:
2019-08-21 19:37:17.216 19 DEBUG zun.common.exception [req-29ee84cd-d329-4c35-9075-99c643ff2aab bc48f489fae54c0a85962bc7b0b94d95 0916103ff8de4b088d1be43db1f30c38 default - -] Image gbraad/openstack-client:alpine could not be found. wrapped /var/lib/kolla/venv/lib/python2.7/site-packages/zun/common/exception.py:128

hongbin (hongbin034) wrote :

@Radosław,

A few questions:

* Which version of Zun you were using (i.e. master, stable/stein, stable/rocky, etc.)
* How did you install Zun (i.e. Kolla, Devstack, or manual install)

Also, it would be helpful if you can paste me the full log (with sensitive information removed).

@Hongbin,

1) Latest released (for Stein): 3.0.0
2) Kolla-Ansible

Full API logs: http://paste.openstack.org/show/FanzVkEZA1FMsUJb2DCF/

Thanks for handling.
I will try stable/stein too. Forgot I was on release actually.

Yeah, debugged that now:
in stable/stein the request works and the logs show:

client:
RESP: [202] Content-Length: 1015 Content-Type: application/json Date: Thu, 22 Aug 2019 08:07:40 GMT Location: http://cloud.internal.uco.uwb.edu.pl:9517/v1/containers/65392f0b-1494-4b81-be7e-e455daa5623a OpenStack-API-Maximum-Version: container 1.32 OpenStack-API-Minimum-Version: container 1.1 OpenStack-API-Version: container 1.30 Server: Apache vary: OpenStack-API-Version x-openstack-request-id: req-fe692e08-a8dc-4de3-8c7f-36dd8c01231f

API:
2019-08-22 10:07:44.485 18 WARNING zun.compute.api [req-fe692e08-a8dc-4de3-8c7f-36dd8c01231f bc48f489fae54c0a85962bc7b0b94d95 0916103ff8de4b088d1be43db1f30c38 default - -] Skip validation since image search failed with unexpected exception: Image gbraad/openstack-client:alpine could not be found.: ImageNotFound: Image gbraad/openstack-client:alpine could not be found.

So the unmatched req id issue is gone as well.

I found a related fix: https://review.opendev.org/#/c/657516/1/zun/compute/api.py

So it turns out the real problem is that "image validation" does work only with official images, not the rest.

hongbin (hongbin034) wrote :

Right,

Thanks for digging into the issue. The "image validation" feature relies on docker search, which has several limitation. I recommend to turn it off as well.

hongbin (hongbin034) wrote :

The image validation feature basically gives users a pre-flight check on provided image, but it turned to be broken in several scenario. In later release, this feature will be turned off by default.

We plan to deprecate this feature, so I am going to mark this bug as "not fix".

Changed in zun:
status: New → Won't Fix

Fine by me. Thanks for insights.

summary: - Default cloud shell broken due to unavailable image
+ [stein] Image validation does not work for unofficial images from Docker
+ Hub
description: updated
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers