docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64, level=fatal msg="Error determining repository tags: http: unexpected EOF reading trailer"

[wes hayutin]

2020-02-07 19:17:43 | Running skopeo inspect docker://docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64
2020-02-07 19:18:22 | Error inspecting image: docker://docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64
2020-02-07 19:18:22 | time="2020-02-07T19:18:22Z" level=fatal msg="Error determining repository tags: http: unexpected EOF reading trailer"
2020-02-07 19:18:22 |
2020-02-07 19:18:22 | END return value: 1

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_590/706465/2/gate/tripleo-ci-centos-7-containers-multinode-rocky/590e608/logs/undercloud/home/zuul/overcloud_prep_containers.log

This is causing failures in several rocky jobs.

[wes hayutin (weshayutin)]

https://34221fa108f22ffa4288-f918abb3dcdfb01ed824790c230ea283.ssl.cf5.rackcdn.com/706215/5/gate/tripleo-ci-centos-7-containers-multinode-rocky/86decbf/logs/undercloud/home/zuul/overcloud_prep_containers.log

[wes hayutin (weshayutin)]

look for additional hits w/

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Error%20inspecting%20image%3A%20docker%3A%2F%2Fdocker.io%2Fceph%2Fdaemon%3Av3.2.10-stable-3.2-luminous%5C%22

[wes hayutin (weshayutin)]

it's only happening recently and not all the time.
http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-rocky

[chandan kumar (chkumar246)]

The job appears to be green http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-rocky with few failures.

[Francesco Pantano (fmount)]

Sounds like it's related to the "list all tags" call which is blocked by the registry [1].
I'm not sure why it's not always blocked, but basically when a 401 status is returned, the error you see in $subject is generated.
Solution 1: shrink the container metadata
Solution 2: can we query less destructively

[1] https://github.com/containers/skopeo/blob/1094c7d61dddb5db0ff65cadadf6693a7e160fd7/cmd/skopeo/inspect.go#L175

[2] http://pastebin.fr/60858

[Dimitri Savineau (dsavineau)]

I've remove some unused ceph container tags (~150) if it helps.

But it's definitly something between skopeo and the docker.io registry.

[Marios Andreou (marios-b)]

right now skopeo inspect docker://docker.io/ceph/daemon:v3.2.10-stable-3.2-luminous-centos-7-x86_64

works for me .... as per comment #4 above http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode-rocky looks pretty green.

I am considering moving this to invalid or fix-released ... unless folks can point to this happening in ci logs recently?

[Francesco Pantano (fmount)]

Thanks Marios, same thoughts from my side.
Let's see how the CI goes (and if future failures will be related to this issue), then we can move it to invalid since I think it's not a real tripleo bug.