quay.io/app-sre/grafana:6.7.4 went missing from quay leading to fail all content provider jobs

Bug #1920873 reported by chandan kumar on 2021-03-23
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Critical
Unassigned

Bug Description

Most of the content provider jobs are failing as quay.io/app-sre/grafana:6.7.4 image does not exists on quay.io.

Here is the traceback:
https://447f476af5555fa473a2-ba0bbef8fa5bd9d33ddbd8694210833c.ssl.cf5.rackcdn.com/781622/2/check/tripleo-ci-centos-8-content-provider/e902500/logs/quickstart_install.log

```
TASK [container-build : Pull failed containers from RDO registry] **************
Tuesday 23 March 2021 05:35:37 +0000 (0:00:00.230) 0:46:04.613 *********
failed: [undercloud] (item=quay.io/app-sre/grafana:6.7.4) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": [
        "podman",
        "pull",
        "docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4"
    ],
    "delta": "0:00:00.874778",
    "end": "2021-03-23 05:35:38.377970",
    "item": "quay.io/app-sre/grafana:6.7.4",
    "rc": 125,
    "start": "2021-03-23 05:35:37.503192"
}

STDERR:

Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...
  manifest unknown: manifest unknown
Error: error pulling image "docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4": unable to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: unable to pull image: Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown

MSG:

non-zero return code
...ignoring

TASK [container-build : List down missing containers] **************************
Tuesday 23 March 2021 05:35:38 +0000 (0:00:01.337) 0:46:05.951 *********
ok: [undercloud] => (item={'msg': 'non-zero return code', 'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4'], 'stdout': '', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...\n manifest unknown: manifest unknown\nError: error pulling image "docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4": unable to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: unable to pull image: Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown', 'rc': 125, 'start': '2021-03-23 05:35:37.503192', 'end': '2021-03-23 05:35:38.377970', 'delta': '0:00:00.874778', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...', ' manifest unknown: manifest unknown', 'Error: error pulling image "docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4": unable to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: unable to pull image: Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown'], 'item': 'quay.io/app-sre/grafana:6.7.4', 'ansible_loop_var': 'item'})

TASK [container-build : Pull failed containers from dockerhub] *****************
Tuesday 23 March 2021 05:35:38 +0000 (0:00:00.117) 0:46:06.069 *********
failed: [undercloud] (item=quay.io/app-sre/grafana:6.7.4) => {
    "ansible_loop_var": "item",
    "changed": true,
    "cmd": [
        "podman",
        "pull",
        "grafana:6.7.4"
    ],
    "delta": "0:00:00.716062",
    "end": "2021-03-23 05:35:39.697942",
    "item": "quay.io/app-sre/grafana:6.7.4",
    "rc": 125,
    "start": "2021-03-23 05:35:38.981880"
}

STDERR:

Trying to pull registry.access.redhat.com/grafana:6.7.4...
  name unknown: Repo not found
Trying to pull registry.redhat.io/grafana:6.7.4...
  unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
Trying to pull docker.io/library/grafana:6.7.4...
  denied: requested access to the resource is denied
Error: error pulling image "grafana:6.7.4": unable to pull grafana:6.7.4: 3 errors occurred:
 * Error initializing source docker://registry.access.redhat.com/grafana:6.7.4: Error reading manifest 6.7.4 in registry.access.redhat.com/grafana: name unknown: Repo not found
 * Error initializing source docker://registry.redhat.io/grafana:6.7.4: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
 * Error initializing source docker://grafana:6.7.4: Error reading manifest 6.7.4 in docker.io/library/grafana: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

MSG:

non-zero return code
...ignoring

TASK [container-build : List down missing containers] **************************
Tuesday 23 March 2021 05:35:39 +0000 (0:00:01.208) 0:46:07.278 *********
ok: [undercloud] => (item={'msg': 'non-zero return code', 'cmd': ['podman', 'pull', 'grafana:6.7.4'], 'stdout': '', 'stderr': 'Trying to pull registry.access.redhat.com/grafana:6.7.4...\n name unknown: Repo not found\nTrying to pull registry.redhat.io/grafana:6.7.4...\n unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\nTrying to pull docker.io/library/grafana:6.7.4...\n denied: requested access to the resource is denied\nError: error pulling image "grafana:6.7.4": unable to pull grafana:6.7.4: 3 errors occurred:\n\t* Error initializing source docker://registry.access.redhat.com/grafana:6.7.4: Error reading manifest 6.7.4 in registry.access.redhat.com/grafana: name unknown: Repo not found\n\t* Error initializing source docker://registry.redhat.io/grafana:6.7.4: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication\n\t* Error initializing source docker://grafana:6.7.4: Error reading manifest 6.7.4 in docker.io/library/grafana: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required', 'rc': 125, 'start': '2021-03-23 05:35:38.981880', 'end': '2021-03-23 05:35:39.697942', 'delta': '0:00:00.716062', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull grafana:6.7.4', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Trying to pull registry.access.redhat.com/grafana:6.7.4...', ' name unknown: Repo not found', 'Trying to pull registry.redhat.io/grafana:6.7.4...', ' unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication', 'Trying to pull docker.io/library/grafana:6.7.4...', ' denied: requested access to the resource is denied', 'Error: error pulling image "grafana:6.7.4": unable to pull grafana:6.7.4: 3 errors occurred:', '\t* Error initializing source docker://registry.access.redhat.com/grafana:6.7.4: Error reading manifest 6.7.4 in registry.access.redhat.com/grafana: name unknown: Repo not found', '\t* Error initializing source docker://registry.redhat.io/grafana:6.7.4: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication', '\t* Error initializing source docker://grafana:6.7.4: Error reading manifest 6.7.4 in docker.io/library/grafana: errors:', 'denied: requested access to the resource is denied', 'unauthorized: authentication required'], 'item': 'quay.io/app-sre/grafana:6.7.4', 'ansible_loop_var': 'item'})

TASK [container-build : Fail the job if missing containers exists] *************
Tuesday 23 March 2021 05:35:39 +0000 (0:00:00.131) 0:46:07.409 *********
fatal: [undercloud]: FAILED! => {
    "changed": false
}

MSG:

Failed to pull following containers: ['quay.io/app-sre/grafana:6.7.4']
```

that image is maintained by different team with in organization which we donot have control.
In order to fix it, we need to pull container images from docker.io to quay.io/tripleo.

chandan kumar (chkumar246) wrote :

Here is the series of patches proposed to mitigate the issue; https://review.opendev.org/q/e0b0d859efb179c6868020b3fc10c2155ed96820

wes hayutin (weshayutin) on 2021-03-24
Changed in tripleo:
status: Triaged → Fix Released

This issue was fixed in the openstack/tripleo-common 13.2.0 release.

This issue was fixed in the openstack/tripleo-common 11.6.0 release.

This issue was fixed in the openstack/tripleo-common 12.4.4 release.

This issue was fixed in the openstack/tripleo-common 15.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers