tripleo-ci promotion failing on "pull ppc64le tagged containers"

[wes hayutin]

2020-01-29 20:55:12,019 p=30334 u=centos | TASK [containers-promote : Pull ppc64le tagged container images from trunk.registry.rdoproject.org registry] ***
2020-01-29 20:56:06,452 p=30334 u=centos | failed: [localhost] (item=aodh-api) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "index": 0, "item": "aodh-api",
 "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-api - code: None message: unexpected EOF"}
2020-01-29 20:57:00,705 p=30334 u=centos | failed: [localhost] (item=aodh-evaluator) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "index": 1, "item": "aodh
-evaluator", "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-evaluator - code: None message: unexpected EOF"}
2020-01-29 20:57:54,371 p=30334 u=centos | failed: [localhost] (item=aodh-listener) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "index": 2, "item": "aodh-
listener", "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-listener - code: None message: unexpected EOF"}
2020-01-29 20:58:47,712 p=30334 u=centos | failed: [localhost] (item=aodh-notifier) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "index": 3, "item": "aodh-
notifier", "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-notifier - code: None message: unexpected EOF"}
2020-01-29 20:59:40,940 p=30334 u=centos | failed: [localhost] (item=barbican-api) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "index": 4, "item": "barbic
an-api", "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/centos-binary-barbican-api - code: None message: unexpected EOF"}
^[OD2020-01-29 21:00:35,143 p=30334 u=centos | failed: [localhost] (item=barbican-keystone-listener) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "index":
5, "item": "barbican-keystone-listener", "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/centos-binary-barbican-keystone-listener - code: None message: unexpected EOF"}
2020-01-29 21:01:35,377 p=30334 u=centos | failed: [localhost] (item=barbican-worker) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": false, "index": 6, "item": "bar
bican-worker", "msg": "Error pulling trunk.registry.rdoproject.org/tripleomaster/centos-binary-barbican-worker - code: None message: unexpected EOF"}

Looking at the registry.. it looks like we have a match on:

 "name": "centos-binary-aodh-api:27e3154390c2690006d9fcc0656e923d9b812b67_1c9b2be6"
 "name": "centos-binary-aodh-api:27e3154390c2690006d9fcc0656e923d9b812b67_1c9b2be6_ppc64le",

    {
      "metadata": {
        "name": "centos-binary-aodh-api:27e3154390c2690006d9fcc0656e923d9b812b67_1c9b2be6",
        "namespace": "tripleomaster",
        "selfLink": "/oapi/v1/namespaces/tripleomaster/imagestreamtags/centos-binary-aodh-api%3A27e3154390c2690006d9fcc0656e923d9b812b67_1c9b2be6",
        "uid": "62788320-3cf5-11e8-93c1-fa163e7f8b33",
        "resourceVersion": "104247530",
        "creationTimestamp": "2020-01-29T08:55:14Z"
      },
      "tag": null,
      "generation": 1,
      "lookupPolicy": {
        "local": false
      },
      "image": {
        "metadata": {
          "name": "sha256:73323084422753b8983d7b2cee42590e7f7031249973e991f53038a6244bdedf",
          "creationTimestamp": null
        },
        "dockerImageReference": "docker-registry.default.svc:5000/tripleomaster/centos-binary-aodh-api@sha256:73323084422753b8983d7b2cee42590e7f7031249973e991f53038a6244bdedf",
        "dockerImageMetadata": {
          "kind": "DockerImage",
          "apiVersion": "1.0",
          "Id": "",
          "Created": null,
          "ContainerConfig": {}
        },
        "dockerImageMetadataVersion": "1.0",
        "dockerImageLayers": null
      }
    },
    {
      "metadata": {
        "name": "centos-binary-aodh-api:27e3154390c2690006d9fcc0656e923d9b812b67_1c9b2be6_ppc64le",
        "namespace": "tripleomaster",
        "selfLink": "/oapi/v1/namespaces/tripleomaster/imagestreamtags/centos-binary-aodh-api%3A27e3154390c2690006d9fcc0656e923d9b812b67_1c9b2be6_ppc64le",
        "uid": "62788320-3cf5-11e8-93c1-fa163e7f8b33",
        "resourceVersion": "104247530",
        "creationTimestamp": "2020-01-29T14:53:45Z"
      },
      "tag": null,
      "generation": 1,
      "lookupPolicy": {
        "local": false
      },

Code is here:

https://github.com/rdo-infra/ci-config/blob/79bcc9c64b82f1c6806139118e5b9a3663dcdb76/ci-scripts/container-push/roles/containers-promote/tasks/manifest-push.yml#L40-L49

[Marios Andreou (marios-b)]

hmm not sure if there is something wrong or we need some kind of special handline for the ppc tagged containers.

i can reproduce this using just podman... i went to [1] and found centos-binary-aodh-api:03e7a3a58585eb6d751cb6ea765973241f12f2f9_6e3b098e_ppc64le so its definitely there

attaching the trace here but basically pulling _x86_64 is OK but pulling _ppc64le gives the error:

podman pull trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-api:03e7a3a58585eb6d751cb6ea765973241f12f2f9_6e3b098e_ppc64le
==> Copying blob 1373db2e09f7 done
  unexpected EOF
Error: error pulling image "trunk.registry.rdoproject.org/tripleomaster/centos

BUT
podman pull trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-api:03e7a3a58585eb6d751cb6ea765973241f12f2f9_6e3b098e_x86_64
Trying to pull trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-api:03e7a3a58585eb6d751cb6ea765973241f12f2f9_6e3b098e_x86_64
==> Writing manifest to image destination
Storing signatures
bf4f198f197705611482a8a3dfc9acc1da7d914492a3a51335aa1fb27737bc4d
[marios@story1501rhel8containers ~]$

[Marios Andreou (marios-b)]

sorry in comment #1 i forgot to add [1]

[1] https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomaster/imagestreamtags/

[wes hayutin (weshayutin)]

 I think adding another test or two two to molecule may come out of this. Most importantly.. the promoter code should be a little more rigorous in checking the ppc containers and manifest but also fall back to just an x86_64 promotion if another arch fails.

[Marios Andreou (marios-b)]

ci-config master: Handle errors pulling ppc64le containers when creating manifests https://review.rdoproject.org/r/24771

[wes hayutin (weshayutin)]

https://review.rdoproject.org/r/#/c/24771/1/ci-scripts/container-push/roles/containers-promote/tasks/manifest-push.yml