OCI: issue when uploading multiarch images to DockerHub

[Lucas Kanashiro]

Hi,

I have noticed some weird behavior where images are built fine by LP in all architectures (some needed a retry) but when you go check on DockerHub there are tags missing for some of the architectures.

For instance, the Server team's Redis image is built from this OCI recipe:

https://launchpad.net/~canonical-server/ubuntu-server-oci/+oci/redis/+recipe/redis

as you can see it was built fine in amd64, arm64, ppc64el and s390x. If you check its DockerHub page s390x is missing:

https://hub.docker.com/repository/docker/squeakywheel/redis/tags?page=1

Another instance of this problem can be identified in our Grafana image which is built from this OCI recipe:

https://launchpad.net/~canonical-server/ubuntu-server-oci/+oci/grafana/+recipe/grafana

There are some old failures but the latest build in all architectures succeeded, but in its DockerHub page ppc64el and s390x are missing:

https://hub.docker.com/repository/docker/squeakywheel/grafana/tags?page=1

I faced a similar issue with Prometheus (it was a missing a tag for amd64), but then I triggered builds for all the architectures and then all of them were correctly uploaded. This bug was mentioned in LP #1904376. I am not sure what is happening.

[Thiago F. Pappacena (pappacena)]

The image on DockerHub was just replaced, but I managed to get the manifest details a bit before that. It should be helpful to debug what happened:

$ docker manifest inspect squeakywheel/redis:edge
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1453,
         "digest": "sha256:ebb25a3d1f55cc332ebe54af58f40b437696a766ca22a9c86b39b2fcc2e939f0",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1453,
         "digest": "sha256:f067b83f44dbc3db1a63165af0b20b59f594658fc7ae8343a4b4a8e28391a2c0",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1454,
         "digest": "sha256:4f13c671b4e2f0bdaf4596d1b8cfff04c25a120e149fa8d1e2abc721b42acff6",
         "platform": {
            "architecture": "ppc64el",
            "os": "linux"
         }
      }
   ]
}

[Thiago F. Pappacena (pappacena)]

And the builds that generated the previous manifest:

https://launchpad.net/~canonical-server/ubuntu-server-oci/+oci/redis/+recipe/redis/+build/491
https://launchpad.net/~canonical-server/ubuntu-server-oci/+oci/redis/+recipe/redis/+build/490
https://launchpad.net/~canonical-server/ubuntu-server-oci/+oci/redis/+recipe/redis/+build/489
https://launchpad.net/~canonical-server/ubuntu-server-oci/+oci/redis/+recipe/redis/+build/492

[Thiago F. Pappacena (pappacena)]

Lucas, I've checked your comment in #1904376. Are the architectures missing *only when you unselect them* when requesting a build, or is it happening at random too?

Not uploading architectures that you didn't request was intentional (it's a way to remove architectures, and it avoids confusion about what was used when building an image).

I'm not sure if we should include more opinions about changing this behavior, or just add a better description on what is going to happen at the "request a build" page.

[Lucas Kanashiro (lucaskanashiro)]

The scenario with an unexpected behavior for me was:

1- Build is requested for all architectures (in our case amd64, arm64, ppc64el, and s390x).
2- 2 builds passed (amd64, arm64) and 2 failed (ppc64el, s390x) due to some flaky tests failing during build.
3- The successful builds are uploaded to DockerHub.
4- I retrigger builds for the failing architectures.
5- They build fine and get uploaded.
6- The tags for the architectures initially uploaded are gone.

What I was discussing with my teammates is that LP may upload tags in batches and it desconsiders the previous ones on DockerHub. Is this correct? I would not expect the need to request new builds even for the successfully ones. Moreover, I am not sure how this works exactly but if there is an architecture specific bug I would not expect to trigger builds for all architectures because of this fix, the digest hash of the image will change and users might think that there is something new but actually there is not.

[Colin Watson (cjwatson)]

Would it help to have a specific facility for retrying OCI recipe builds? That way we'd be able to tell the difference between "retried some failures" and "specifically requested builds for fewer architectures".

[Lucas Kanashiro (lucaskanashiro)]

I think that would be something good to have. And also make the difference between those two options clear in the web interface, because at a first glance I was not considering this would be the default behavior.