[UBUNTU 22.04] Podman play kube: brings in unwanted (untrusted) k8s pause

[bugproxy]

SRU Justification:
------------------

[ Impact ]

 * Pods no longer need k8s/pause,
   but podman play kube still fetches it.

 * That can be seen as a security problem,
   since podman tries to pull this untrusted image.

 * https://github.com/containers/podman/issues/12254

[ Test Plan ]

 * Like described on upstream issue:

 * $ bin/podman images
   REPOSITORY TAG IMAGE ID CREATED SIZE
   $ printf "apiVersion: v1\nkind: Pod\nmetadata:\n name: foo\n" | env \
     CONTAINER_HELPER_PAUSE_PAUSE=bin/pause bin/podman play kube -
   Pod:
   738622313f1f37b32814664a8dc86d2df36dd5036e661e1d15623686e26c2616

 * $ bin/podman images
   REPOSITORY TAG IMAGE ID CREATED SIZE
   localhost/podman-pause 4.0.0-dev-1636547894 99f3b83b4245 5 seconds ago 1.65 MB
   k8s.gcr.io/pause 3.5 ed210e3e4a5b 7 months ago 690 kB

 * It's expected to see localhost/podman-pause, but not the k8s one.

[ Where problems could occur ]

 * Problems could occur if someone makes accidentally use of the image.
   which should't be the case.

 * Or if there is no local podman-pause or it doesn't built properly.

 * In case of issues with the modification in func pullImage(*),
   the general pull of images could be harmed.

[ Other Info ]

 * The PR 12280 fixes this with commits f517510bc8c11f6ba3145facc10ce351084a4ce4.
   This commit is upstream since 4.0.0.

 * Since there is a libpod 4.3.1+ds1-5 in lunar-proposed,
   lunar is (soon) not affected.
__________

There is a security problem (podman would try to pull an untrusted image, the pause image) that needs to be fixed in Ubuntu 22.04.

The required fix is described & provided here:
https://github.com/containers/podman/issues/12254

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-02-21 07:56 EDT-------
This is a more detailed description of the problem, including SRU relevant information

SRU Justification:
==================

[Problem Statement]
* For IBM hyper protect virtual servers v2 (aka HPCR) we plan to leverage the `podman play kube` functionality to bring up OCI containers based on k8s pod definitions in a secure enclave
* since this will be running in a secure enclave, our customers can control network connectivity, in particular connectivity to the container registries needed to pull images
* the podman version available in Ubuntu 22.04 (podman v3.4.4) automatically pulls a `pause` image from `k8s.gcr.io/pause`. This has the disadvantage that connectivity is needed to `k8s.gcr.io` and in addition this pull in a potentially untrusted image
* this behaviour has been fixed in a later version of podman via https://github.com/containers/podman/issues/12254 in favour of pre-packaging a podman specific version of a pause container

[Impact]
* with the current behaviour of podman HPCR cannot run in a private-only network configuration without access to `k8s.gcr.io`. Mitigation: HPCR could try to pre-package a copy of k8s.gcr.io/pause
* HPCR relies on k8s.gcr.io/pause but we do not have open source approval for that container

[Test Plan]
* start any k8s payload using `podman play kube`. Then verify that `k8s.gcr.io/pause` is not part of the running containers

[Where problems could occur]
* the `k8s.gcr.io/pause` container is only needed to keep the cluster up, afaik there is no direct dependency on that container name by any other container or component

[Seth Arnold (seth-arnold)]

Hello Frank, why is this bug private?

Is there a CVE assigned to this issue?

Thanks

[Frank Heimes (fheimes)]

Hi Seth, because a security concern was raised, I wanted to mark it as a Security related bug.
But I agree, it should/could have been Public Security (I guess I picked the wrong type).
And no, I'm not aware about an assigned CVE for this.

[Frank Heimes (fheimes)]

Looks like f517510bc "play kube: don't force-pull infra image" got upstream accepted with (libpod) v4.0.0.
So this affects jammy and kinetic:
 libpod | 3.4.4+ds1-1ubuntu1 | jammy/universe | source
 libpod | 3.4.4+ds1-1ubuntu1 | kinetic/universe | source
 libpod | 3.4.4+ds1-1ubuntu1 | lunar/universe | source
 libpod | 4.3.1+ds1-5 | lunar-proposed/universe | source
With the updated version in -proposed, lunar is safe.

I've created a quilt patch based on f517510bc and did a test build in PPA for jammy:
https://launchpad.net/~fheimes/+archive/ubuntu/lp2007972

[Marc Deslauriers (mdeslaur)]

The debdiff in comment #5 looks good, but the same version of the libpod package is present in both jammy and kinetic, so fixing this will require fixing both releases.

Could you please attach debdiffs for both releases, with appropriate version numbers, such as 3.4.4+ds1-1ubuntu1.22.04.1 and 3.4.4+ds1-1ubuntu1.1.22.10.1?

Thanks!

[Frank Heimes (fheimes)]

I regenerated the debdiff(s) and attach them here one by one.
But I believe for security updates the kinetic version should be "3.4.4+ds1-1ubuntu1.22.10.1" rather than "3.4.4+ds1-1ubuntu1.1.22.10.1" - if I am not mistaken.
(Please notice the additional but in my opinion obsolete ".1" right from the term "ubuntu" in the kinetic version string.)

With "3.4.4+ds1-1ubuntu1.22.10.1" the kinetic versioning also seems to be conform to the jammy version: "3.4.4+ds1-1ubuntu1.22.04.1".
(If I'm wrong, please let me know and I change it back.)

[Leonidas S. Barbosa (leosilvab)]

Hi @frank,

I was trying to build it for kinetic locally and in the security-proposed ppa, and I'm facing this issue: https://launchpadlibrarian.net/672043238/buildlog_ubuntu-kinetic-amd64.libpod_3.4.4+ds1-1ubuntu1.22.10.1~test1_BUILDING.txt.gz

If you have any clue. I did try using -updates, but some issue/error.

[Leonidas S. Barbosa (leosilvab)]

If you have any special steps to build it, could you share so I can give it a try here.

[Frank Heimes (fheimes)]

Hi @leonidas, well, I usually do PPA builds only, so nothing special.
Here I personally think that libpod never really built on kinetic, I see the same version in jammy and believe it just came from jammy. To me it seems to fail to build due to issues in the slightly different dependencies, compared to jammy (libpod itself is exactly the same in J and K).

But the question here is IF we really still need to patch kinetic:
- because kinetic will hit it's EOL already next month
- and the bug itself was mainly opened against jammy

(So I am personally fine if we can set kinetic to Won't Fix, since it's close to its EOL, and just focus on getting it fixed in jammy.)

[Leonidas S. Barbosa (leosilvab)]

hi @frank, ok, it sounds good to me skip Kinetic so.

Thanks!

[Frank Heimes (fheimes)]

So we agreed to skip kinetic, since it's proximity to it's EOL in about a month.
Updating the 'affecting kinetic' entry to Won't Fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libpod - 3.4.4+ds1-1ubuntu1.22.04.1

---------------
libpod (3.4.4+ds1-1ubuntu1.22.04.1) jammy-security; urgency=medium

  * Add d/p/lp-2007972-play-kube-don-t-force-pull-infra-image.patch
    to prevent play kube from unwanted force-pull of infra image
    and with that unwanted (untrusted) k8s pause (LP: #2007972).

 -- Frank Heimes <email address hidden> Wed, 22 Feb 2023 10:46:22 +0100

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-06-19 07:52 EDT-------
Thanks for all your work.
With the fix being released to Jammy, we can close this BZ/LP.

==> Changing status to CLOSED.

[Jeremy Stanley (fungi)]

It appears the Jammy backport may have led to the observed regression in bug 2024394 by inadvertently requiring a newer golang-github-containernetworking-plugins than is available currently in Jammy.