containerd upgrade to 1.6.x for focal and jammy

[Adam Dyess]

Starting with kubernetes 1.26, containerd 1.5.9 is out of support [0]

Could there be a containerd version built for both focal and jammy?

just for testing sake, I pulled the latest amd release [1] onto a focal machine and installed it without discovering anything noticeable. I was able to start new workloads which all started just fine. But within a few hours, there's been a 1.6.10 release so that could be another starting point.

Without a new release, Charmed-Kubernetes will not be able to start with kubelet 1.26.0 due to the following error:

Nov 14 15:50:52 juju-18daae-0 kubelet.daemon[182698]: E1114 15:50:52.511970 182698 run.go:74] "command failed" err="failed to run Kubelet: validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

[0] https://containerd.io/releases/#kubernetes-support
[1] https://github.com/containerd/containerd/releases/download/v1.6.9/containerd-1.6.9-linux-amd64.tar.gz

[Lucas Kanashiro (lucaskanashiro)]

Thanks for letting us know Adam. This is on the Server team radar.

[Chris Sanders (chris.sanders)]

Confirming I've hit this today with a kubadm based upgrade. Upgrading from 1.25 to 1.26 brings the kubelet down with the same error message as above.

For anyone else hitting this I've worked around it with a downgrade of the Kubelet until containerd is upgraded to 1.6+.

sudo apt install -y --allow-downgrades --allow-change-held-packages kubelet=1.25.5-00

[Adam Dyess (addyess)]

another work-around employed in charmed-kubernetes is to upgrade containerd after apt installation with the upstream release (1.6.8 for example) of containerd.

This script builds a multi-arch charm resource: https://github.com/charmed-kubernetes/charm-containerd/blob/main/build-resources.sh

which is used by the containerd charm to install the correct binaries atop the apt installation.

[Neil Wilson (neil-aldur)]

Any chance of bumping this up the priority. containerd 1.5 is EOL, which means we have an LTS that can no longer support the latest version of Kubernetes.

[Lucas Kanashiro (lucaskanashiro)]

The version 1.6.12 has been backported from Lunar, the packages for all supported releases were already uploaded to the -unapproved queue and we are waiting for the SRU team.

[Andreas Hasenack (ahasenack)]

I'm reviewing this today as part of my SRU shift.

[Andreas Hasenack (ahasenack)]

Hello Adam, or anyone else affected,

Accepted containerd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.6.12-0ubuntu1~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Adam, or anyone else affected,

Accepted containerd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.6.12-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Adam, or anyone else affected,

Accepted containerd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.6.12-0ubuntu1~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Neil Wilson (neil-aldur)]

On Jammy, without proposed kubernetes install via kubeadm v1.26.1 fails with

[ERROR CRI]: container runtime is not running: output: time="2023-02-01T16:31:16Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

with proposed the installation via kubeadm completes successfully and the nodes are available.

 kubectl get nodes
NAME STATUS ROLES AGE VERSION
srv-2qjps Ready control-plane 30s v1.26.1

[Neil Wilson (neil-aldur)]

On Focal, without proposed, kubernetes install with kubeadm 1.26.1 fails:

[ERROR CRI]: container runtime is not running: output: time="2023-02-02T09:46:28Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

with proposed the installation via kubeadm completes successfully and the nodes are available:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
srv-rsxxm Ready control-plane 26s v1.26.1

[Neil Wilson (neil-aldur)]

On Bionic, without proposed, kubernetes install with kubeadm 1.26.1 fails:

[ERROR CRI]: container runtime is not running: output: time="2023-02-02T10:00:08Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

with proposed the installation via kubeadm completes successfully and the nodes are available:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
srv-4utbs Ready control-plane 24s v1.26.1

[Andreas Hasenack (ahasenack)]

I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions.

There is no SRU freeze ongoing at the moment.

There is no halted phasing on the previous update.

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.6.12-0ubuntu1~22.04.1

---------------
containerd (1.6.12-0ubuntu1~22.04.1) jammy; urgency=medium

  * Backport version 1.6.12-0ubuntu1 from Lunar (LP: #1996909, #1996534).

 -- Lucas Kanashiro <email address hidden> Tue, 03 Jan 2023 17:58:41 -0300

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for containerd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.6.12-0ubuntu1~20.04.1

---------------
containerd (1.6.12-0ubuntu1~20.04.1) focal; urgency=medium

  * Backport version 1.6.12-0ubuntu1 from Lunar (LP: #1996909, #1996534).
    - d/rules: set GO111MODULE to off.
    - d/control: b-d on golang-1.18-go instead of golang-go.
    - d/rules: build with Golang 1.18.
    - d/p/do-not-rebuild-manpage-during-installation.patch: avoid running go
      script during installation.

 -- Lucas Kanashiro <email address hidden> Thu, 17 Nov 2022 14:38:11 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.6.12-0ubuntu1~18.04.1

---------------
containerd (1.6.12-0ubuntu1~18.04.1) bionic; urgency=medium

  * Backport version 1.6.12-0ubuntu1 from Lunar (LP: #1996909, #1996534).
    - d/rules: set GO111MODULE to off.
    - d/control: b-d on golang-1.18-go instead of golang-go.
    - d/rules: build with Golang 1.18.
    - d/control: do not b-d on libbtrfs-dev, it is not available in Bionic.
    - d/p/do-not-rebuild-manpage-during-installation.patch: avoid running go
      script during installation.

 -- Lucas Kanashiro <email address hidden> Thu, 17 Nov 2022 15:43:46 -0300