[Regression] unsupported protocol scheme

Bug #1867398 reported by Jorge Niedbalski on 2020-03-13
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
containerd (Ubuntu)
Status tracked in Focal
Bionic
High
Jorge Niedbalski
Eoan
Undecided
Jorge Niedbalski
Focal
Undecided
Jorge Niedbalski

Bug Description

[Description]

Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic

[Affected Releases]

 containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
 containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
 containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x

[Impact]

Reported upstream: https://github.com/containerd/containerd/issues/4108

User Impact:

Since the Ubuntu bionic-updates bump of the version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
a regression was introduced.

The following endpoint description stopped working when scheduling pods with k8s 1.16-1.17 isn't longer working.

    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
      endpoint = ["niedbalski-bastion.cloud.sts:5000"]

As an example, creating a k8s pod defined as following:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
    - name: busybox
      image: niedbalski-bastion.cloud.sts:5000/busybox:latest
      command:
        - sleep
        - "3600"
  imagePullSecrets:
    - name: regcred
  restartPolicy: Always

Will fail in the current Bionic-updates version with the following error:

" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"

Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"

[Test Case]

1) Configure a private docker repository repository

2) Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/

3) Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)

Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.

[Possible workaround and solution]

As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit.

[Regression Potential]

** The change proposed on the SRU takes in consideration both cases
1) a endpoint without a schema 2) a endpoint with a schema.

1) worked in 1.2.6 as explained in the "Impact section" and stopped
being supported with the current Bionic version 1.3.3, 2) Should work
on both cases.

In neither case this should break existing endpoint definitions
now new deployments of containerd.

[Other Info]

** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue.

description: updated
Jorge Niedbalski (niedbalski) wrote :
Changed in containerd (Ubuntu Focal):
status: New → Fix Released
Changed in containerd (Ubuntu Eoan):
status: New → Fix Released
Jorge Niedbalski (niedbalski) wrote :

Attached is the debdiff that fixes this problem in bionic, it's a backport of commit [0]

[0] https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3

tags: added: sts-needs-sponsor
tags: added: sts-sponsor-dgadomski
removed: sts-needs-sponsor
tags: added: sts-needs-sponsor sts-sponsors
removed: sts-sponsor-dgadomski
tags: added: sts-sponsor-dgadomski
removed: sts-needs-sponsor sts-sponsors
Dariusz Gadomski (dgadomski) wrote :

Hey Jorge,

I've checked the bionic patch and it looks ok. I just need to ask you for a couple of minor improvements:

1. Package version number should be rather 1.3.3-0ubuntu1~18.04.2 than 1.3.3-0ubuntu1~18.04.1ubuntu1
2. Patch commit id - the patch says 58769373c5509297749e9e12f0a99fb43653fa07 as the commit ID, but I was not able to find it in the upstream git. However, there's an identical with ID a022c218194c05449ad69b69c48fc6cac9d6f0b3. Can you please double check that commit ID?
3. Since it's an upstream patch it should contain the Origin: field, preferably pointing to the upstream commit (as required by [1]), e.g.Origin: upstream, https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3
4. Can you please update the bug description according to the SRU template [2]?

[1] https://dep-team.pages.debian.net/deps/dep3/
[2] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template.3C.3CAnchor.28srubug.29.3E.3E

Thanks!

Jorge Niedbalski (niedbalski) wrote :
description: updated
Jorge Niedbalski (niedbalski) wrote :

Thank you for reviewing @dgadomski, Uploaded a debdiff that address your
comments and updated the SRU template accordingly.

description: updated
description: updated
Robie Basak (racb) on 2020-03-26
tags: added: regression-update

Hello Jorge, or anyone else affected,

Accepted containerd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.3.3-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in containerd (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Robie Basak (racb) wrote :

Accepted. Thanks!

During SRU verification, please make sure to test the different scenarios mentioned in the Regression Potential section, and report what was tested and the results. Thanks again!

Eric Desrochers (slashd) on 2020-03-26
Changed in containerd (Ubuntu Bionic):
assignee: nobody → Jorge Niedbalski (niedbalski)
importance: Undecided → High
Jorge Niedbalski (niedbalski) wrote :
Download full text (4.2 KiB)

Deployed the following bundle: http://paste.ubuntu.com/p/tdjqQ3GjJ2/

Followed the reproducer steps.

### With current bionic-updates version 1.3.3-0ubuntu1~18.04.1, problem reproduced. #####

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl delete pod --all
pod "busybox" deleted
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ juju run --application kubernetes-worker "sudo grep -i niedbalski /etc/containerd/* | grep -i endpoint"
- Stdout: |
    /etc/containerd/config.toml: endpoint = ["niedbalski-bastion.cloud.sts:5000"]
  UnitId: kubernetes-worker/0
- ReturnCode: 1
  Stdout: ""
  UnitId: kubernetes-worker/1

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ juju run --application kubernetes-worker "sudo dpkg -l |grep containerd"
- Stdout: |
    ii containerd 1.3.3-0ubuntu1~18.04.1 amd64 daemon to control runC
  UnitId: kubernetes-worker/0

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl delete pod --all
pod "busybox" deleted
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
juju-3a79d2-00268738-4 Ready <none> 13d v1.16.8
juju-3a79d2-00268738-5 Ready,SchedulingDisabled <none> 13d v1.16.8
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
No resources found in default namespace.
(reverse-i-search)`deplo': kubectl edit ^Cployment microbot
(reverse-i-search)`appl': juju run --^Cplication kubernetes-worker "sudo dpkg -l |grep containerd"

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl apply -f busybox.yaml
pod/busybox created
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 0/1 ErrImagePull 0 3s 10.1.84.4 juju-3a79d2-00268738-4 <none> <none>

### With current -proposed version 1.3.3-0ubuntu1~18.04.2, problem is fixed. #####

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
juju-3a79d2-00268738-4 Ready <none> 13d v1.16.8
juju-3a79d2-00268738-5 Ready,SchedulingDisabled <none> 13d v1.16.8
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
No resources found in default namespace.
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl apply -f busybox.yaml
pod/busybox created
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
juju-3a79d2-00268738-4 Ready <none> 13d v1.16.8
juju-3a79d2-00268738-5 Ready,SchedulingDisabled <none> 13d v1.16.8
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
NAME READY STATUS RESTAR...

Read more...

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Changed in containerd (Ubuntu Eoan):
status: Fix Released → Confirmed
Changed in containerd (Ubuntu Focal):
status: Fix Released → Confirmed
Changed in containerd (Ubuntu Eoan):
assignee: nobody → Jorge Niedbalski (niedbalski)
Changed in containerd (Ubuntu Focal):
assignee: nobody → Jorge Niedbalski (niedbalski)
Jorge Niedbalski (niedbalski) wrote :
Jorge Niedbalski (niedbalski) wrote :
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package containerd - 1.3.3-0ubuntu2

---------------
containerd (1.3.3-0ubuntu2) focal; urgency=high

  * d/p/0001-Improve-host-fallback-behaviour-in-docker-remote.patch:
    Fixes regression introduced in 1.3.3 update, LP: #1867398.

 -- Jorge Niedbalski <email address hidden> Thu, 26 Mar 2020 21:24:48 -0300

Changed in containerd (Ubuntu Focal):
status: Confirmed → Fix Released
Brian Murray (brian-murray) wrote :

Hello Jorge, or anyone else affected,

Accepted containerd into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.3.3-0ubuntu1~19.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in containerd (Ubuntu Eoan):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-eoan
removed: verification-done

This bug was fixed in the package containerd - 1.3.3-0ubuntu1~18.04.2

---------------
containerd (1.3.3-0ubuntu1~18.04.2) bionic; urgency=high

  * d/p/0001-Improve-host-fallback-behaviour-in-docker-remote.patch:
    Fixes regression introduced in 1.3.3 update (LP: #1867398)

 -- Jorge Niedbalski <email address hidden> Thu, 19 Mar 2020 23:13:11 -0300

Changed in containerd (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for containerd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers