[Regression] unsupported protocol scheme

[Jorge Niedbalski]

[Description]

Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic

[Affected Releases]

 containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
 containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
 containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x

[Impact]

Reported upstream: https://github.com/containerd/containerd/issues/4108

User Impact:

Since the Ubuntu bionic-updates bump of the version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
a regression was introduced.

The following endpoint description stopped working when scheduling pods with k8s 1.16-1.17 isn't longer working.

    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
      endpoint = ["niedbalski-bastion.cloud.sts:5000"]

As an example, creating a k8s pod defined as following:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
    - name: busybox
      image: niedbalski-bastion.cloud.sts:5000/busybox:latest
      command:
        - sleep
        - "3600"
  imagePullSecrets:
    - name: regcred
  restartPolicy: Always

Will fail in the current Bionic-updates version with the following error:

" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"

Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"

[Test Case]

1) Configure a private docker repository repository

2) Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/

3) Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)

Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.

[Possible workaround and solution]

As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit.

[Regression Potential]

** The change proposed on the SRU takes in consideration both cases
1) a endpoint without a schema 2) a endpoint with a schema.

1) worked in 1.2.6 as explained in the "Impact section" and stopped
being supported with the current Bionic version 1.3.3, 2) Should work
on both cases.

In neither case this should break existing endpoint definitions
now new deployments of containerd.

[Other Info]

** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue.

[Jorge Niedbalski (niedbalski)]

Attached is the debdiff that fixes this problem in bionic, it's a backport of commit [0]

[0] https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3

[Dariusz Gadomski (dgadomski)]

Hey Jorge,

I've checked the bionic patch and it looks ok. I just need to ask you for a couple of minor improvements:

1. Package version number should be rather 1.3.3-0ubuntu1~18.04.2 than 1.3.3-0ubuntu1~18.04.1ubuntu1
2. Patch commit id - the patch says 58769373c5509297749e9e12f0a99fb43653fa07 as the commit ID, but I was not able to find it in the upstream git. However, there's an identical with ID a022c218194c05449ad69b69c48fc6cac9d6f0b3. Can you please double check that commit ID?
3. Since it's an upstream patch it should contain the Origin: field, preferably pointing to the upstream commit (as required by [1]), e.g.Origin: upstream, https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3
4. Can you please update the bug description according to the SRU template [2]?

[1] https://dep-team.pages.debian.net/deps/dep3/
[2] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template.3C.3CAnchor.28srubug.29.3E.3E

Thanks!

[Jorge Niedbalski (niedbalski)]

Thank you for reviewing @dgadomski, Uploaded a debdiff that address your
comments and updated the SRU template accordingly.

[Robie Basak (racb)]

Hello Jorge, or anyone else affected,

Accepted containerd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.3.3-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Accepted. Thanks!

During SRU verification, please make sure to test the different scenarios mentioned in the Regression Potential section, and report what was tested and the results. Thanks again!

[Jorge Niedbalski (niedbalski)]

Deployed the following bundle: http://paste.ubuntu.com/p/tdjqQ3GjJ2/

Followed the reproducer steps.

### With current bionic-updates version 1.3.3-0ubuntu1~18.04.1, problem reproduced. #####

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl delete pod --all
pod "busybox" deleted
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ juju run --application kubernetes-worker "sudo grep -i niedbalski /etc/containerd/* | grep -i endpoint"
- Stdout: |
    /etc/containerd/config.toml: endpoint = ["niedbalski-bastion.cloud.sts:5000"]
  UnitId: kubernetes-worker/0
- ReturnCode: 1
  Stdout: ""
  UnitId: kubernetes-worker/1

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ juju run --application kubernetes-worker "sudo dpkg -l |grep containerd"
- Stdout: |
    ii containerd 1.3.3-0ubuntu1~18.04.1 amd64 daemon to control runC
  UnitId: kubernetes-worker/0

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl delete pod --all
pod "busybox" deleted
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
juju-3a79d2-00268738-4 Ready <none> 13d v1.16.8
juju-3a79d2-00268738-5 Ready,SchedulingDisabled <none> 13d v1.16.8
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
No resources found in default namespace.
(reverse-i-search)`deplo': kubectl edit ^Cployment microbot
(reverse-i-search)`appl': juju run --^Cplication kubernetes-worker "sudo dpkg -l |grep containerd"

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl apply -f busybox.yaml
pod/busybox created
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 0/1 ErrImagePull 0 3s 10.1.84.4 juju-3a79d2-00268738-4 <none> <none>

### With current -proposed version 1.3.3-0ubuntu1~18.04.2, problem is fixed. #####

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
juju-3a79d2-00268738-4 Ready <none> 13d v1.16.8
juju-3a79d2-00268738-5 Ready,SchedulingDisabled <none> 13d v1.16.8
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
No resources found in default namespace.
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl apply -f busybox.yaml
pod/busybox created
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
juju-3a79d2-00268738-4 Ready <none> 13d v1.16.8
juju-3a79d2-00268738-5 Ready,SchedulingDisabled <none> 13d v1.16.8
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
NAME READY STATUS RESTAR...

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.3.3-0ubuntu2

---------------
containerd (1.3.3-0ubuntu2) focal; urgency=high

  * d/p/0001-Improve-host-fallback-behaviour-in-docker-remote.patch:
    Fixes regression introduced in 1.3.3 update, LP: #1867398.

 -- Jorge Niedbalski <email address hidden> Thu, 26 Mar 2020 21:24:48 -0300

[Brian Murray (brian-murray)]

Hello Jorge, or anyone else affected,

Accepted containerd into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.3.3-0ubuntu1~19.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.3.3-0ubuntu1~18.04.2

---------------
containerd (1.3.3-0ubuntu1~18.04.2) bionic; urgency=high

  * d/p/0001-Improve-host-fallback-behaviour-in-docker-remote.patch:
    Fixes regression introduced in 1.3.3 update (LP: #1867398)

 -- Jorge Niedbalski <email address hidden> Thu, 19 Mar 2020 23:13:11 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for containerd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Jorge Niedbalski (niedbalski)]

The verification for eoan series has been performed using:

* Bundle http://paste.ubuntu.com/p/zBQfXWq77R/
* Following containerd config: https://paste.ubuntu.com/p/GDpjp2fd4t/

Annotations: Status: Pending
IP: 10.1.8.11
IPs:
  IP: 10.1.8.11
Containers:
  busybox:
    Container ID:
    Image: niedbalski-bastion.cloud.sts:5000/busybox:latest
    Image ID:
    Port: <none>
    Host Port: <none>
    Command:
      sleep
      3600
    State: Waiting
      Reason: ErrImagePull
    Ready: False
    Restart Count: 0
    Environment: <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vwm4f (ro)
Conditions:
  Type Status
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  default-token-vwm4f:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-vwm4f
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled <unknown> default-scheduler Successfully assigned default/busybox to juju-775746-00268738-1-4
  Warning FailedMount 17s kubelet, juju-775746-00268738-1-4 MountVolume.SetUp failed for volume "default-token-vwm4f" : failed to sync secret cache: timed out waiting for the condition
  Normal Pulling 11s kubelet, juju-775746-00268738-1-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
  Warning Failed 7s kubelet, juju-775746-00268738-1-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
  Warning Failed 7s kubelet, juju-775746-00268738-1-4 Error: ErrImagePull
  Normal BackOff 7s kubelet, juju-775746-00268738-1-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
  Warning Failed 7s kubelet, juju-775746-00268738-1-4 Error: ImagePullBackOff
---

After applying the -proposed version 1.3.3-0ubuntu1~19.10.2

ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl apply -f busybox.yaml
pod/busybox created
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
busybox 1/1 Running 0 4s 10.1.8.31 juju-775746-00268738-1-4 <none> <none>
ubuntu@niedbalski-bastion:~/stsstack-bundles/kubernetes/00268738$ juju run...

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.3.3-0ubuntu1~19.10.2

---------------
containerd (1.3.3-0ubuntu1~19.10.2) eoan; urgency=high

  * d/p/0001-Improve-host-fallback-behaviour-in-docker-remote.patch:
    Fixes regression introduced in 1.3.3 update, LP: #1867398.

 -- Jorge Niedbalski <email address hidden> Thu, 26 Mar 2020 21:46:43 -0300