| 2020-03-13 22:32:11 |
Jorge Niedbalski |
bug |
|
|
added bug |
| 2020-03-13 22:32:21 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Bionic |
|
| 2020-03-13 22:32:21 |
Jorge Niedbalski |
bug task added |
|
containerd (Ubuntu Bionic) |
|
| 2020-03-13 22:33:54 |
Jorge Niedbalski |
description |
[Environment]
Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic
[Description]
Reported upstream: https://github.com/containerd/containerd/issues/4108
The bump of to version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
Caused a regression.
The following endpoint description works with containerd 1.2.X without defining
a protocol scheme. (/etc/containerd/config.toml).
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
endpoint = ["niedbalski-bastion.cloud.sts:5000"]
This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't
works using the same registry mirror definition.
The pod definition is:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: niedbalski-bastion.cloud.sts:5000/busybox:latest
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
restartPolicy: Always
New pods fail with the following error:
" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
[Steps to reproduce]
Configure a private docker repository repository
Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/
Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)
Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.
[Possible workaround and solution]
As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit. |
[Environment]
Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic
[Affected Releases]
containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x
[Description]
Reported upstream: https://github.com/containerd/containerd/issues/4108
The bump of to version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
Caused a regression.
The following endpoint description works with containerd 1.2.X without defining
a protocol scheme. (/etc/containerd/config.toml).
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
endpoint = ["niedbalski-bastion.cloud.sts:5000"]
This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't
works using the same registry mirror definition.
The pod definition is:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: niedbalski-bastion.cloud.sts:5000/busybox:latest
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
restartPolicy: Always
New pods fail with the following error:
" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
[Steps to reproduce]
Configure a private docker repository repository
Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/
Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)
Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.
[Possible workaround and solution]
As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit. |
|
| 2020-03-13 22:34:03 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Focal |
|
| 2020-03-13 22:34:03 |
Jorge Niedbalski |
bug task added |
|
containerd (Ubuntu Focal) |
|
| 2020-03-13 22:34:03 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Eoan |
|
| 2020-03-13 22:34:03 |
Jorge Niedbalski |
bug task added |
|
containerd (Ubuntu Eoan) |
|
| 2020-03-14 07:51:02 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2020-03-16 05:20:24 |
Tianon Gravi |
bug |
|
|
added subscriber Tianon Gravi |
| 2020-03-20 02:21:20 |
Jorge Niedbalski |
attachment added |
|
Bionic debdiff https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1867398/+attachment/5339108/+files/fix-1867398-bionic.debdiff |
|
| 2020-03-20 02:21:56 |
Jorge Niedbalski |
containerd (Ubuntu Focal): status |
New |
Fix Released |
|
| 2020-03-20 02:22:00 |
Jorge Niedbalski |
containerd (Ubuntu Eoan): status |
New |
Fix Released |
|
| 2020-03-20 02:22:40 |
Jorge Niedbalski |
tags |
|
sts-needs-sponsor |
|
| 2020-03-20 02:23:09 |
Jorge Niedbalski |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2020-03-20 07:52:49 |
Dariusz Gadomski |
tags |
sts-needs-sponsor |
sts-sponsor-dgadomski |
|
| 2020-03-20 07:54:06 |
Dariusz Gadomski |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2020-03-20 07:54:16 |
Dariusz Gadomski |
bug |
|
|
added subscriber STS Sponsors |
| 2020-03-20 13:20:21 |
Jorge Niedbalski |
tags |
sts-sponsor-dgadomski |
sts-needs-sponsor sts-sponsors |
|
| 2020-03-20 14:28:31 |
Dariusz Gadomski |
tags |
sts-needs-sponsor sts-sponsors |
sts-sponsor-dgadomski |
|
| 2020-03-23 13:30:28 |
Jorge Niedbalski |
description |
[Environment]
Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic
[Affected Releases]
containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x
[Description]
Reported upstream: https://github.com/containerd/containerd/issues/4108
The bump of to version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
Caused a regression.
The following endpoint description works with containerd 1.2.X without defining
a protocol scheme. (/etc/containerd/config.toml).
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
endpoint = ["niedbalski-bastion.cloud.sts:5000"]
This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't
works using the same registry mirror definition.
The pod definition is:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: niedbalski-bastion.cloud.sts:5000/busybox:latest
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
restartPolicy: Always
New pods fail with the following error:
" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
[Steps to reproduce]
Configure a private docker repository repository
Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/
Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)
Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.
[Possible workaround and solution]
As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit. |
[Description]
Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic
[Affected Releases]
containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x
[Impact]
Reported upstream: https://github.com/containerd/containerd/issues/4108
The bump of to version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
Caused a regression.
The following endpoint description works with containerd 1.2.X without defining
a protocol scheme. (/etc/containerd/config.toml).
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
endpoint = ["niedbalski-bastion.cloud.sts:5000"]
This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't
works using the same registry mirror definition.
The pod definition is:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: niedbalski-bastion.cloud.sts:5000/busybox:latest
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
restartPolicy: Always
New pods fail with the following error:
" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
[Test Case]
Configure a private docker repository repository
Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/
Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)
Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.
[Possible workaround and solution]
As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit.
[Regression Potential]
** Not identified yet any regression potential, this functionality fixes an existing regression introduced in the latest update.
[Other Info]
** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue. |
|
| 2020-03-23 13:31:19 |
Jorge Niedbalski |
attachment added |
|
Patch for bionic - Iter 2 https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1867398/+attachment/5340471/+files/fix-1867398-bionic.debdiff |
|
| 2020-03-24 21:07:16 |
George Kraft |
bug |
|
|
added subscriber George Kraft |
| 2020-03-26 01:42:38 |
Jorge Niedbalski |
description |
[Description]
Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic
[Affected Releases]
containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x
[Impact]
Reported upstream: https://github.com/containerd/containerd/issues/4108
The bump of to version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
Caused a regression.
The following endpoint description works with containerd 1.2.X without defining
a protocol scheme. (/etc/containerd/config.toml).
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
endpoint = ["niedbalski-bastion.cloud.sts:5000"]
This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't
works using the same registry mirror definition.
The pod definition is:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: niedbalski-bastion.cloud.sts:5000/busybox:latest
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
restartPolicy: Always
New pods fail with the following error:
" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
[Test Case]
Configure a private docker repository repository
Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/
Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)
Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.
[Possible workaround and solution]
As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit.
[Regression Potential]
** Not identified yet any regression potential, this functionality fixes an existing regression introduced in the latest update.
[Other Info]
** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue. |
[Description]
Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic
[Affected Releases]
containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x
[Impact]
Reported upstream: https://github.com/containerd/containerd/issues/4108
User Impact:
Since the Ubuntu bionic-updates bump of the version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
a regression was introduced.
The following endpoint description stopped working when scheduling pods with k8s 1.16-1.17 isn't longer working.
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
endpoint = ["niedbalski-bastion.cloud.sts:5000"]
As an example, A pod defined as following:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: niedbalski-bastion.cloud.sts:5000/busybox:latest
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
restartPolicy: Always
Will fail with the following error:
" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
[Test Case]
1) Configure a private docker repository repository
2) Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/
3) Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)
Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.
[Possible workaround and solution]
As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit.
[Regression Potential]
** The change proposed on the SRU takes in consideration both cases
1) a endpoint without a schema 2) a endpoint with a schema.
1) worked in 1.2.6 as explained in the "Impact section" and stopped
being supported with the current Bionic version 1.3.3, 2) Should work
on both cases.
In neither case this should break existing endpoint definitions
now new deployments of containerd.
[Other Info]
** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue. |
|
| 2020-03-26 01:43:17 |
Jorge Niedbalski |
description |
[Description]
Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic
[Affected Releases]
containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x
[Impact]
Reported upstream: https://github.com/containerd/containerd/issues/4108
User Impact:
Since the Ubuntu bionic-updates bump of the version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
a regression was introduced.
The following endpoint description stopped working when scheduling pods with k8s 1.16-1.17 isn't longer working.
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
endpoint = ["niedbalski-bastion.cloud.sts:5000"]
As an example, A pod defined as following:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: niedbalski-bastion.cloud.sts:5000/busybox:latest
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
restartPolicy: Always
Will fail with the following error:
" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
[Test Case]
1) Configure a private docker repository repository
2) Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/
3) Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)
Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.
[Possible workaround and solution]
As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit.
[Regression Potential]
** The change proposed on the SRU takes in consideration both cases
1) a endpoint without a schema 2) a endpoint with a schema.
1) worked in 1.2.6 as explained in the "Impact section" and stopped
being supported with the current Bionic version 1.3.3, 2) Should work
on both cases.
In neither case this should break existing endpoint definitions
now new deployments of containerd.
[Other Info]
** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue. |
[Description]
Kubernetes 1.16.17
Containerd 1.3.3
Ubuntu Bionic
[Affected Releases]
containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x
[Impact]
Reported upstream: https://github.com/containerd/containerd/issues/4108
User Impact:
Since the Ubuntu bionic-updates bump of the version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841
a regression was introduced.
The following endpoint description stopped working when scheduling pods with k8s 1.16-1.17 isn't longer working.
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"]
endpoint = ["niedbalski-bastion.cloud.sts:5000"]
As an example, creating a k8s pod defined as following:
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: niedbalski-bastion.cloud.sts:5000/busybox:latest
command:
- sleep
- "3600"
imagePullSecrets:
- name: regcred
restartPolicy: Always
Will fail in the current Bionic-updates version with the following error:
" failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4
Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts"
Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull
Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff
Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest"
[Test Case]
1) Configure a private docker repository repository
2) Modify the containerd registry mirror config as follows:
** http://paste.ubuntu.com/p/yP63WMkVT6/
3) Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/)
Status of the scheduled pod should be ImagePullBackOff
and the before mentioned error should be raised.
[Possible workaround and solution]
As a workaround change the endpoint to support the scheme (https://)
Provide a fallback mechanism for URL parsing validation to fallback to http or https.
I suspect that this change introduced on 1.3.3 through
0b29c9c) may be the offending commit.
[Regression Potential]
** The change proposed on the SRU takes in consideration both cases
1) a endpoint without a schema 2) a endpoint with a schema.
1) worked in 1.2.6 as explained in the "Impact section" and stopped
being supported with the current Bionic version 1.3.3, 2) Should work
on both cases.
In neither case this should break existing endpoint definitions
now new deployments of containerd.
[Other Info]
** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue. |
|
| 2020-03-26 15:29:48 |
Robie Basak |
tags |
sts-sponsor-dgadomski |
regression-update sts-sponsor-dgadomski |
|
| 2020-03-26 15:55:40 |
Robie Basak |
containerd (Ubuntu Bionic): status |
New |
Fix Committed |
|
| 2020-03-26 15:55:42 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-03-26 15:55:44 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2020-03-26 15:55:47 |
Robie Basak |
tags |
regression-update sts-sponsor-dgadomski |
regression-update sts-sponsor-dgadomski verification-needed verification-needed-bionic |
|
| 2020-03-26 17:04:33 |
Eric Desrochers |
containerd (Ubuntu Bionic): assignee |
|
Jorge Niedbalski (niedbalski) |
|
| 2020-03-26 17:04:39 |
Eric Desrochers |
containerd (Ubuntu Bionic): importance |
Undecided |
High |
|
| 2020-03-26 17:04:50 |
Eric Desrochers |
removed subscriber STS Sponsors |
|
|
|
| 2020-03-26 17:04:54 |
Eric Desrochers |
bug |
|
|
added subscriber Eric Desrochers |
| 2020-03-26 17:05:03 |
Eric Desrochers |
bug |
|
|
added subscriber Dariusz Gadomski |
| 2020-03-26 22:09:18 |
Jorge Niedbalski |
tags |
regression-update sts-sponsor-dgadomski verification-needed verification-needed-bionic |
regression-update sts-sponsor-dgadomski verification-done verification-done-bionic |
|
| 2020-03-26 23:06:27 |
Jorge Niedbalski |
containerd (Ubuntu Eoan): status |
Fix Released |
Confirmed |
|
| 2020-03-26 23:06:31 |
Jorge Niedbalski |
containerd (Ubuntu Focal): status |
Fix Released |
Confirmed |
|
| 2020-03-26 23:07:03 |
Jorge Niedbalski |
containerd (Ubuntu Eoan): assignee |
|
Jorge Niedbalski (niedbalski) |
|
| 2020-03-26 23:07:05 |
Jorge Niedbalski |
containerd (Ubuntu Focal): assignee |
|
Jorge Niedbalski (niedbalski) |
|
| 2020-03-27 01:01:56 |
Jorge Niedbalski |
attachment added |
|
Backport of Bug fix for eoan https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1867398/+attachment/5341942/+files/fix-1867398-eoan.debdiff |
|
| 2020-03-27 01:02:20 |
Jorge Niedbalski |
attachment added |
|
Backport of Bug fix for Focal https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1867398/+attachment/5341943/+files/fix-1867398-focal.debdiff |
|
| 2020-03-27 03:26:28 |
Launchpad Janitor |
containerd (Ubuntu Focal): status |
Confirmed |
Fix Released |
|
| 2020-03-31 20:45:58 |
Brian Murray |
containerd (Ubuntu Eoan): status |
Confirmed |
Fix Committed |
|
| 2020-03-31 20:46:10 |
Brian Murray |
tags |
regression-update sts-sponsor-dgadomski verification-done verification-done-bionic |
regression-update sts-sponsor-dgadomski verification-done-bionic verification-needed verification-needed-eoan |
|
| 2020-04-02 18:21:38 |
Launchpad Janitor |
containerd (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2020-04-02 18:21:43 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2020-04-13 22:28:33 |
Jorge Niedbalski |
tags |
regression-update sts-sponsor-dgadomski verification-done-bionic verification-needed verification-needed-eoan |
regression-update sts-sponsor-dgadomski verification-done verification-done-bionic verification-done-eoan |
|
| 2020-04-14 08:55:16 |
Launchpad Janitor |
containerd (Ubuntu Eoan): status |
Fix Committed |
Fix Released |
|
