docker pull fails in gate testing

Bug #1608171 reported by Takashi Kajinami
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
storlets
Invalid
Undecided
Unassigned

Bug Description

Docker pull failure often happens in gate testing, which results in functest failures.
Currently I can't find its root cause, but we should think what we can do to improve this instability.

Case 1:

2016-07-31 04:10:24.901815 | TASK [docker_base_jre_image : command] *****************************************
2016-07-31 04:10:34.599852 | failed: [127.0.0.1] (item=docker pull ubuntu:14.04) => {"changed": true, "cmd": ["docker", "pull", "ubuntu:14.04"], "delta": "0:00:09.458964", "end": "2016-07-31 04:10:34.551804", "failed": true, "item": "docker pull ubuntu:14.04", "rc": 1, "start": "2016-07-31 04:10:25.092840", "stderr": "Tag 14.04 not found in repository docker.io/library/ubuntu", "stdout": "14.04: Pulling from library/ubuntu\nb6b59ba2b1e5: Pulling fs layer\n401f6712926f: Pulling fs layer\n3667a1481f9c: Pulling fs layer\n2e82ca87ee2d: Pulling fs layer\n9eb73e5ef68f: Pulling fs layer\n9eb73e5ef68f: Verifying Checksum\n3667a1481f9c: Verifying Checksum\n3667a1481f9c: Download complete\n2e82ca87ee2d: Verifying Checksum\n2e82ca87ee2d: Download complete\nb6b59ba2b1e5: Verifying Checksum\nb6b59ba2b1e5: Download complete\nb6b59ba2b1e5: Pull complete\nPulling repository docker.io/library/ubuntu", "stdout_lines": ["14.04: Pulling from library/ubuntu", "b6b59ba2b1e5: Pulling fs layer", "401f6712926f: Pulling fs layer", "3667a1481f9c: Pulling fs layer", "2e82ca87ee2d: Pulling fs layer", "9eb73e5ef68f: Pulling fs layer", "9eb73e5ef68f: Verifying Checksum", "3667a1481f9c: Verifying Checksum", "3667a1481f9c: Download complete", "2e82ca87ee2d: Verifying Checksum", "2e82ca87ee2d: Download complete", "b6b59ba2b1e5: Verifying Checksum", "b6b59ba2b1e5: Download complete", "b6b59ba2b1e5: Pull complete", "Pulling repository docker.io/library/ubuntu"], "warnings": []}
2016-07-31 04:10:34.869674 | failed: [127.0.0.1] (item=docker tag ubuntu:14.04 127.0.0.1:5001/ubuntu_14.04) => {"changed": true, "cmd": ["docker", "tag", "ubuntu:14.04", "127.0.0.1:5001/ubuntu_14.04"], "delta": "0:00:00.044181", "end": "2016-07-31 04:10:34.821504", "failed": true, "item": "docker tag ubuntu:14.04 127.0.0.1:5001/ubuntu_14.04", "rc": 1, "start": "2016-07-31 04:10:34.777323", "stderr": "Error response from daemon: could not find image: no such id: ubuntu:14.04", "stdout": "", "stdout_lines": [], "warnings": []}
2016-07-31 04:10:35.114313 | failed: [127.0.0.1] (item=docker push 127.0.0.1:5001/ubuntu_14.04) => {"changed": true, "cmd": ["docker", "push", "127.0.0.1:5001/ubuntu_14.04"], "delta": "0:00:00.050354", "end": "2016-07-31 04:10:35.079133", "failed": true, "item": "docker push 127.0.0.1:5001/ubuntu_14.04", "rc": 1, "start": "2016-07-31 04:10:35.028779", "stderr": "Repository does not exist: 127.0.0.1:5001/ubuntu_14.04", "stdout": "The push refers to a repository [127.0.0.1:5001/ubuntu_14.04] (len: 0)", "stdout_lines": ["The push refers to a repository [127.0.0.1:5001/ubuntu_14.04] (len: 0)"], "warnings": []}

Case 2:

2016-07-31 16:47:28.691242 | TASK [docker_registry : command] ***********************************************
2016-07-31 16:47:30.052104 | fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["docker", "pull", "busybox"], "delta": "0:00:01.189576", "end": "2016-07-31 16:47:30.029021", "failed": true, "rc": 1, "start": "2016-07-31 16:47:28.839445", "stderr": "Tag latest not found in repository docker.io/library/busybox", "stdout": "Using default tag: latest\nlatest: Pulling from library/busybox\na5d4c53980c6: Pulling fs layer\nb41c5284db84: Pulling fs layer\na5d4c53980c6: Verifying Checksum\nPulling repository docker.io/library/busybox", "stdout_lines": ["Using default tag: latest", "latest: Pulling from library/busybox", "a5d4c53980c6: Pulling fs layer", "b41c5284db84: Pulling fs layer", "a5d4c53980c6: Verifying Checksum", "Pulling repository docker.io/library/busybox"], "warnings": []}

description: updated
Revision history for this message
Eran Rom (eranr) wrote :

One way to overcome this is to "docker save" the ubuntu image and make it part of the repo.
a gzipped tar of ubuntu 14.04 yields a 64MB file. Is that acceptable?
Also, do we know how often the image is updated if at all?

Revision history for this message
Eran Rom (eranr) wrote :

Kota had a better idea.
Get help from infra to create a vm image with docker ubuntu 14.04 image!

Revision history for this message
Takashi Kajinami (kajinamit) wrote :

writing down my previous thoughts

* Why do these docker pull fail?
** We can't find any 'error'
** Is there any timeout settings when executing command in ansible?
** Does it solve the problem to use docker ansible module?

On the other hand, it makes sense to me to include docker image on vm image, to avoid pulling it from the external repository.

Revision history for this message
Takashi Kajinami (kajinamit) wrote :

At first, we can ask infra team about their ideas about docker pull failures.
Also, we can refer Kolla project, which also uses docker container.

Revision history for this message
Takashi Kajinami (kajinamit) wrote (last edit ):

We no longer see this problem so I'll close this as Invalid. If we again hit the issue, then we can revisit any available solution.

Changed in storlets:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.