Docker trusty image is very slow

Bug #1759440 reported by Will Storey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-images
New
Undecided
Unassigned
apt (Ubuntu)
Invalid
Undecided
Unassigned
Trusty
New
Undecided
Unassigned

Bug Description

I have strange slowness using the trusty image. I see no such slowness with the Artful image.

I expect there is something wrong with my system/configuration, but I'm at a loss to explain what is going on.

The difference appears to be IO related:

root@gisli:~# docker run -it ubuntu:trusty /bin/bash
root@5e4f73d56fa7:/# time apt-cache search iorate

real 0m8.666s
user 0m0.710s
sys 0m7.957s
root@5e4f73d56fa7:/# exit
root@gisli:~# docker run -it ubuntu:artful /bin/bash
root@5da05fb579d5:/# time apt-cache search iorate

real 0m0.049s
user 0m0.029s
sys 0m0.020s

Basic things feel very sluggish in it. apt-get update, installing packages, etc.

However dd output looks fine:

root@gisli:~# docker run -it ubuntu:trusty /bin/bash
root@87749c754dc9:/# dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.43277 s, 749 MB/s
root@gisli:~# docker run -it ubuntu:artful /bin/bash
root@7ab8fa9cd28a:/# dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc status=progress
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.37745 s, 780 MB/s

As do things like transferring data to the image via nc, running sha256sum on large files. I haven't been able to pin down what the problem is.

This is running on an Ubuntu Artful host using Docker 18.03.0-ce.

root@gisli:~# docker pull ubuntu:trusty
trusty: Pulling from library/ubuntu
Digest: sha256:ed49036f63459d6e5ed6c0f238f5e94c3a0c70d24727c793c48fded60f70aa96
Status: Image is up to date for ubuntu:trusty

Is there anything I could try? Or should this be expected?

Thank you!

Tags: docker
Revision history for this message
Will Storey (horgh) wrote :

I believe I figured out the problem. After running strace -f on while running apt-cache search, I saw a large amount of time spent doing this repeatedly:

[pid 202] getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
[pid 202] fcntl(13436, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 202] getrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
[pid 202] fcntl(13437, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)

(with increasing fd numbers to fcntl()). It looks like apt-cache is trying to set FD_CLOEXEC on every possible fd.

I found this Ask Ubuntu question: https://askubuntu.com/questions/477757/why-is-apt-cache-so-slow

From a comment on one of the answers I did this:

echo 'APT::Architectures { "amd64"; };' > /etc/apt/apt.conf.d/01arch

Afterwards strace no longer shows the above behaviour. And apt is much faster:

root@snorri:~# docker run -it ubuntu:trusty /bin/bash
aroot@96c54574923b:/# time apt-cache search test

real 0m10.200s
user 0m0.644s
sys 0m9.532s
root@96c54574923b:/# exit
root@snorri:~# docker run -it ubuntu:trusty /bin/bash
root@9f97376b8dd2:/# echo 'APT::Architectures { "amd64"; };' > /etc/apt/apt.conf.d/01arch
root@9f97376b8dd2:/# time apt-cache search test

real 0m0.054s
user 0m0.028s
sys 0m0.008s

Would this be something we could fix in the image somehow?

Revision history for this message
Will Storey (horgh) wrote :

Actually the fix does not appear complete. The fcntl() loop went away from the apt-cache search output, but I still see it when running apt-get install.

Revision history for this message
Will Storey (horgh) wrote :
Changed in apt (Ubuntu):
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.