[library] Docker loads images very slow

[Aleksandr Shaposhnikov]

During MOS masternode installation docker containers loaded very slow with tons of messages like this one:
EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts:

It looks like we are don't using recommended AUFS for docker. IMHO docker trying to using growing containers and remount them each and every operation that requires container to be extended.

The solution will be probably just to load aufs kernel module and configure docker to use aufs if it's configured other way.

[Matthew Mosesohn (raytrac3r)]

This really needs to be marked invalid. We can't change from devicemapper to AUFS. Both CentOS and Fedora kernels don't support it and the upstream EPEL/Fedora packages are built to use devicemapper. We're married to devicemapper as long as Fuel Master stays on CentOS and this can't be changed. The two things we can do, however, is upgrade Docker to 1.0 (which was met with resistance) and replace 2.6 kernel with 3.10 LTS kernel (which was also met with resistance). The only way to make this bug reachable is to rewrite Fuel Master to run on Ubuntu 14.04 LTS.

[Matthew Mosesohn (raytrac3r)]

AUFS is not possible without going through quite extreme efforts, but my alternative solution to this problem is already well underway. It involves tackling the problem at both the build phase and installation phase:
1 - during make iso, run docker daemon with a custom directory, socket, and pidfile (inside the build workspace). For build on ubuntu, set the storage backend to devicemapper so it's compatible with CentOS.
2 - don't save the images. just kill off the docker daemon
3 - compress the "docker" library directory and distribute in Fuel ISO (it's 13mb larger than the compressed saved images, but saves ~3 more minutes off build time)
4 - During provision, stop docker daemon, extract the archive into /var/lib/docker, then start docker. Images are fully unpacked and ready in ~2mins instead of 8-15 mins.

This doesn't solve any of the performance hits for actually starting containers up. That should be addressed in https://bugs.launchpad.net/fuel/+bug/1322641 , which is waiting on the new kernel firmware package.

Expect a patchset that addresses this in the very near future.

[Matthew Mosesohn (raytrac3r)]

This bug is stalled because Ubuntu 12.04.4 has devicemapper bugs that cause docker build to crash when using devicemapper as the storage backend. 14.04 seems to work, but we need to upgrade our build environment.

[Matthew Mosesohn (raytrac3r)]

Moving to 6.1. I have several patches ready to merge, but this is blocked because it requires docker upgrade and there are no Fuel Python resources to handle that task.

[Bogdan Dobrelya (bogdando)]

@Matthew, please update the status of this issue. What are current blockers? Should it be superseded by https://blueprints.launchpad.net/fuel/+spec/support-ubuntu-trusty?

[Matthew Mosesohn (raytrac3r)]

Docker upgrade improved performance, but docker squash still isn't in Fuel 6.1. It's a big change to add after feature freeze, so let's move to 7.0.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/177719

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthew Mosesohn (<email address hidden>) on branch: master
Review: https://review.openstack.org/177719
Reason: Abandoning because this fix introduces sporatic failures and doesn't save a significant amount of time on deploy

[Mykola Grygoriev (mgrygoriev)]

Tested on KVM VM with 4G RAM.

[root@fuel ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2014.2.2-7.0"
  api: "1.0"
  build_number: "37"
  build_id: "2015-07-14_14-09-16"
  nailgun_sha: "3416a04bbfb0f8766e610484914ccfe9f4a6b704"
  python-fuelclient_sha: "6e24282c7a7faa6c67f6a97aae78431bb5c4c414"
  astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
  fuel-library_sha: "be169b35505b184fa1018e14e4884f2117eba0c0"
  fuel-ostf_sha: "62785c16f8399f30526d24c52bb9ca23e1585bfb"
  fuelmain_sha: "de23bc4605c696938e383ed17c0ea78f26a918d2"

1. Use docker-io from nailgun repo
[root@fuel ~]# rpm -qa | grep docker-io
docker-io-1.4.1-1.mira1.x86_64
[root@fuel ~]# time docker load -i /var/www/nailgun/docker/images/fuel-images.tar

real 8m58.993s
user 0m0.213s
sys 0m2.444s

2. Update docker-io from epel
[root@fuel ~]# rpm -qa | grep docker-io
docker-io-1.5.0-1.el6.x86_64
[root@fuel ~]# time docker load -i /var/www/nailgun/docker/images/fuel-images.tar

real 1m41.904s
user 0m0.280s
sys 0m2.451s

[Aleksandr Didenko (adidenko)]

mos-linux, could you please update docker to 1.5.0 for Fuel-master node?

[Matthew Mosesohn (raytrac3r)]

We should move to docker 1.6.1 or newer (latest is 1.7.1) because of published security vulnerabilities:
http://seclists.org/oss-sec/2015/q2/389

[Alexei Sheplyakov (asheplyakov)]

Matthew,

> We should move to docker 1.6.1 or newer

Are you sure it can work with 2.6.32 kernel? What about performance?

[Alexei Sheplyakov (asheplyakov)]

Mykola,

> [root@fuel ~]# time docker load -i /var/www/nailgun/docker/images/fuel-images.tar

> real 1m41.904s
> user 0m0.280s
> sys 0m2.451s

Which kernel is that? What about the performance of the actual containers? Any kernel panics?

[Alexei Sheplyakov (asheplyakov)]

> mos-linux, could you please update docker to 1.5.0 for Fuel-master node?

Please file a bug according to https://github.com/asheplyakov/mos-package-customization-policy/blob/master/policy.rst

(And no, just because the containers load faster doesn't mean we should upgrade docker)

[Mykola Grygoriev (mgrygoriev)]

Alexei,

I just updated docker-io on master after CentOS provisioning to the latest one for CentOS6. So kernel is the same as we use from ISO. It is 2.6.32-504.1.3.el6.mos64.x86_64

[Matthew Mosesohn (raytrac3r)]

Any update here?

[Aleksander Mogylchenko (amogylchenko)]

We can not move to docker 1.7.x because of multiple issues with its integration with CentOS 6:
http://www.projectatomic.io/blog/2015/07/docker-centos-6-and-you/

Docker 1.6.x is delayed by this task currently:
https://bugs.launchpad.net/fuel/+bug/1476245

[Matthew Mosesohn (raytrac3r)]

This would be nice to fix in 7.0, but it looks like we are out of time.

[Dmitry Pyzhov (dpyzhov)]

Blocked by migration to CentOS 7

[Matthew Mosesohn (raytrac3r)]

Initial results for centos7 in a VM are very promising. It looks like image load is done in 1 minute on Docker 1.7.1 on CentOS 7.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/247020

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/247020
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9eefd81f2357e9ead2afb73b6405b4de3712edd6
Submitter: Jenkins
Branch: master

commit 9eefd81f2357e9ead2afb73b6405b4de3712edd6
Author: Dmitry Teselkin <email address hidden>
Date: Wed Nov 18 19:27:15 2015 +0300

    Use LVM volume group as backend for docker

    In order to improve docker container performance LVM volume
    group can be used as a backend for docker containers.

    Requires: https://review.openstack.org/#/c/247691

    Blueprint: master-on-centos7

    Is NOT compatible with CentOS6 master node

    Related-Bug: #1333458

    Change-Id: If66f5abf5e35e3cbb5cc1f0092104eea80c2ea80

[Aleksander Mogylchenko (amogylchenko)]

Fixed with migration to Centos7

[ElenaRossokhina (esolomina)]

Verified on #478
[root@nailgun ~]# rpm -qa | grep docker-i
fuel-docker-images-8.0.0-1.x86_64
[root@nailgun ~]# time docker load -i /var/www/nailgun/docker/images/fuel-images.tar

real 0m3.068s
user 0m0.198s
sys 0m0.993s
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "478"
  build_id: "478"

[Roman Podoliaka (rpodolyaka)]

Release notes:

LVM is now used as backend for Docker containers on Fuel master node, which greatly speeds up uploading of images to Docker.