IBP images build is way too slow

[Vladimir Kozhukalov]

Due to a very suboptimal IO layering building the target OS (Ubuntu)
IBP images takes an unreasonably long time (more than an hour!) if
the master node runs in the KVM VM.

fuel_agent (which builds Ubuntu images) runs in the mcollective
container. A container filesystem is backed by dm-snapshot which
is known to have a poor write performance, especially for large
files (like OS images). To add an insult to injury dm-snapshot
storage is a loopback file residing in /var/lib/docker.
Thus the data written to the OS image being built goes through
a number of layers:

ext4 -> loop -> dm-snapshot (/var/lib/docker) -> loop ->
     -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
     -> hard drive

If the master node runs on a VM rather than a physcial hardware
the above chain gets even more complicated (and write performance
gets really awful)

ext4 -> loop -> dm-snapshot (/var/lib/docker) -> loop ->
     -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
     -> virtual hard drive (qemu) -> qcow2 -> ext4 (host)

We need to remove the extra layers to make IO performance reasonable.

[Alexei Sheplyakov (asheplyakov)]

Actually it's better to add a dedicated raw or lvm2 backed virtual hard drive for building images.
This way we can avoid the overhead due to qcow2. Using a host LV also avoids overhead due to fsync'img the image (by the host kernel)

[Aleksander Mogylchenko (amogylchenko)]

It should be noted that using cache='unsafe' is indeed unsafe and can lead to data losses. Users should be aware of possible risks.

[Alexei Sheplyakov (asheplyakov)]

The suggested "solution" is actually a way to irreversibly corrupt filesystems if qemu crashes,
or the kernel (either the guest or the host one) oopses, or the physical machine gets power
cycled unexpectedly.

Please never suggest such a "workaround" to customers and/or partners. *Pretty please*.
Instead tell them to use the classical provisioning for Ubuntu based OpenStack clusters
(until fuel_agent gets fixed).

Building Ubuntu IBP images used to took ~ 10 minutes in Fuel 6.0, there's absolutely no reason
why fuel_agent should be any slower (python is not *that* bad). To improve IBP build time
the following design mistakes should be addressed:

a) building filesystem images in docker container is a bad idea. The container filesystem
     is backed by dm-snapshot which is known to have poor write performance which drops
     as O(1/written_data_size) (the more data you've written the slower it gets).
     Possible solution: use a dedicated filesystem (preferably ext4 without a journal) as
     temporary storage for building IBP images, mount it in the container as a volume.

b) Switch off journalling when building the image
    (the patch https://review.openstack.org/186030 has been recently).

c) (Long term) Stop using filesystem images as a backup format (use tarballs instead).

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/198675

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/198676

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/198681

[OpenStack Infra (hudson-openstack)]

Change abandoned by Evgeny Konstantinov (<email address hidden>) on branch: master
Review: https://review.openstack.org/197543
Reason: here's the one that substitutes this https://review.openstack.org/#/c/200505/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/198477
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e1c00fb8d8fe07fcf1ed2e38e1f4389bcacd311d
Submitter: Jenkins
Branch: master

commit e1c00fb8d8fe07fcf1ed2e38e1f4389bcacd311d
Author: Alexei Sheplyakov <email address hidden>
Date: Thu Jul 2 18:13:01 2015 +0300

    docker: create a host volume for building IBP images

    Makes IBP images build time reasonable, especially if the master
    node runs in a VM.

    fuel_agent (which builds Ubuntu images) runs in the mcollective
    container. In general OS setup generates quite a lot of IO.
    fuel_agent builds the OS image in a loopback filesystem, which
    results in an extra overhead. A container filesystem is backed
    by dm-snapshot which is known to have a poor write performance,
    especially for large files (like OS images). By default docker
    storage is a loopback file residing in /var/lib/docker. Thus
    the data written to the OS image being built goes through a number
    of layers:

    ext4 -> loop -> dm-snapshot (/var/lib/docker-> loop ->
         -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
         -> hard drive

    If the master node runs on a VM rather than a physcial hardware
    the above chain gets even more complicated:

    ext4 -> loop -> dm-snapshot (/var/lib/docker-> loop ->
         -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
         -> virtual hard drive (qemu) -> qcow2 -> ext4 (host)

    The write performance of such a setup is awful, as a result making
    the basic Ubuntu installation takes more than 15 minutes.

    Using the volume makes it possible to eliminate the extra layers,
    so the written data goes through

    ext4 -> loop -> ext4 -> dm-linear (os-var) -> hard drive

    Related-Bug: #1457643
    Change-Id: Id50bbea95aced153aedf831deea15025bd69c406

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/201947

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/201947
Committed: https://git.openstack.org/cgit/stackforge/fuel-agent/commit/?id=d28f340a7c448748bb48514311874a9429b23ee0
Submitter: Jenkins
Branch: master

commit d28f340a7c448748bb48514311874a9429b23ee0
Author: Alexei Sheplyakov <email address hidden>
Date: Fri Jul 3 11:08:42 2015 +0300

    fuel_agent: image build directory can be passed via command line

    Make it possible to set the image build directory via the command
    line. Also create the image build directory if it doesn't exist.

    The default image build directory (/tmp) can be too slow/small.
    In particular if fuel_agent runs in a container (which is the case
    on the Fuel master node) /tmp resides in the container filesystem
    which is backed by dm-snapshot. dm-snapshot is known to have a poor
    write performance, especially for large(er) files like OS images.
    By default docker storage is a loopback file residing in
    /var/lib/docker. Thus the data written to the OS image being built
    goes through a number of layers:

    ext4 -> loop -> dm-snapshot (/var/lib/docker-> loop ->
         -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
         -> hard drive

    which causes substantial IO overhead. If the master node runs on a VM
    rather than a physcial hardware the above chain gets even more
    complicated (and write performance gets really afwul):

    ext4 -> loop -> dm-snapshot (/var/lib/docker-> loop ->
         -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
         -> virtual hard drive (qemu) -> qcow2 -> ext4 (host)

    Using a host volume (mounted at /var/lib/fuel/ibp) as an image build
    directory eliminates the extra layers so the written data goes through

    ext4 -> loop -> ext4 -> dm-linear (os-var) -> hard drive

    so the IBP image build time gets reasonable.

    Related-Bug: #1457643
    Change-Id: Idcafd23e659fcc7c0d7e6087501799834234b34c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/198478
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=61f71f54e1f63656b3bf7ef7ba7db6166c0956bf
Submitter: Jenkins
Branch: master

commit 61f71f54e1f63656b3bf7ef7ba7db6166c0956bf
Author: Alexei Sheplyakov <email address hidden>
Date: Fri Jul 3 11:08:42 2015 +0300

    Make sure fuel_agent builds IBP images in a proper directory

    Use /var/lib/fuel/ibp as the image build directory. This directory
    will be backed by a volume in the mcollective container which reduces
    IO overhead and makes image build time reasonable.

    fuel_agent (which builds Ubuntu images) runs in the mcollective
    container. A container filesystem is backed by dm-snapshot which
    is known to have a poor write performance, especially for large
    files (like OS images). By default docker storage is a loopback file
    residing in /var/lib/docker. Thus the data written to the OS image
    being built goes through a number of layers:

    ext4 -> loop -> dm-snapshot (/var/lib/docker-> loop ->
         -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
         -> hard drive

    If the master node runs on a VM rather than a physcial hardware
    the above chain gets even more complicated:

    ext4 -> loop -> dm-snapshot (/var/lib/docker-> loop ->
         -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
         -> virtual hard drive (qemu) -> qcow2 -> ext4 (host)

    Using the volume makes it possible to eliminate the extra layers,
    so the written data goes through

    ext4 -> loop -> ext4 -> dm-linear (os-var) -> hard drive

    Note: this patch can be merged before the ones which actually make
    /var/lib/fuel/ibp a host volume (obviously there won't be any
    performance gain until fuel-library/fuel-main bits get merged).

    Merge-After: https://review.openstack.org/201947
    Related-Bug: #1457643
    Change-Id: Ic4d258523aa13b3879a730edb2d910522ec24c93

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 6.1-updates - we added another fix for this issue to be included into maintenance updates for 6.1 - https://bugs.launchpad.net/fuel/+bug/1470831 - as simpler, less invasive and easier to apply.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/198480
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=a7aad9dc88290b5b874a41a16c8c7e2a692dd5d1
Submitter: Jenkins
Branch: master

commit a7aad9dc88290b5b874a41a16c8c7e2a692dd5d1
Author: Alexei Sheplyakov <email address hidden>
Date: Thu Jul 2 18:11:36 2015 +0300

    docker: create a mount point for IBP image build directory

    Makes IBP images build time reasonable, especially if the master
    node runs in a VM.

    fuel_agent (which builds Ubuntu images) runs in the mcollective
    container. In general OS setup generates quite a lot of IO.
    fuel_agent builds the OS image in a loopback filesystem, which
    results in an extra overhead. A container filesystem is backed
    by dm-snapshot which is known to have a poor write performance,
    especially for large files (like OS images). By default docker
    storage is a loopback file residing in /var/lib/docker. Thus
    the data written to the OS image being built goes through a number
    of layers:

    ext4 -> loop -> dm-snapshot (/var/lib/docker-> loop ->
         -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
         -> hard drive

    If the master node runs on a VM rather than a physcial hardware
    the above chain gets even more complicated:

    ext4 -> loop -> dm-snapshot (/var/lib/docker-> loop ->
         -> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
         -> virtual hard drive (qemu) -> qcow2 -> ext4 (host)

    The write performance of such a setup is awful, as a result making
    the basic Ubuntu installation takes more than 15 minutes.

    Using the volume makes it possible to eliminate the extra layers,
    so the written data goes through

    ext4 -> loop -> ext4 -> dm-linear (os-var) -> hard drive

    Related-Bug: #1457643
    Change-Id: Ie47be6227e48049bc1dd3bb46fb1e58fd06d194f

[Alexei Sheplyakov (asheplyakov)]

> we added another fix for this issue to be included into maintenance updates for 6.1 - https://bugs.launchpad.net/fuel/+bug/1470831 - as simpler, less invasive and easier to apply.

That fix is incomplete (although it mitigates the problem)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexei Sheplyakov (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/198675
Reason: Performance? What's that? Just say a client to buy a hardware RAID.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexei Sheplyakov (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/198676
Reason: Performance does not matter.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexei Sheplyakov (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/198681
Reason: Performance does not matter (Fuel motto)

[Maksym Strukov (unbelll)]

No regression found. Verified in 7.0-301

{"build_id": "301", "build_number": "301", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "301", "build_number": "301", "api": "1.0", "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release": "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release": "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}