IBP: speed up image building

[Alexander Gordeev]

the less invasive, the most preferable and the most safer solution is to disable journalling of EXT4 temporary in order to speed up image building and fit within 1h task timeout.

So, during image building journalling will be disabled. Right after that it will be turned on again.

It doesn't increase a rate of potential data loss. Moreover, it was tested on 6.0 images.

Steps to reproduce:
1) create new cluster with ubuntu
2) deploy it

Expected results: deploy succeded
Actual results: deploy failed due to time was out for image building task.

[Alexander Gordeev (a-gordeev)]

Marking as fix commited for 7.0: https://review.openstack.org/#/c/186030/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/197943

[Alexander Gordeev (a-gordeev)]

related old bug https://bugs.launchpad.net/fuel/+bug/1470831

[Alexander Gordeev (a-gordeev)]

I've tried trick with disabling journal during the image building.

build time on VBox w patch applied: 11:21:29.976 - 11:37:02.160 ~ 15 mins
w/o patch applied: 11:42:57.511 - 12:06:46.282 ~ 25 mins

on KVM, cache=writeback w patch applied: 14:05:25.325 - 14:27:41.651 ~ 22 mins
w/o patch applied: 14:31:39.885 - 16:18:48.914 ~ 108 mins

[Alexander Gordeev (a-gordeev)]

I think switching to the another cache mode for KVM will start to produce the results which are much closer to VBox.

[Vitaly Sedelnik (vsedelnik)]

This fix will be included into maint update 2 for MOS 6.1 - targetting to 6.1-updates for now, to be changed to 6.1-mu-2 when created

[Big Switch Networks (fuel-bugs-internal)]

We tried with Dell R220 as a Fuel bare metal master node, and we still see the issue. sanpshot attached.

[Alexander Gordeev (a-gordeev)]

Confirmed to be reproducible on real H/W

[Egor Kotko (ykotko)]

Verified on hardware:
{"build_id": "2015-07-14_04-10-58", "build_number": "35", "release_versions": {"2014.2.2-7.0": {"VERSION": {"build_id": "2015-07-14_04-10-58", "build_number": "35", "api": "1.0", "fuel-library_sha": "be169b35505b184fa1018e14e4884f2117eba0c0", "nailgun_sha": "3577169e209596a8e4a95d1c41d2dde099a3945f", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "f35a361775ce949a7827439fc7780dff8c57d7f6", "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8", "fuel-ostf_sha": "62785c16f8399f30526d24c52bb9ca23e1585bfb", "release": "7.0", "fuelmain_sha": "de23bc4605c696938e383ed17c0ea78f26a918d2"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "be169b35505b184fa1018e14e4884f2117eba0c0", "nailgun_sha": "3577169e209596a8e4a95d1c41d2dde099a3945f", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "f35a361775ce949a7827439fc7780dff8c57d7f6", "astute_sha": "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8", "fuel-ostf_sha": "62785c16f8399f30526d24c52bb9ca23e1585bfb", "release": "7.0", "fuelmain_sha": "de23bc4605c696938e383ed17c0ea78f26a918d2"}

[Vitaly Sedelnik (vsedelnik)]

Alexander, please create errata for this issue and invite me, Alexander Gordeev, Alexey Shtokolov and Vadim Rovachev as reviewers to the CR.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: stable/6.1
Change author: Alexandr Nevenchannyy <email address hidden>
Review: https://review.fuel-infra.org/9668

[Vitaly Sedelnik (vsedelnik)]

We have seen this issue on a baremetal master node at partner environment (BigSwitch) - image generation took 1h10m and the deployment failed with timeout. Next deployment succeded. Here is the snapshot - https://drive.google.com/a/bigswitch.com/file/d/0B07f4p28b_XUYlBSSHpYRDFFUEU/view?usp=drive_web

However, we had no chance to disable journalling and see if timeout of 1 hour is enough with the fix.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/197943
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=dac001b4fee136c72e471b2cae6000944d1ef971
Submitter: Jenkins
Branch: stable/6.1

commit dac001b4fee136c72e471b2cae6000944d1ef971
Author: Alexander Gordeev <email address hidden>
Date: Wed May 27 17:48:10 2015 +0300

    IBP: disable ext4 journaling during the image building

    In order to speed-up image building process, fuel-agent will
    turn off journalling. That should noticeably reduce the cost of
    fsync calls. Expected speed-up is about 2x-3x.

    Additionaly this change will help image building job to fit within
    3600 secs timeout.

    Change-Id: I24fcc8ba16c092b1bc936928890717c6e9501aa7
    Related-Bug: #1441679
    Closes-Bug: #1470831
    (cherry picked from commit c5970ebe7298ddaee6dd0994042931795f73d06f)