run_hook timeout smaller than needed

[Anastasia Palkina]

"build_id": "2015-04-07_22-54-31",
"ostf_sha": "67d00b954039975dbd97a0cff2600806dfde7947",
"build_number": "291",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-07_22-54-31", "ostf_sha": "67d00b954039975dbd97a0cff2600806dfde7947", "build_number": "291", "api": "1.0", "nailgun_sha": "96495457f8455dff98a01d5e613fc8afbf8a6fe2", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "30c8ce5cd074c8897960fa3a7f24d6a1165011ea", "astute_sha": "5041b2fb508e6860c3cb96474ca31ec97e549e8b", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2ca546b86e651d5638dbb1be9bae44b86c84a893", "fuellib_sha": "dc2d90a519dd60b0db803068e58b10274766b278"}}}, "auth_required": true,
"api": "1.0",
"nailgun_sha": "96495457f8455dff98a01d5e613fc8afbf8a6fe2",
"openstack_version": "2014.2-6.1",
"production": "docker",
"python-fuelclient_sha": "30c8ce5cd074c8897960fa3a7f24d6a1165011ea",
"astute_sha": "5041b2fb508e6860c3cb96474ca31ec97e549e8b",
"feature_groups": ["mirantis"],
"release": "6.1",
"fuelmain_sha": "2ca546b86e651d5638dbb1be9bae44b86c84a893",
"fuellib_sha": "dc2d90a519dd60b0db803068e58b10274766b278"

1. Create 2 environments with Ubuntu
2. Add needed nodes to them
3. Start both deployments in parallel
4. Both deployment has failed

[629] Error running provisioning: Failed to execute hook .
---
priority: 100
type: shell
uids:
- master
parameters:
  retries: 1
  cmd: 'fuel-image ''{"image_data": {"/boot": {"container": "gzip", "uri": "http://10.20.0.2:8080/targetimages/env_2_ubuntu_1404_amd64-boot.img.gz",
    "format": "ext2"}, "/": {"container": "gzip", "uri": "http://10.20.0.2:8080/targetimages/env_2_ubuntu_1404_amd64.img.gz",
    "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos": [{"name":
    "ubuntu", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/",
    "priority": null, "suite": "trusty", "type": "deb"}, {"name": "ubuntu-updates",
    "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/",
    "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name": "ubuntu-security",
    "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/",
    "priority": null, "suite": "trusty-security", "type": "deb"}, {"name": "mos",
    "section": "main restricted", "uri": "http://10.20.0.2:8080/2014.2-6.1/ubuntu/x86_64",
    "priority": 1050, "suite": "mos6.1", "type": "deb"}, {"name": "mos-updates", "section":
    "main restricted", "uri": "http://mirror.fuel-infra.org/mos/ubuntu/", "priority":
    1050, "suite": "mos6.1-updates", "type": "deb"}, {"name": "mos-security", "section":
    "main restricted", "uri": "http://mirror.fuel-infra.org/mos/ubuntu/", "priority":
    1050, "suite": "mos6.1-security", "type": "deb"}, {"name": "mos-holdback", "section":
    "main restricted", "uri": "http://mirror.fuel-infra.org/mos/ubuntu/", "priority":
    1100, "suite": "mos6.1-holdback", "type": "deb"}], "codename": "trusty"}'''
  cwd: "/"
  timeout: 1200

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/172009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/172009
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=27e28c49d47e887e97d31f0598390862fe61ca74
Submitter: Jenkins
Branch: master

commit 27e28c49d47e887e97d31f0598390862fe61ca74
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Apr 9 14:03:06 2015 +0300

    IBP Increased image building timeout up to 3600

    When testing on a VM with low-performance disks
    it is not always possible to build image in 1200
    because dpkg does a lot of IO operations.

    Change-Id: Ibf73a40e5c58ec15fddf1c85a13d7f9ce3001694
    Closes-Bug: #1441679

[Anastasia Palkina (apalkina)]

Verified on ISO #300

"build_id": "2015-04-09_22-54-31", "ostf_sha": "4bda5bbf9ea033189f16518032c063d43e4d0e5c", "build_number": "300", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-09_22-54-31", "ostf_sha": "4bda5bbf9ea033189f16518032c063d43e4d0e5c", "build_number": "300", "api": "1.0", "nailgun_sha": "d6e351189666e8afa01003e643e63216ef7abd26", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "5041b2fb508e6860c3cb96474ca31ec97e549e8b", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2ca546b86e651d5638dbb1be9bae44b86c84a893", "fuellib_sha": "e9c3ba332b05120c967b20260c7b223afc1b4f1a"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "d6e351189666e8afa01003e643e63216ef7abd26", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "5041b2fb508e6860c3cb96474ca31ec97e549e8b", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2ca546b86e651d5638dbb1be9bae44b86c84a893", "fuellib_sha": "e9c3ba332b05120c967b20260c7b223afc1b4f1a"

Image building + provisioning = 30-35 minutes on VBox for Ubuntu IBP

[Peter Zhurba (pzhurba)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "446"
  build_id: "2015-05-21_04-04-09"
  nailgun_sha: "403c6b7ea3c62bb4fda27eb9cedee37f7144558c"
  python-fuelclient_sha: "e19f1b65792f84c4a18b5a9473f85ef3ba172fce"
  astute_sha: "795f8a045400fe82ccc30ae018e85324b3fa1de5"
  fuel-library_sha: "a03efb582b06bfe8d9776dce244d4a2f2e2ba886"
  fuel-ostf_sha: "3dd25a018f2a5c47ec6c885436b3ba69690ef1b9"
  fuelmain_sha: "5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93"

Env
Fuel - KVM Vm 2GB 100GB HDD eth0 eth1
Controllers - 3 KVM Vm 3GB 100GB HDD eth0 eth1
Compute - 1 KVM Vm 4GB 200GB HDD eth0 eth1

Steps to reproduce
 Deploy Fuel

1 with routed public network on eth1
2 create in gui env with ubuntu 3 and deafult setting
3 deploy

4 Get error Failed to execute hook 'shell'

[Vladimir Kozhukalov (kozhukalov)]

No errors. Image building timed out. Two possible reasons for this:
1) Slow internet connection (least likely)
2) Low-performance hard drive on the master node (most likely)

If you use KVM based lab, please try to use cache='unsafe' for your master node hard drive. It'll make host system ignore sync-s from guest OS and so will decrease significantly the time that image building is going to take.

[Vladimir Kozhukalov (kozhukalov)]

https://bugs.launchpad.net/fuel/+bug/1457643 here is the doc bug about this.

[Vladimir Kozhukalov (kozhukalov)]

IMO timeout=3600 is more than enough for image building.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/186030

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/197943

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/197943
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=dac001b4fee136c72e471b2cae6000944d1ef971
Submitter: Jenkins
Branch: stable/6.1

commit dac001b4fee136c72e471b2cae6000944d1ef971
Author: Alexander Gordeev <email address hidden>
Date: Wed May 27 17:48:10 2015 +0300

    IBP: disable ext4 journaling during the image building

    In order to speed-up image building process, fuel-agent will
    turn off journalling. That should noticeably reduce the cost of
    fsync calls. Expected speed-up is about 2x-3x.

    Additionaly this change will help image building job to fit within
    3600 secs timeout.

    Change-Id: I24fcc8ba16c092b1bc936928890717c6e9501aa7
    Related-Bug: #1441679
    Closes-Bug: #1470831
    (cherry picked from commit c5970ebe7298ddaee6dd0994042931795f73d06f)