Deployment failed on provisioning step, and every node has error status

[Vasily Gorin]

RC1 (build #288)

Steps to reproduce:
1. Create new environment
2. Choose Neutron, VLAN
3. Choose Ceph for images
4. Choose Sahara
5. Choose Ceilometer
6. Add 1 controller+ceph
7. Add 1 compute+ceph
8. Add 1 cinder+ceph
9. Add 2 mongo
10. Change disk configuration for both Mongo nodes. Change 'MongoDB' volume for vdc
11. Verify networks
12. Deploy the environment

Expected result:
Deployment should be successfully

Actual result:
Deployment was failed on provisioning step with error message :
 "Failed to execute hook 'shell' Failed to run command cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": "http://10.109.0.2:8080/targetimages/env_1_ubuntu_1404_amd64-boot.img.gz", "format": "ext2"}, "/": {"container": "gzip", "uri": "http://10.109.0.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz", "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos": [{"name": "ubuntu", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty", "type": "deb"}, {"name": "ubuntu-updates", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name": "ubuntu-security", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty-security", "type": "deb"}, {"name": "mos", "section": "main restricted", "uri": "http://10.109.0.2:8080/2015.1.0-7.0/ubuntu/x86_64", "priority": 1050, "suite": "mos7.0", "type": "deb"}, {"name": "mos-updates", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/", "priority": 1050, "suite": "mos7.0-updates", "type": "deb"}, {"name": "mos-security", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/", "priority": 1050, "suite": "mos7.0-security", "type": "deb"}, {"name": "mos-holdback", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/", "priority": 1100, "suite": "mos7.0-holdback", "type": "deb"}, {"name": "Auxiliary", "section": "main restricted", "uri": "http://10.109.0.2:8080/2015.1.0-7.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary", "type": "deb"}], "codename": "trusty"}'"

file /var/log/docker-logs/fuel-agent-env-1.log was not found on master.

[Vasily Gorin (vgorin)]

If you need a cluster, connect with me. I will have this one some time

[Alexander Gordeev (a-gordeev)]

Something bad happened with ext4 filesystem on the particular virtual machine on which fuel master node is running.

Astute got complains in the logs about not found command:
2015-09-10T10:43:46 debug: [636] 96ce1175-ea4d-4aff-b60e-aa83fa01e69e: cmd: cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp --log-file ....
cwd: /
stdout:
stderr: sh: fa_build_image: command not found
exit code: 127

This complain looked like `fa_build_image` binary was not installed into mcollective container.
However, it was installed.

# rpm -qa|grep fuel-agent
fuel-agent-7.0.0-135.1.git082a47b.noarch

and package contained /usr/bin/fa_build_image:

# rpm -ql fuel-agent-7.0.0-135.1.git082a47b.noarch | grep fa_build
/usr/bin/fa_build_image

PATH was absolutely correct
[root@nailgun ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

BUT
[root@nailgun ~]# stat /usr/bin/fa_build_image
stat: cannot stat `/usr/bin/fa_build_image': Input/output error
[root@nailgun ~]# fa_build_image
bash: fa_build_image: command not found

no chance to read it due to corrupted fs.

In addition, log with kernel messages from fuel master node proves that: http://paste.openstack.org/show/455065/

Marking as Invalid. It was just broken virtual env.

[Alexei Sheplyakov (asheplyakov)]

> BUT
> [root@nailgun ~]# stat /usr/bin/fa_build_image
> stat: cannot stat `/usr/bin/fa_build_image': Input/output error

Perhaps cache='unsafe' parameter has been set in the virtual hard drive configuration (it should have been called 'please_wipe_out_my_files' or something like that so it's effect is more obvious).