[Upgrade] Add validation which checks if there is enough disk space for ugprade

[Tatyanka]

Steps to reproduce:
1. Deploy simple flat environmnet(Centos + nova Flat 1 controller + 1 compute)
2. Run OSTF to validate cluster
3. Download latest tarball (in my case 11)
4. unpack it to /home directory on master ndoe
5. Run upgrade script (with enabled docker and openstack options)
6. Upgrade fails on upgrade nginx container(by reason a small free space size)
7. Rallback starts (and it is really good)

Expected:
Rallback finishes with success, all new containers were removed, supervisor starts all old container, UI is available, cluster is operational

Actual Result:
Rallback hangs on step of stoping services using supervisor. (2014-06-23 16:03:39 INFO 17651 (supervisor_client) Stop all services
)
But actually services are stopped
[root@nailgun docker-logs]# supervisorctl status
dhcrelay_monitor BACKOFF Exited too quickly (process log may have details)
docker-astute STOPPED Jun 23 03:59 PM
docker-cobbler STOPPED Jun 23 03:59 PM
docker-mcollective STOPPED Jun 23 03:59 PM
docker-nailgun STOPPED Jun 23 03:59 PM
docker-nginx STOPPED Jun 23 04:00 PM
docker-ostf STOPPED Jun 23 03:59 PM
docker-postgres STOPPED Jun 23 03:59 PM
docker-rabbitmq STOPPED Jun 23 03:59 PM
docker-rsync STOPPED Jun 23 03:59 PM
docker-rsyslog STOPPED Jun 23 04:00 PM

Seems we should verify status of container and try to stop only those one that hase running state.
As result I waiting near one our, and my deployment is non functional anymore...

P.S. See logs for details :)

[Nastya Urlapova (aurlapova)]

May be we should check and|or notify user about available free space before start upgrades|rollback?

[Tatyanka (tatyana-leontovich)]

Good point, but any way we should has plan B to get rallback passed. More strange in this case that rallback hang. We should set some timeouts and interrupt task, and mark services with error status or something like that :)

[Evgeniy L (rustyrobot)]

1. we have timeout [1]
2. Tatyana, are you sure that it hanged? How long did you wait?

[1] https://github.com/stackforge/fuel-web/blob/master/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/config.yaml#L24

[Tatyanka (tatyana-leontovich)]

Acrtually I left it yesterday, back today and it still hangs (( on step stopping supervisors ((

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/103910

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/103912

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/103912
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=887b5bf7500834439a56bc718089efc6814793fd
Submitter: Jenkins
Branch: master

commit 887b5bf7500834439a56bc718089efc6814793fd
Author: Evgeniy L <email address hidden>
Date: Tue Jul 1 18:45:47 2014 +0400

    Upgrade wraper, added error handling

    blueprint upgrade-to-5-1
    Related-bug: #1333350
    Change-Id: I56ff3b2b1a2f0d4c63787785640af531986eab75

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/103910
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=ccd0b30bb31d4c49221250be9dcaf62e82909b7e
Submitter: Jenkins
Branch: master

commit ccd0b30bb31d4c49221250be9dcaf62e82909b7e
Author: Evgeniy L <email address hidden>
Date: Mon Jun 30 21:28:15 2014 +0400

    Upgrades, implemented free space validator

    * renamed checker to health_checker
    * created new module before_upgrade_checker
      where we keep all our checkers
    * added flexible free space calculation
      system which recognizes mount points
    * simplifed upgrade manager now it has
      only general upgarde logic

    Closes-bug: #1333350
    blueprint upgrade-to-5-1
    Change-Id: Ic382001101d0ef178a048fc4aad8100de2f89317

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/104165

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/104165
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=a46d1aedf61571b2fa162671b51d30bd8520932f
Submitter: Jenkins
Branch: stable/5.0

commit a46d1aedf61571b2fa162671b51d30bd8520932f
Author: Evgeniy L <email address hidden>
Date: Mon Jun 30 21:28:15 2014 +0400

    Upgrades, implemented free space validator

    * renamed checker to health_checker
    * created new module before_upgrade_checker
      where we keep all our checkers
    * added flexible free space calculation
      system which recognizes mount points
    * simplifed upgrade manager now it has
      only general upgarde logic

    Closes-bug: #1333350
    blueprint upgrade-to-5-0-1
    Change-Id: Ic382001101d0ef178a048fc4aad8100de2f89317