Grenade job occasional failures due to lack of resources

[Andrey Pavlov]

The following error occurs in grenade job logs from time to time:

+ die 154 'Cluster is in Error state'
2015-11-23 18:42:57.740 | + local exitcode=0
2015-11-23 18:42:57.740 | [Call Trace]
2015-11-23 18:42:57.740 | /opt/stack/new/sahara/devstack/upgrade/resources.sh:251:create
2015-11-23 18:42:57.740 | /opt/stack/new/sahara/devstack/upgrade/resources.sh:154:die
2015-11-23 18:42:57.744 | [ERROR] /opt/stack/new/sahara/devstack/upgrade/resources.sh:154 Cluster is in Error state
2015-11-23 18:42:58.746 | 1 die /opt/stack/new/devstack/functions-common
2015-11-23 18:42:58.746 | 154 create /opt/stack/new/sahara/devstack/upgrade/resources.sh
2015-11-23 18:42:58.746 | 251 main /opt/stack/new/sahara/devstack/upgrade/resources.sh
2015-11-23 18:42:58.746 | Exit code: 1

in sahara logs:

2015-11-23 18:42:38.882 1642 ERROR sahara.service.ops [instance: none, cluster: c63a4d23-3818-41d8-becd-fe8d72c0af7a] HeatStackException: Heat stack failed with status Resource CREATE failed: resources.worker: Conflict: resources[0].resources.inst: Port 07b72a08-581b-4394-b79a-93fa987a5a4e is still in use. (HTTP 409) (Request-ID: req-fda465cd-00a5-4424-bbfa-fb73d2dd1368)

in heat logs:

2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource Traceback (most recent call last):
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource File "/opt/stack/old/heat/heat/engine/resource.py", line 625, in _action_recorder
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource yield
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource File "/opt/stack/old/heat/heat/engine/resource.py", line 695, in _do_action
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args)
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource File "/opt/stack/old/heat/heat/engine/scheduler.py", line 309, in wrapper
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource step = next(subtask)
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource File "/opt/stack/old/heat/heat/engine/resource.py", line 669, in action_handler_task
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource while not check(handler_data):
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource File "/opt/stack/old/heat/heat/engine/resources/openstack/nova/server.py", line 760, in check_create_complete
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource check = self.client_plugin()._check_active(server_id)
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource File "/opt/stack/old/heat/heat/engine/clients/os/nova.py", line 226, in _check_active
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource 'code': fault.get('code', _('Unknown'))
2015-11-23 18:42:27.255 30445 ERROR heat.engine.resource ResourceInError: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/249810

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/249810
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=9cd2f595be11a774e489549924058584bb60ee3e
Submitter: Jenkins
Branch: master

commit 9cd2f595be11a774e489549924058584bb60ee3e
Author: Andrey Pavlov <email address hidden>
Date: Thu Nov 26 14:52:59 2015 +0300

    Launching 1 instance in grenade instead of 2

    Second node group in grenade test was removed so that
    only 1 instance would be launched to reduce resources
    consumption. Also custom flavor creation was added.

    Change-Id: Ib68380c3e5ea3b46bd3989741aa51b85a7b236e0
    Closes-bug: #1519842