Attempt to deploy after stop and settings change gives server error

[Kyrylo Romanenko]

Detailed bug description:
 <put your information here>

Steps to reproduce:
1. Setup environement: nodes, networks, verify networks etc.
Settings: Neutron VLAN, Cinver LVM, Ironic.
2. Start Deployment. Wait several minutes.
3. Stop deployment.
4. Go to http://<Fuel-IP>:8000/#cluster/1/settings/general and add two new Repositories.
5. Click "Deploy Changes."

Expected results:
Deployment should start again.
Actual result:
Got server error from Fuel.

Reproducibility:
After every stopping of deployment.

Workaround:
Re-create cluster with all settings.

Impact:
Makes troubles when you forget for some settings.

Description of the environment:
# shotgun2 short-report
cat /etc/fuel_build_id:
 143
cat /etc/fuel_build_number:
 143
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-library9.0-9.0.0-1.mos8252.noarch
 fuel-agent-9.0.0-1.mos272.noarch
 nailgun-mcagents-9.0.0-1.mos732.noarch
 fuel-misc-9.0.0-1.mos8252.noarch
 shotgun-9.0.0-1.mos87.noarch
 python-packetary-9.0.0-1.mos129.noarch
 fuel-bootstrap-cli-9.0.0-1.mos272.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8610.noarch
 fuel-mirror-9.0.0-1.mos129.noarch
 fuel-openstack-metadata-9.0.0-1.mos8610.noarch
 fuel-notify-9.0.0-1.mos8252.noarch
 fuel-ostf-9.0.0-1.mos920.noarch
 fuel-setup-9.0.0-1.mos6324.noarch
 python-fuelclient-9.0.0-1.mos301.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-9.0.0-1.mos6324.noarch
 fuel-utils-9.0.0-1.mos8252.noarch
 fuel-nailgun-9.0.0-1.mos8610.noarch
 fuel-release-9.0.0-1.mos6324.noarch
 rubygem-astute-9.0.0-1.mos732.noarch
 fuelmenu-9.0.0-1.mos263.noarch
 fuel-ui-9.0.0-1.mos2635.noarch
 fuel-migrate-9.0.0-1.mos8252.noarch

Additional repositories:
ironic deb http://perestroika-repo-tst.infra.mirantis.net/review/CR-18315/mos-repos/ubuntu/9.0 mos9.0 main restricted 1200
nova deb http://perestroika-repo-tst.infra.mirantis.net/review/CR-18323/mos-repos/ubuntu/9.0 mos9.0 main restricted 1200

[Kyrylo Romanenko (kromanenko)]

Fuel Giagnostic Snapshot.
https://drive.google.com/a/mirantis.com/file/d/0B6E70aHvCcRQSGMyS2xDZ3VoRVU/view?usp=sharing

[Anton Samoylov (asamoylov)]

from nailgun app.log:

2016-04-04 09:43:09.734 INFO [7efc8ad2e880] (base) Trying to start deployment at environment '1'
2016-04-04 09:43:09.795 INFO [7efc8ad2e880] (manager) Trying to start deployment at cluster 'Ironic'
2016-04-04 09:43:09.826 ERROR [7efc8ad2e880] (base) Error during execution of deployment task on environment '1': Cannot perform the actions because there are running tasks [<Task 'provision' 1d6530db-69fb-4019-b68e-5ebe0f85a6ff (1) running>, <Task 'deployment' 2f148a40-c2d6-4d6d-b8cc-493e40c18b6f (1) running>]
2016-04-04 09:43:09.826 ERROR [7efc8ad2e880] (base) Unexpected exception occured
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 285, in content_json
    resp = func(cls, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 564, in PUT
    task = task_manager.execute()
  File "/usr/lib/python2.7/site-packages/nailgun/task/manager.py", line 174, in execute
    self.check_no_running_deployment(self.cluster)
  File "/usr/lib/python2.7/site-packages/nailgun/task/manager.py", line 132, in check_no_running_deployment
    'running tasks {0}'.format(tasks_q.all()))
DeploymentAlreadyStarted: Cannot perform the actions because there are running tasks [<Task 'provision' 1d6530db-69fb-4019-b68e-5ebe0f85a6ff (1) running>, <Task 'deployment' 2f148a40-c2d6-4d6d-b8cc-493e40c18b6f (1) running>]

from api.log:

016-04-04 09:43:09.725 DEBUG [7efc8ad2e880] (logger) Request PUT /api/clusters/1/changes from 172.18.196.49:39902 {}
2016-04-04 09:43:09.828 ERROR [7efc8ad2e880] (logger) Response code '500 Internal Server Error' for PUT /api/clusters/1/changes from 172.18.196.49:39902
2016-04-04 09:43:10.185 DEBUG [7efc8ad2e880] (logger) Request PUT /api/nodes/agent/ from 10.109.26.4:46522

Looks like here we have the same issue as we have in this bug: https://bugs.launchpad.net/fuel/+bug/1561942

According to the bug#1561942, the fix is merged.

Verified on fuel 10.0 ISO build 464:
using similar environment settings as provided in the bug description, i couldn't reproduce the issue.

[Anton Samoylov (asamoylov)]

I'm changing status to Fix Commited according to my comment above.

[Dmitry Belyaninov (dbelyaninov)]

There is error on deployment for 9.1 snapshot #195

UI message:
http://paste.openstack.org/show/564803/

[Dmitry Belyaninov (dbelyaninov)]

Reverified on snapshot #223