systest deploy_neutron_stop_reset_on_provisioning failed with Internal Server Error code 500 [Cannot perform the actions because there are running tasks [<Task 'provision' 12305c24-b34a-4787-920b-73fada75c1a7 (1) running>]]

Bug #1561942 reported by Artem Hrechanychenko
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Roman Prykhodchenko
Mitaka
Fix Released
High
Roman Prykhodchenko

Bug Description

Detailed bug description:
systest deploy_neutron_stop_reset_on_provisioning failed with HTTPError: HTTP Error 500: Internal Server Error

2016-03-25 01:11:39,869 - INFO fuel_web_client.py:1901 -- Wait for slave-01 node online status
2016-03-25 01:11:40,054 - INFO fuel_web_client.py:1901 -- Wait for slave-02 node online status
2016-03-25 01:11:40,709 - INFO fuel_web_client.py:1416 -- Assigned networks are: {'enp0s6': ['private'], 'enp0s7': ['storage'], 'enp0s4': ['public'], 'enp0s5': ['management'], 'enp0s3': ['fuelweb_admin']}
2016-03-25 01:11:40,856 - INFO fuel_web_client.py:760 -- Deploy cluster 1
2016-03-25 01:11:40,856 - INFO fuel_web_client.py:814 -- Launch deployment of a cluster #1
2016-03-25 01:11:40,960 - ERROR http.py:105 -- Internal Server Error code 500 [Cannot perform the actions because there are running tasks [<Task 'provision' 12305c24-b34a-4787-920b-73fada75c1a7 (1) running>]]
2016-03-25 01:11:40,966 - ERROR __init__.py:66 -- deploy_cluster_changes raised: HTTPError()
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.cluster_actions/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.cluster_actions/fuelweb_test/helpers/decorators.py", line 146, in wrapped
    response = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.cluster_actions/fuelweb_test/models/nailgun_client.py", line 157, in deploy_cluster_changes
    "/api/clusters/{}/changes/".format(cluster_id)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.cluster_actions/fuelweb_test/helpers/http.py", line 83, in put
    return self._open(req)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.cluster_actions/fuelweb_test/helpers/http.py", line 92, in _open
    return self._get_response(req)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.cluster_actions/fuelweb_test/helpers/http.py", line 116, in _get_response
    return self.opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error

After reverting env:
root@nailgun ~]# fuel task
id | status | name | cluster | progress | uuid
---|--------|-----------------|---------|----------|-------------------------------------
2 | ready | stop_deployment | 1 | 100 | 6cc80250-2a5b-4c2e-8758-80eeb3d0c7d7

Steps to reproduce:
 Scenario:
            1. Create cluster in HA mode with 1 controller
            2. Add 1 node with controller role
            3. Add 1 node with compute role
            4. Run provisioning task
            5. Stop provisioning
            6. Reset settings
            7. Add 1 node with cinder role
            8. Re-deploy cluster
            9. Run OSTF

Expected results:
 test successfully passed

Actual result:
 Internal Server Error code 500 [Cannot perform the actions because there are running tasks [<Task 'provision' 12305c24-b34a-4787-920b-73fada75c1a7 (1) running>]]

Reproducibility:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.cluster_actions/57/console

Description of the environment:
cat /etc/fuel_build_id:
 108
cat /etc/fuel_build_number:
 108
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 liberty-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6317.noarch
 rubygem-astute-9.0.0-1.mos727.noarch
 fuel-library9.0-9.0.0-1.mos8190.noarch
 shotgun-9.0.0-1.mos85.noarch
 fuel-agent-9.0.0-1.mos265.noarch
 fuel-ui-9.0.0-1.mos2608.noarch
 fuel-nailgun-9.0.0-1.mos8568.noarch
 fuel-migrate-9.0.0-1.mos8190.noarch
 nailgun-mcagents-9.0.0-1.mos727.noarch
 fuel-misc-9.0.0-1.mos8190.noarch
 python-packetary-9.0.0-1.mos128.noarch
 fuel-bootstrap-cli-9.0.0-1.mos265.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8568.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos128.noarch
 fuel-openstack-metadata-9.0.0-1.mos8568.noarch
 fuel-notify-9.0.0-1.mos8190.noarch
 fuel-setup-9.0.0-1.mos6317.noarch
 python-fuelclient-9.0.0-1.mos292.noarch
 fuelmenu-9.0.0-1.mos263.noarch
 fuel-9.0.0-1.mos6317.noarch
 fuel-utils-9.0.0-1.mos8190.noarch
 fuel-ostf-9.0.0-1.mos919.noarch

Revision history for this message
Artem Hrechanychenko (agrechanichenko) wrote :
summary: - systest deploy_neutron_stop_reset_on_provisioning failed with HTTPError:
- HTTP Error 500: Internal Server Error
+ systest deploy_neutron_stop_reset_on_provisioning failed with Internal
+ Server Error code 500 [Cannot perform the actions because there are
+ running tasks [<Task 'provision' 12305c24-b34a-4787-920b-73fada75c1a7
+ (1) running>]]
tags: added: swarm-blocker
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

It seems like a problem with stop action that previous tasks were still running

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Fuel Python Team (fuel-python)
Changed in fuel:
status: New → Confirmed
Dmitry Pyzhov (dpyzhov)
tags: added: area-python
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Roman Prykhodchenko (romcheg)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Roman Prykhodchenko (romcheg) wrote :

The patch can be found here: https://review.openstack.org/#/c/302169/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/302169
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=33fa154b79096734212ee62ddf89be491ea28792
Submitter: Jenkins
Branch: master

commit 33fa154b79096734212ee62ddf89be491ea28792
Author: Roman Prykhodchenko <email address hidden>
Date: Wed Apr 6 12:44:14 2016 +0200

    Do not check deleted tasks before deployment

    After soft deletion was introduced for tasks the task
    manager's logic of telling whether there are any in-progress
    tasks for a specific clusters got wrong.

    This patch changes TaskCollection to not take deleted
    tasks into account.

    Closes-bug: #1561942
    Change-Id: Iafacba2acd80cc32a514ef337ddd39699820c4ab

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Aleksei Stepanov (penguinolog) wrote :

Still fail with the same error
ISO 9.0 #193

Changed in fuel:
status: Fix Committed → Confirmed
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 9.0 → 10.0
status: Confirmed → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/306544

tags: removed: swarm-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/306544
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=3ac1f844d6cf41e9a6d99d4f5ebbe9ba611f1120
Submitter: Jenkins
Branch: stable/mitaka

commit 3ac1f844d6cf41e9a6d99d4f5ebbe9ba611f1120
Author: Roman Prykhodchenko <email address hidden>
Date: Wed Apr 6 12:44:14 2016 +0200

    Do not check deleted tasks before deployment

    After soft deletion was introduced for tasks the task
    manager's logic of telling whether there are any in-progress
    tasks for a specific clusters got wrong.

    This patch changes TaskCollection to not take deleted
    tasks into account.

    Closes-bug: #1561942
    Change-Id: Iafacba2acd80cc32a514ef337ddd39699820c4ab

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

for mitaka verified on 285 iso

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.