Stop deployment while adding compute node to existing cluster leads to the destruction of the cluster

Bug #1529691 reported by Artem Hrechanychenko
44
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Artem Roma
5.1.x
In Progress
Critical
Rodion Tikunov
6.0.x
In Progress
Critical
Rodion Tikunov
6.1.x
Fix Released
Critical
Rodion Tikunov
7.0.x
Fix Released
Critical
Rodion Tikunov
8.0.x
Fix Released
Critical
Artem Roma
Mitaka
Fix Released
Critical
Artem Roma

Bug Description

ISO # 361, KVM

Steps to reproduce :
1. Deploy any environment with 3 controllers, 1 compute and NeutronTUN or NeutronVLAN
2. Add 1 compute to the cluster
3. Deploy changes
4. Stop deployment at the middle of re-deployment of non-primary controllers
5. Run OSTF

Actual result:
After stopped deployment process - non primary controller's are deleted and new compute - become "ready".

From astute log:
2015-12-28 18:15:04 INFO [847] Replace running task 56f35e3d-7018-40d3-989d-0f9fbaa674f6 to new 644c1467-89ee-4b2d-bd6a-6d97c4625ebe with data:
{"args"=>
  {"engine"=>
    {"url"=>"http://10.109.5.2:80/cobbler_api",
     "username"=>"cobbler",
     "password"=>"rGKTvfux",
     "master_ip"=>"10.109.5.2"},
   "nodes"=>
    [{"admin_ip"=>"10.109.5.10",
      "uid"=>"3",
      "roles"=>["controller"],
      "slave_name"=>"node-3"},
     {"admin_ip"=>"10.109.5.6",
      "uid"=>"2",
      "roles"=>["controller"],
      "slave_name"=>"node-2"}],
   "stop_task_uuid"=>"56f35e3d-7018-40d3-989d-0f9fbaa674f6",
   "task_uuid"=>"644c1467-89ee-4b2d-bd6a-6d97c4625ebe"},
 "respond_to"=>"stop_deployment_resp",
 "method"=>"remove_nodes",
 "api_version"=>"1"}
2015-12-28 18:15:04 INFO [847] Processing RPC call 'stop_deploy_task'

There is no way to run OSTF via WEB-UI, only via CLI with --force flag

[root@nailgun ~]# fuel --env 1 health --check smoke --force
[ 1 of 12] [failure] 'Create instance flavor' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[ 2 of 12] [failure] 'Check create, update and delete image actions using Glance v2' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[ 3 of 12] [failure] 'Create volume and boot instance from it' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[ 4 of 12] [failure] 'Create volume and attach it to instance' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[ 5 of 12] [failure] 'Check network connectivity from instance via floating IP' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[ 6 of 12] [failure] 'Create keypair' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[ 7 of 12] [failure] 'Create security group' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[ 8 of 12] [failure] 'Check network parameters' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[ 9 of 12] [failure] 'Launch instance' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[10 of 12] [failure] 'Launch instance with file injection' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[11 of 12] [failure] 'Launch instance, create snapshot, launch instance from snapshot' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct
[12 of 12] [failure] 'Create user and authenticate with it.' (0.0 s) Can not set proxy for Health Check.Make sure that network configuration for controllers is correct

Keystone service - is completely dead on primary controller
root@node-1:~# source openrc
root@node-1:~# keystone catalog
/usr/lib/python2.7/dist-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/usr/lib/python2.7/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py:376: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
Authorization Failed: Unable to establish connection to http://10.109.7.3:5000/v2.0/tokens
root@node-1:~# glance image-list
Unable to establish connection to http://10.109.7.3:5000/v2.0/tokens

Revision history for this message
Artem Hrechanychenko (agrechanichenko) wrote :
summary: - Stop deployment while adding node to existing clusterleads to the
+ Stop deployment while adding node to existing cluster leads to the
destruction of the cluster
description: updated
summary: - Stop deployment while adding node to existing cluster leads to the
- destruction of the cluster
+ Stop deployment while adding compute node to existing cluster leads to
+ the destruction of the cluster
Revision history for this message
Artem Hrechanychenko (agrechanichenko) wrote :

no workarounds - redeploy is not possible

Revision history for this message
Artem Hrechanychenko (agrechanichenko) wrote :

Deployment has failed. Method granular_deploy. Failed to execute hook 'shell' Failed to run command cd / && apt-get update

---
priority: 1900
type: shell
id:
parameters:
  retries: 3
  cmd: apt-get update
  cwd: /
  timeout: 1800
  interval: 1
uids:
- '1'
- '3'
- '2'
- '5'
- '4'
.
Inspect Astute logs for the details

description: updated
Dmitry Pyzhov (dpyzhov)
tags: added: area-library
removed: area-python
Changed in fuel:
assignee: nobody → Fuel Python Team (fuel-python)
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Stop deployment is dangerous operation and you can break your cluster if you do it in wrong moment. We show warning about it. Sorry, won't fix.

Changed in fuel:
status: New → Won't Fix
tags: added: area-python
removed: area-library
Revision history for this message
Dmitry Tyzhnenko (dtyzhnenko) wrote :

@DmitryP, maybe we should disable stop deploy action, if we can't support this action in scale story? What will happen if deploying failed by expected reason, for example lag of the internet connection?

Revision history for this message
Vladimir Khlyunev (vkhlyunev) wrote :

Status "Wont fix" restored for 8.0

Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/mitaka
Changed in fuel:
milestone: 8.0 → 9.0
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

@Dima P, while we support stop deployment operation and from description you see that case was absolutely valid + user will not have workaround -- I've confirmed issue. If your opinion that "stop deployment" is a dangerous thing, please talk with Sheena or Nathan about dropping of "stop deployment from Fuel".

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

So I can confirm that there is 2 behaviors after stop deploy:
1. After stop deployment nodes go to offline , if it do not back online - we ran task for nodes removal - as result - we get only those nodes that appears online in time

2 Second case - Nodes are back online in time (like my second deployment of this case), so I am able to re-deploy env - and it is failed with Deployment has failed. Method granular_deploy. Failed to execute hook 'sync_time' command: cd / && ntpdate -u $(awk '/^server/ { if ($2 !~ /127\.127\.[0-9]+\.[0-9]+/) {ORS=" "; print $2}}' /etc/ntp.conf)

I run re-deployment one more time (for now of environment in error status) and it fails again Deployment has failed. Method granular_deploy. Failed to execute hook 'sync_time' command: cd / && ntpdate -u $(awk '/^server/ { if ($2 !~ /127\.127\.[0-9]+\.[0-9]+/) {ORS=" "; print $2}}' /etc/ntp.conf)
root@node-1:~# cd / && ntpdate -u $(awk '/^server/ { if ($2 !~ /127\\.127\\.[0-9]+\\.[0-9]+/) {ORS=\" \"; print $2}}' /etc/ntp.conf)
awk: 1: unexpected character '\'
awk: line 1: runaway string constant " \"; print ...
18 Jan 12:15:07 ntpdate[25266]: no servers can be used, exiting
root@node-1:/# cd / && ntpdate -d -u $(awk '/^server/ { if ($2 !~ /127\\.127\\.[0-9]+\\.[0-9]+/) {ORS=\" \"; print $2}}' /etc/ntp.conf)
awk: 1: unexpected character '\'
awk: line 1: runaway string constant " \"; print ...
18 Jan 12:16:12 ntpdate[25517]: ntpdate 4.2.6p5@1.2349-o Wed Nov 11 17:58:55 UTC 2015 (1)
18 Jan 12:16:12 ntpdate[25517]: no servers can be used, exiting
root@node-1:/#

So the main question from mt here, could we do not remove offline nodes after stop deployment, because it looks really dangerous: User has ready cluster - add compute - stop deployment - may lost all offline nodes wihtout any notification/confirmation

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"
[root@nailgun ~]#

Revision history for this message
Arthur Svechnikov (asvechnikov) wrote :

There was discussion for similar issue https://<email address hidden>/msg71961.html
So appropriate solution still hasn't been found

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"
[root@nailgun ~]#

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

I believe there should not be nodes removal without user confirmation. When User decide to delete the cluster - we ask him/her really ? if so please put the name, and from other side - we just remove node )I believe we should ask about solution one more time in openstack dev

Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

We could not solve it in current arch because system do not know is it first deploy or after deploy (add additional nodes). Also current arch mark as deploying primary controller for many cases. We are erase all nodes which are deploying in the moment of stop deployment, so this is expected behavior. Yes, it is very bad behavior, but only thing that i can suggest is add warning: popup for UI and confirmation for cli.

Dmitry Pyzhov (dpyzhov)
tags: added: team-bugfix
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

@Dmitry, because of "Stop deployment is dangerous operation and you can break your cluster if you do it in wrong moment" this bug should be critical as it have the worst of possible impacts on operations. The stop deployment feature shall be disabled and reworked ASAP

tags: added: life-cycle-management
Changed in fuel:
importance: High → Critical
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Artem Roma (aroma-x)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/272636

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Artem Roma (<email address hidden>) on branch: master
Review: https://review.openstack.org/272636

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/273456

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/273456
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=b2c4b7d10207391d52102cc2dd4e140a90e31608
Submitter: Jenkins
Branch: master

commit b2c4b7d10207391d52102cc2dd4e140a90e31608
Author: Artem Roma <email address hidden>
Date: Thu Jan 28 12:58:09 2016 +0200

    Add 'deployed_before' flag to cluster that forbids stop deployment

    When cluster is successfuly deployed, that is, has 'operational' status,
    'deployed_before' key is added to generated attributes of the cluster.
    This flag then is used to determine whether stop action is allowed for
    the cluster.

    Change-Id: I8ecfca5819352ac27d7cec91e22f8006d8c3d8de
    Closes-Bug: #1529691

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/274676

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/8.0)

Reviewed: https://review.openstack.org/274676
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=5daabdb6f640d28999c92c574ad80ae782514812
Submitter: Jenkins
Branch: stable/8.0

commit 5daabdb6f640d28999c92c574ad80ae782514812
Author: Artem Roma <email address hidden>
Date: Thu Jan 28 12:58:09 2016 +0200

    Add 'deployed_before' flag to cluster that forbids stop deployment

    When cluster is successfuly deployed, that is, has 'operational' status,
    'deployed_before' key is added to generated attributes of the cluster.
    This flag then is used to determine whether stop action is allowed for
    the cluster.

    Change-Id: I8ecfca5819352ac27d7cec91e22f8006d8c3d8de
    Closes-Bug: #1529691
    (cherry picked from commit b2c4b7d10207391d52102cc2dd4e140a90e31608)

Vasily Gorin (vgorin)
tags: added: on-verification
Vasily Gorin (vgorin)
tags: removed: on-verification
tags: added: on-verification
Revision history for this message
Dmitry Tyzhnenko (dtyzhnenko) wrote :

Verified on 8.0-521

I've got a message - 'Stop action is forbidden for the cluster' when stop deployment

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "521"
  build_id: "521"
  fuel-nailgun_sha: "bae6d0062e0825d81409a04bcb4979302f8c65ea"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "29829b131ca802830bc5a9a131c83cd0f43f702b"
  fuel-ostf_sha: "7bcddf18020f2d94a553a441ff57dff9632865df"
  fuel-mirror_sha: "c25d8931e30322ecf43246c8a957e376259b685c"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

tags: removed: on-verification
Revision history for this message
Ksenia Svechnikova (kdemina) wrote :

Do we have a separate bp for the stop-deployment reworking or it has become forbidden on continuing basis?

Revision history for this message
Vladimir Sharshov (vsharshov) wrote :
Vladimir (vushakov)
tags: added: on-verification
Revision history for this message
Vladimir (vushakov) wrote :

Verified on:
    Fuel 9.0 ISO #101

tags: removed: on-verification
Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

Please inform that this behavior for 10.0 and 9.0 will be changed. Please check this bug https://bugs.launchpad.net/fuel/+bug/1571713

Revision history for this message
JohnsonYi (yichengli) wrote :

There is still same issue in my test environment as comment #3, it seems that problem still exist.

And another case during new node deployment also may cause the destruction of a cluster, I reported a bug here: https://bugs.launchpad.net/fuel/+bug/1555932

I found that every time when a new node was deploying, all exist nodes will flush the puppet scripts.

This potential risk may be a nightmare for who has a cluster in large scale.

Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Need to add 7.0 to project list

tags: added: customer-found
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/337463

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/7.0)

Reviewed: https://review.openstack.org/337463
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=e96920644888c46b3091bc6130a4df197c4987b8
Submitter: Jenkins
Branch: stable/7.0

commit e96920644888c46b3091bc6130a4df197c4987b8
Author: Artem Roma <email address hidden>
Date: Thu Jan 28 12:58:09 2016 +0200

    Add 'deployed_before' flag to cluster that forbids stop deployment

    When cluster is successfuly deployed, that is, has 'operational' status,
    'deployed_before' key is added to generated attributes of the cluster.
    This flag then is used to determine whether stop action is allowed for
    the cluster.

    Conflicts:
     nailgun/nailgun/objects/cluster.py
     nailgun/nailgun/test/integration/test_task_logging.py
     nailgun/nailgun/test/integration/test_verify_networks_task_manager.py
     nailgun/nailgun/test/unit/test_objects.py

    Change-Id: I8ecfca5819352ac27d7cec91e22f8006d8c3d8de
    Closes-Bug: #1529691
    (cherry picked from commit b2c4b7d10207391d52102cc2dd4e140a90e31608)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/338729

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/6.0)

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/338880

Revision history for this message
Rodion Tikunov (rtikunov) wrote :

Fix for this bug based on suggestion, what environments have a deployed_before flag in True value for early created clusters. This flag has not presented in envs <8.0. That cause the 500 error instead of 400 "Stop action is forbidden for the cluster".
So how we can setting up deployed_before flag for existing envs?
Do we need to backport patch https://review.openstack.org/#/c/321414/ for =<7.0? It seems that error 500 is similar with bug [1]

https://review.openstack.org/#/c/321414/
https://bugs.launchpad.net/fuel/+bug/1551339

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-web (stable/7.0)

Related fix proposed to branch: stable/7.0
Review: https://review.openstack.org/340827

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to tools/sustaining (master)

Related fix proposed to branch: master
Change author: Rodion Tikunov <email address hidden>
Review: https://review.fuel-infra.org/23264

Revision history for this message
Rodion Tikunov (rtikunov) wrote :

There is the patch for 5.1.x: https://review.openstack.org/#/c/341486/

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to tools/sustaining (master)

Reviewed: https://review.fuel-infra.org/23264
Submitter: Vitaly Sedelnik <email address hidden>
Branch: master

Commit: 36e35b67bb69da57e1c75356606e82924a1ca310
Author: Rodion Tikunov <email address hidden>
Date: Thu Jul 14 14:56:42 2016

Script for setting up deployed_before atribute

MOS versions <=7.0 should prevent `stop deployment` operation when
environment has already deployed. This requires that clusters have
`deployed_before` atribute in generated attributes of environment.

This script sets deployed_before in True value for all environments.
This script should be run on master node.

Change-Id: I554488217759cfedd3ce69af032b0e9426df6fb6
Related-Bug: #1529691

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-web (stable/7.0)

Reviewed: https://review.openstack.org/340827
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=2b0efd5eeb42431e03692f658792187b313083ea
Submitter: Jenkins
Branch: stable/7.0

commit 2b0efd5eeb42431e03692f658792187b313083ea
Author: Artem Roma <email address hidden>
Date: Tue Mar 15 12:37:07 2016 +0200

    Set 'deployed_before' attribute in cluster attributes if not present

    Before this change if master node had been upgraded to Fuel of
    versions >= 8.0, attempt to re/deploy existing clusters would have failed
    as attributes of those clusters would have lack 'deployed_before' field.

    Since processing of the flag is temporary solution it should not be
    coded in data base migration logic. Thus the change introduces creating
    of the attribute with given value if it is not already present.

    Conflicts:
     nailgun/nailgun/test/unit/test_objects.py

    Change-Id: I302ddb340d2ecf14b1fa414fda71aa7f32363bdf
    Closes-Bug: #1602281
    Related-Bug: #1529691
    (cherry picked from commit 48ee65f15e346cff46e0bf0cfba9dbe9afe43528)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/6.1)

Reviewed: https://review.openstack.org/338729
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=a961ebc2511a83f9b1f60db235921a2f103033e9
Submitter: Jenkins
Branch: stable/6.1

commit a961ebc2511a83f9b1f60db235921a2f103033e9
Author: Artem Roma <email address hidden>
Date: Thu Jan 28 12:58:09 2016 +0200

    Add 'deployed_before' flag to cluster that forbids stop deployment

    When cluster is successfuly deployed, that is, has 'operational' status,
    'deployed_before' key is added to generated attributes of the cluster.
    This flag then is used to determine whether stop action is allowed for
    the cluster.

    Conflicts:
     nailgun/nailgun/objects/cluster.py
     nailgun/nailgun/test/unit/test_objects.py

    Change-Id: I8ecfca5819352ac27d7cec91e22f8006d8c3d8de
    Closes-Bug: #1529691
    (cherry picked from commit e96920644888c46b3091bc6130a4df197c4987b8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-web (stable/6.1)

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/342855

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-web (stable/6.1)

Reviewed: https://review.openstack.org/342855
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=e4d09ec013843058837af395ba3db5d63a063a62
Submitter: Jenkins
Branch: stable/6.1

commit e4d09ec013843058837af395ba3db5d63a063a62
Author: Artem Roma <email address hidden>
Date: Tue Mar 15 12:37:07 2016 +0200

    Set 'deployed_before' attribute in cluster attributes if not present

    Before this change if master node had been upgraded to Fuel of
    versions >= 8.0, attempt to re/deploy existing clusters would have failed
    as attributes of those clusters would have lack 'deployed_before' field.

    Since processing of the flag is temporary solution it should not be
    coded in data base migration logic. Thus the change introduces creating
    of the attribute with given value if it is not already present.

    Conflicts:
     nailgun/nailgun/test/unit/test_objects.py

    Change-Id: I302ddb340d2ecf14b1fa414fda71aa7f32363bdf
    Closes-Bug: #1602281
    Related-Bug: #1529691
    (cherry picked from commit 48ee65f15e346cff46e0bf0cfba9dbe9afe43528)
    (cherry picked from commit 2b0efd5eeb42431e03692f658792187b313083ea)

tags: added: on-verification
Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on MOS 6.1 + MU7 updates.

After the fix it's not possible to stop deployment process: "Stop action is forbidden for the cluster"

tags: removed: on-verification
tags: added: on-verification
Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on MOS 7.0 + MU5 updates.

Scenario: add controller to cluster(3controllers, 2computes, 1cinder). Try to stop deployment in the middle of installing Openstack to non-primary controllers.
After the fix it's not possible to stop deployment process: "Stop action is forbidden for the cluster"

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/6.0)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/6.0
Review: https://review.openstack.org/338880
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (stable/5.1)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/5.1
Review: https://review.openstack.org/341486
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.