[nailgun] Deploy button is active after rollback/redeploy

[Aleksey Kasatkin]

production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "406"
  build_id: "2014-08-06_11-23-14"
  astute_sha: "99a790ad1b7526cbbd5bf8add0cb2b4e503fccd4"
  fuellib_sha: "513ec5cdcdef74c7419d5bae967b9edc7da8dbd7"
  ostf_sha: "be71965998364bf8e6415bd38b75c84b63aab867"
  nailgun_sha: "f64b06c788e2b92fcb8e678ea6d0c9b86e8d0ab7"
  fuelmain_sha: "124ea87f1ac1c06e27613fe3b31fd5fc6b39e82d"

1. Install 5.0-26
2. Upgrade to 5.1
3. Deploy 5.0 ubuntu/simple/neutron/cinder (1 ctrl + 1 comp/cind)
4. Update env to 5.0.2 - it's failed
5. Rollback env - it's passed
6. Deploy button is active

The following screenshots shows the results of pressing 'Deploy' after rollback. Cluster have some changes in the same time. So, UI reflect these changes properly. Problem is on nailgun side.

Same thing happens sometimes after successful update/rollback, after deploy/reset/deploy, after simple deploy.

[Aleksey Kasatkin (alekseyk-ru)]

Deployment is started on second try but it's hanged and nothing is done.

[Aleksey Kasatkin (alekseyk-ru)]

Deployment is not started actually so 'stop deployment' doesn't work. But progress bar and stop button remain visible and active.

[Aleksey Kasatkin (alekseyk-ru)]

2014-08-08 08:43:12 DEBUG [7f09f7f53740] (logger) Response code '400 Bad Request' for PUT /api/clusters/3/stop_deployment from 10.20.0.1:40025

[Aleksey Kasatkin (alekseyk-ru)]

[root@fuel tmp]# fuel env
id | status | name | mode | release_id | changes | pending_release_id
---|-------------|--------|-----------|------------|--------------------------------------|-------------------
3 | operational | 3e4rty | multinode | 2 | [{u'node_id': 6, u'name': u'disks'}] | None

[Aleksey Kasatkin (alekseyk-ru)]

Created separate ticket about stop error: https://bugs.launchpad.net/fuel/+bug/1354401

[Aleksey Kasatkin (alekseyk-ru)]

Reproduced:

[root@fuel tmp]# fuel env
id | status | name | mode | release_id | changes | pending_release_id
---|-------------|------|-----------|------------|--------------------------------------|-------------------
2 | operational | asdf | multinode | 2 | [{u'node_id': 4, u'name': u'disks'}] | None

[Ihor Kalnytskyi (ikalnytskyi)]

Can't reproduce this. Could you please provide more repro steps? Maybe, you have done some additional steps such disk partitioning or network assignment?

[Aleksey Kasatkin (alekseyk-ru)]

Reproduced on iso http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_master_iso/65/

'Deploy' button become active after deployment then ostf run.

[root@fuel ~]# fuel env
id | status | name | mode | release_id | changes | pending_release_id
---|-------------|------|-----------|------------|--------------------------------------|-------------------
2 | operational | fghd | multinode | 2 | [{u'node_id': 5, u'name': u'disks'}] | None

Env is: ubuntu/simple/gre/ceph, 1 ctrl + 1 comp + 2 ceph

[Evgeniy L (rustyrobot)]

Igor, try to restore db dump from the snapshot and run nailgun in fake mode, and if there is a problem you will be able to debug it easily.

[Ihor Kalnytskyi (ikalnytskyi)]

Obviously the issue is that in some reason the cluster has non-empty changes attribute and unfortunately current logs don't show us the why that changes wasn't reseted.

Since the issue is incredibly hard to reproduce, I think we can move it to 6.0.

[Artem Panchenko (apanchenko-8)]

Reproduced this issue using upgrade tarball # 491. My steps:

1. Installed Fuel 5.0
2. Created and deployed new environment (Ubuntu, HA, NeutronGre, 3 controller+cinder and 2 compute+cinder nodes)
3. Upgraded master node to 5.1 using upgrade tarball
4. Ran update of environment from 5.0 to 5.0.2. It was successful and env passed OSTF checks.

But 'Deploy changes' button became available because disk configuration on one node was changed (see screenshot).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/117464
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=d2670f37c05b450114cb68b191308db74772cd31
Submitter: Jenkins
Branch: master

commit d2670f37c05b450114cb68b191308db74772cd31
Author: Nikolay Markov <email address hidden>
Date: Thu Aug 28 13:44:25 2014 +0400

    Added logging to pending changes

    Related-Bug: #1354299
    Change-Id: I00a57d0ed4a5fc5ef51cc3be4cf55f7dc790dc14

[Aleksey Kasatkin (alekseyk-ru)]

  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "491"
  build_id: "2014-09-01_00-01-17"
  astute_sha: "bc60b7d027ab244039f48c505ac52ab8eb0a990c"
  fuellib_sha: "2cfa83119ae90b13a5bac6a844bdadfaf5aeb13f"
  ostf_sha: "4dcd99cc4bfa19f52d4b87ed321eb84ff03844da"
  nailgun_sha: "d25ed02948a8be773e2bd87cfe583ef7be866bb2"
  fuelmain_sha: "109812be3425408dd7be192b5debf109cb1edd4c"

1. Create env, add nodes, deploy
2. Reset env,
3. Add/remove nodes
4. Deploy

There are cluster changes after deployment is complete:

id | status | name | mode | release_id | changes | pending_release_id
---|-------------|-------|-----------|------------|--------------------------------------|-------------------
1 | operational | kghkg | multinode | 2 | [{u'node_id': 2, u'name': u'disks'}] | None

[Nastya Urlapova (aurlapova)]

Guys, why you moved the high priority bug to 6.0?

[Nikolay Markov (nmarkov)]

That's because it's not easy to reproduce and takes a lot of time to research for it's causes. I added necessary logging which will simplify this process, but still we decided that this thing won't get in 5.1, we may backport it to 5.1.1 if things get worse. I'm changing priority to Medium.

[Aleksey Kasatkin (alekseyk-ru)]

  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "499"
  build_id: "2014-09-05_00-01-17"
  astute_sha: "b622d9b36dbdd1e03b282b9ee5b7435ba649e711"
  fuellib_sha: "6ebadd157c5153e9bb3ed53fa73507a5c73572c3"
  ostf_sha: "f7b5d9d0d1cfaba5f1fe1e2c634493e92bce11db"
  nailgun_sha: "b36dccbc28584532ca1bdb679b096cc3f5f6f80a"
  fuelmain_sha: "b3afef75675ae3ab251a4f990f3432968eb746eb"

# fuel env
id | status | name | mode | release_id | changes | pending_release_id
---|-------------|---------------|-----------|------------|--------------------------------------|-------------------
1 | operational | co-gre-cinder | multinode | 1 | [{u'node_id': 3, u'name': u'disks'}] | None

Found out after rollback.

[Aleksey Kasatkin (alekseyk-ru)]

#499 again.

Env: ubuntu/simple/neutron-vlan/ceph, 1 ctrl + 1 comp + 2 ceph

It's here just after deploy:

[root@fuel ~]# fuel env
id | status | name | mode | release_id | changes | pending_release_id
---|-------------|------|-----------|------------|--------------------------------------|-------------------
1 | operational | asdf | multinode | 2 | [{u'node_id': 3, u'name': u'disks'}] | None

[Aleksey Kasatkin (alekseyk-ru)]

Fix https://review.openstack.org/170550 was merged recently that locks disks update for non-bootstrap nodes. So, this bug should be rechecked.

[Oleksiy Molchanov (omolchanov)]

Moved to invalid, as this issue was not updated for more than 3 weeks.

[Denis Klepikov (dklepikov)]

High due to customer-found

reproduced on virtual lab

[root@fuel nailgun]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
    - experimental
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "58"
  build_id: "2014-12-26_14-25-46"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "fde8ba5e11a1acaf819d402c645c731af450aff0"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "5f91157daa6798ff522ca9f6d34e7e135f150a90"
  fuelmain_sha: "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8"

[root@fuel nailgun]# fuel env
id | status | name | mode | release_id | changes | pending_release_id
---|-------------|------|-----------|------------|----------------------------------------------------------------------------|-------------------
7 | operational | ril1 | multinode | 2 | [{u'node_id': 15, u'name': u'disks'}, {u'node_id': 17, u'name': u'disks'}] | None
[root@fuel nailgun]# fuel --env 7 deploy-changes
400 Client Error: Bad Request (No changes to deploy)
[root@fuel nailgun]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------|------------------|---------|-----------|-------------------|-------------------|---------------|--------|---------
14 | ready | Untitled (df:16) | 7 | 10.20.0.3 | d2:8f:3b:a9:96:44 | controller | | True | 7
16 | ready | Untitled (94:dc) | 7 | 10.20.0.5 | 8e:09:bf:42:28:41 | ceph-osd, compute | | True | 7
15 | ready | Untitled (1b:77) | 7 | 10.20.0.7 | 7a:40:cc:8f:74:44 | ceph-osd, compute | | True | 7
18 | ready | Untitled (22:41) | 7 | 10.20.0.4 | 7e:fb:38:a5:07:4b | ceph-osd | | True | 7
17 | ready | Untitled (9a:4b) | 7 | 10.20.0.6 | c2:6c:85:88:84:4d | compute | | True | 7

Possible steps for reproduce:

1 deploy cluster
2-20 mark one node to delete, check fuel env, revert deletion
21 deploy additional node to the env
22-40 mark one node to delete, check fuel env, revert deletion
41 check fuel env

Phantom tasks possible could be found into fuel env.

This issue is hard to reproduce.

[Alexey Stupnikov (astupnikov)]

MOS 6.0 is no longer supported, moving to Won't Fix.