[regression] Deployment enviroment was failed after update fuel deployment yaml files

[Alexey Galkin]

Detailed bug description:

Fuel can't deploy enviroment after update deployment facts via CLI.

Steps to reproduce:

1. Create new enviroment with config:

[cluster]
env_name=glance_celebrer
deploy_timeout=7200
virt_type=kvm
config_mode=ha_compact
release_name=ubuntu
net_provider=neutron
net_segment_type=vlan
debug=true
nova_quota=true
settings:{"volumes_lvm": false, "volumes_ceph": false, "images_ceph": false, "ephemeral_ceph": false,
 "objects_ceph": false, "osd_pool_size": "2", "sahara": false, "murano": false, "ceilometer": false}
node_roles={
 "node_1_controller":{"manufacturer":"KVM", "roles": ["controller"]},
 "node_2_controller":{"manufacturer":"KVM", "roles": ["controller"]},
 "node_3_compute":{"manufacturer":"KVM", "roles": ["compute"]},
 "node_4_compute":{"manufacturer":"KVM", "roles": ["compute"]}}
# Neutron network settings
interfaces={"eth0":[], "eth1": ["public", "management", "storage", "private"]}
networks:{"public":{"ip_ranges":[["172.18.163.70", "172.18.163.79"]],
 "cidr": "172.18.163.64/27", "gateway":"172.18.163.65"},
 "management":{"vlan_start": 101},
 "storage":{"vlan_start": 102},
  "networking_parameters":{"vlan_range": [1000, 1030], "floating_ranges": [["172.18.163.80", "172.18.163.89"]]}}

2. Get fuel nodes list:

[root@fuel ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|-------------------|---------|-----------|-------------------|-------|---------------|--------|---------
6 | discover | node_1_controller | 3 | 10.20.0.6 | 52:54:00:ed:b0:af | | controller | True | 3
7 | discover | node_4_compute | 3 | 10.20.0.4 | 52:54:00:c1:21:3c | | compute | True | 3
8 | discover | node_2_controller | 3 | 10.20.0.5 | 52:54:00:76:3b:7b | | controller | True | 3
5 | discover | node_3_compute | 3 | 10.20.0.3 | 52:54:00:15:e7:88 | | compute | True | 3

3. Get default deployment info:

[root@fuel ~]# fuel deployment --env 3 --default
Default deployment info was downloaded to /root/deployment_3

4. Update deployment info:

[root@fuel ~]# for i in {5..8}; do echo "kombu_compression: gzip" >> deployment_3/${i}.yaml; done

5. Upload changed info:

[root@fuel ~]# fuel deployment --env 3 --upload
deployment facts were uploaded.

6. Verify that the changes are applied:

[root@fuel ~]# rm -rf /root/deployment_3

[root@fuel ~]# fuel deployment --env 3 --download
Current deployment info was downloaded to /root/deployment_3

[root@fuel ~]# for i in {5..8}; do cat /root/deployment_3/${i}.yaml | grep "kombu"; done
kombu_compression: gzip
kombu_compression: gzip
kombu_compression: gzip
kombu_compression: gzip

7. Make deploy changes:

[root@fuel ~]# fuel --debug deploy-changes --env 3
PUT http://10.20.0.2:8000/api/v1/clusters/3/changes data={}
GET http://10.20.0.2:8000/api/v1/nodes/?cluster_id=3
GET http://10.20.0.2:8000/api/v1/transactions/27/
GET http://10.20.0.2:8000/api/v1/transactions/27/
                                                 GET http://10.20.0.2:8000/api/v1/nodes/5/
                                                                                          GET http://10.20.0.2:8000/api/v1/nodes/8/
                                                                                                                                   GET http://10.20.0.2:8000/api/v1/nodes/7/
                                                                                                                                                                            GET http://10.20.0.2:8000/api/v1/nodes/6/
      GET http://10.20.0.2:8000/api/v1/transactions/27/
'role'

Expected result:
 Successful environment deployment

Actual result:
 Error 'role' with staring deploy (but provisioning was successful).
 WebUI Screenshot with error: http://i.imgur.com/utqMzzu.png

Reproducibility:
 Always

Workaround:
 -

Impact:
 It's a very important feature and was often used

Description of the environment:
 Operation system: Linux fuel.domain.tld 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
 Version of components(Shotgun2 report): http://paste.openstack.org/raw/494969
 Fuel: 9.0-220

Additional information:
 Here is what I've found in the "/var/log/nailgun/app.log"

2016-04-21 10:08:42.252 ERROR [7fbec8582880] (manager) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/task/manager.py", line 57, in _call_silently
    to_return = method(task, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nailgun/task/task.py", line 251, in message
    events=reexecutable_filter, force=force
  File "/usr/lib/python2.7/site-packages/nailgun/task/task.py", line 146, in call_deployment_method
    args = getattr(cls, method)(transaction, **kwargs)
  File "/usr/lib/python2.7/site-packages/nailgun/task/task.py", line 376, in task_deploy
    transaction.cluster, nodes
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/deployment_serializers.py", line 893, in serialize_for_lcm
    DeploymentLCMSerializer(), cluster, nodes, ignore_customized
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/deployment_serializers.py", line 878, in _invoke_serializer
    cluster, nodes, ignore_customized=ignore_customized
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/deployment_serializers.py", line 115, in serialize
    self.serialize_customized(cluster, customized_nodes)
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/deployment_serializers.py", line 715, in serialize_customized
    roles.append(role_data.pop('role'))
KeyError: 'role'

[Alexey Galkin (agalkin)]

Set tag "blocker-for-qa" because this bug prevents 'Oslo.messaging message payload compression' feature verification.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/309043

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309043
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=8ed32e49c1b7fc0c0d61f772a46cb516a6f3c205
Submitter: Jenkins
Branch: master

commit 8ed32e49c1b7fc0c0d61f772a46cb516a6f3c205
Author: Artur Svechnikov <email address hidden>
Date: Thu Apr 21 17:48:16 2016 +0300

    Correct processing of customized nodes

    Since node can be customized with LCM
    deployment info, it's not needed to
    convert old serialization style to
    new.

    Change-Id: I4822745849925076cbcedbd028c9e37d91d1aa1c
    Closes-Bug: #1572965

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310124

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310124
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=39c93931ac63eff0b55cdf668f42ff457ae23f6b
Submitter: Jenkins
Branch: stable/mitaka

commit 39c93931ac63eff0b55cdf668f42ff457ae23f6b
Author: Artur Svechnikov <email address hidden>
Date: Thu Apr 21 17:48:16 2016 +0300

    Correct processing of customized nodes

    Since node can be customized with LCM
    deployment info, it's not needed to
    convert old serialization style to
    new.

    Change-Id: I4822745849925076cbcedbd028c9e37d91d1aa1c
    Closes-Bug: #1572965
    (cherry picked from commit 8ed32e49c1b7fc0c0d61f772a46cb516a6f3c205)

[Arthur Svechnikov (asvechnikov)]

regression was caused by commit https://review.openstack.org/#/c/295889/ with adding a new feature

[Alexey Galkin (agalkin)]

Verified as fixed in 9.0-278

[TatyanaGladysheva (tgladysheva)]

Verified as fixed on 10.0 build #1543.