plugin metadata is broken

[Matthew Mosesohn]

Steps to reproduce:
1 - Deploy 9.0-liberty iso 204 with detach-database plugin using all other defaults
2 - Add 1 controller, 1 database node
3 - Deploy

Expected results:
 detach-database hash in astute.yaml on nodes
Actual results:
 plugin is listed in plugins array, but no metadata is present

This was tested across multiple plugins, but I'm reporting this against a well known, established plugin.

[Matthew Mosesohn (raytrac3r)]

Note that this does NOT affect any 8.0 environments at this time.

[Ilya Kutukov (ikutukov)]

Here is attributes_metadata field in database, it seems that validator output was recorded instead of metadata itself

{"yaml_additional_config": {"regex": {"source": ".+", "error": "This field should not be empty"}, "description": "Input must be valid YAML", "weight": 52, "type": "textarea", "valu
e": "#Uncomment to enable remote DB:\n#remote_db: '10.20.99.1'\n#remote_db_password: 'secret'\ncustom_mysql_setup_class: 'galera'\n", "label": "Additional config"}, "metadata": {"we
ight": 90, "label": "Detach Database Plugin"}}

[Ilya Kutukov (ikutukov)]

the previous comment is invalid this is a hints for the gui

[Ilya Kutukov (ikutukov)]

Not reproduced on 8.0 RC1 (529 build)

[Ilya Kutukov (ikutukov)]

If the problem occurred in 2 weeks timeframe as Matthew says this commits could be the reason, because they are affecting cluster attributes processing:

Merge "Changed JSON on Mutable(Dict/List) in Cluster object"
https://github.com/openstack/fuel-web/commit/5e88734cf4c3556ae1b888fc11ff0e52f228c933

Merge "Fixed constructing of a Mutable objects tree"
https://github.com/openstack/fuel-web/commit/b0a0b3cac16dc21cab33b3430080cfb8d9398d39

[Ilya Kutukov (ikutukov)]

DB dump of the environment

[Ilya Kutukov (ikutukov)]

Please, be aware that "Changed JSON on Mutable(Dict/List) in Cluster object" patch could break not only plug-ins metadata, but any part of logic that working with data from JSON fields.

[Alexander Kislitsky (akislitsky)]

@Ilya

Plugin developers are blocked by this bug now and full covering of changed logic will take a time, thus I propose to land this fix for critical bug and implement checking for Mutable DB fields as separate bug: https://bugs.launchpad.net/fuel/+bug/1546063

[Ilya Kutukov (ikutukov)]

Is mutable JSON patch roll-back an option?
I don't think there is any new logic that work only with mutable fields.

[Vladimir Kuklin (vkuklin)]

Folks, I think that we need to revert the patch that caused the problem as this is a critical regression. After that we can fix all the stuff that needs to fixed, test things properly and land the proper code.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/281447

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/281447
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=13e7962001bc036b0f3255a61d9704546aab909e
Submitter: Jenkins
Branch: master

commit 13e7962001bc036b0f3255a61d9704546aab909e
Author: Bulat Gaifullin <email address hidden>
Date: Wed Feb 17 20:49:26 2016 +0300

    Fixed missed plugin attributes in deployment info

    After Mutable DB fields were introduced the changing of nested data in such
    fields leads to saving changes into DB.
    Now we can't use Mutable fields for transfer temporary changed data inside
    the business logic and all places with implicit using of JSON field for
    transferring temporary changed data were removed.
    This is the reason of loosing plugins data from the serialized
    deployment_info. For fix we are adding cluster plugins attributes into
    deployment_info explicitly.

    Change-Id: I0807c816c9d3af22a0d7e31f9ce96e974749c525
    Closes-Bug: #1544505

[Vladimir (vushakov)]

Verified on:
    Fuel 9.0 ISO #135

[Illia Polliul (ipolliul)]

Bug is actual for Fuel 8.0 and was reproduced multiple times.

See details in https://bugs.launchpad.net/fuel/+bug/1595266

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/332929

[Illia Polliul (ipolliul)]

Here are detail how to reproduce the bug:

1) Create an environment with 2 plugins installed and enabled: EMC VNX and Zabbix
2) Deploy environment with 3 Controller nodes + 1 Compute
3) Add one more Compute
4) Press "deploy changes"

Links to plugins code:
https://github.com/openstack/fuel-plugin-external-emc/tree/3.0
https://github.com/openstack/fuel-plugin-external-zabbix/tree/2.5

I suppose that problem is triggered by mixed format of plugins tasks, but this doesn't explain why deployment passes first time.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/332929
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=a5f3cc2a0b5a4d29bb709c72ad5e25416ef33f7e
Submitter: Jenkins
Branch: stable/8.0

commit a5f3cc2a0b5a4d29bb709c72ad5e25416ef33f7e
Author: Bulat Gaifullin <email address hidden>
Date: Wed Feb 17 20:49:26 2016 +0300

    Fixed missed plugin attributes in deployment info

    After Mutable DB fields were introduced the changing of nested data in such
    fields leads to saving changes into DB.
    Now we can't use Mutable fields for transfer temporary changed data inside
    the business logic and all places with implicit using of JSON field for
    transferring temporary changed data were removed.
    This is the reason of loosing plugins data from the serialized
    deployment_info. For fix we are adding cluster plugins attributes into
    deployment_info explicitly.

    Change-Id: I0807c816c9d3af22a0d7e31f9ce96e974749c525
    Closes-Bug: #1544505