Unable to add plugin into existing deployed environment

Bug #1616825 reported by Denis Klepikov
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Julia Aranovich
Mitaka
Fix Released
High
Julia Aranovich
Newton
Fix Released
High
Julia Aranovich
Ocata
Fix Committed
High
Julia Aranovich
StackLight
Incomplete
Undecided
Unassigned

Bug Description

Unable to add plugin into existing deployed environment.

Error message:

You have just activated The StackLight InfluxDB-Grafana Server Plugin. The developer of this plugin hasn't specified it as "hot pluggable", so the cloud environment may stop functioning properly if this plugin is applied in post-deployment stage. Please consult with plugin developer on further appropriate course of actions.

You have just activated The StackLight Infrastructure Alerting Plugin. The developer of this plugin hasn't specified it as "hot pluggable", so the cloud environment may stop functioning properly if this plugin is applied in post-deployment stage. Please consult with plugin developer on further appropriate course of actions.

You have just activated The StackLight Collector Plugin. The developer of this plugin hasn't specified it as "hot pluggable", so the cloud environment may stop functioning properly if this plugin is applied in post-deployment stage. Please consult with plugin developer on further appropriate course of actions.

You have just activated The StackLight Elasticsearch-Kibana Server Plugin. The developer of this plugin hasn't specified it as "hot pluggable", so the cloud environment may stop functioning properly if this plugin is applied in post-deployment stage. Please consult with plugin developer on further appropriate course of actions.

Fuel 9.0

# fuel plugins
id | name | version | package_version | releases
---+-----------------------------+---------+-----------------+----------------------------------------------
1 | detach-rabbitmq | 1.1.1 | 3.0.0 | ubuntu (mitaka-9.0)
2 | elasticsearch_kibana | 0.10.1 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
3 | influxdb_grafana | 0.10.1 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
4 | lma_collector | 0.10.1 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
5 | lma_infrastructure_alerting | 0.10.1 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)

# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+----------+------------------+---------+-------------+-------------------+---------------------+---------------+--------+---------
 6 | ready | Untitled (52:c3) | 1 | 10.220.0.8 | 64:97:b0:1f:52:c3 | standalone-rabbitmq | | 1 | 1
 4 | ready | Untitled (32:39) | 1 | 10.220.0.9 | 64:e6:1b:de:32:39 | standalone-rabbitmq | | 1 | 1
 2 | ready | Untitled (c3:e3) | 1 | 10.220.0.5 | 64:0d:27:6f:c3:e3 | controller, mongo | | 1 | 1
 5 | ready | Untitled (4f:43) | 1 | 10.220.0.4 | 64:73:57:d5:4f:43 | standalone-rabbitmq | | 1 | 1
 8 | ready | Untitled (e8:21) | 1 | 10.220.0.12 | 64:02:16:7a:e8:21 | ceph-osd, compute | | 1 | 1
 7 | ready | Untitled (63:b5) | 1 | 10.220.0.6 | 64:3d:af:42:63:b5 | ceph-osd, compute | | 1 | 1
 9 | discover | Untitled (9a:d6) | | 10.220.0.7 | 64:5c:30:60:9a:d6 | | | 1 |
10 | discover | Untitled (4c:56) | | 10.220.0.11 | 64:9c:18:39:4c:56 | | | 1 |
 3 | ready | Untitled (38:2e) | 1 | 10.220.0.10 | 64:7f:95:72:38:2e | controller, mongo | | 1 | 1
 1 | ready | Untitled (3c:61) | 1 | 10.220.0.3 | 64:e8:29:4f:3c:61 | controller, mongo | | 1 | 1

description: updated
Revision history for this message
Denis Klepikov (dklepikov) wrote :
Revision history for this message
Denis Klepikov (dklepikov) wrote :

# fuel --version
9.0.0
# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+----------+------------------+---------+-------------+-------------------+---------------------+---------------+--------+---------
 9 | discover | Untitled (01:b5) | | 10.220.0.11 | 64:b3:64:95:01:b5 | | | 1 |
10 | discover | Untitled (3b:20) | | 10.220.0.12 | 64:be:95:bf:3b:20 | | | 1 |
 3 | ready | Untitled (c8:74) | 1 | 10.220.0.7 | 64:52:2d:41:c8:74 | controller | | 1 | 1
 1 | ready | Untitled (fc:78) | 1 | 10.220.0.3 | 64:ff:c0:3d:fc:78 | controller | | 1 | 1
 8 | ready | Untitled (2d:ab) | 1 | 10.220.0.10 | 64:29:e6:ad:2d:ab | standalone-rabbitmq | | 1 | 1
 5 | ready | Untitled (1d:8f) | 1 | 10.220.0.9 | 64:7c:6a:46:1d:8f | ceph-osd, compute | | 1 | 1
 6 | ready | Untitled (b5:00) | 1 | 10.220.0.5 | 64:47:d2:ca:b5:00 | standalone-rabbitmq | | 1 | 1
 2 | ready | Untitled (5b:1d) | 1 | 10.220.0.4 | 64:36:c7:d1:5b:1d | controller | | 1 | 1
 7 | ready | Untitled (e6:44) | 1 | 10.220.0.8 | 64:51:b5:0d:e6:44 | standalone-rabbitmq | | 1 | 1
 4 | ready | Untitled (48:a4) | 1 | 10.220.0.6 | 64:1a:95:ef:48:a4 | ceph-osd, compute | | 1 | 1
# fuel plugins
id | name | version | package_version | releases
---+-----------------------------+---------+-----------------+----------------------------------------------
1 | detach-rabbitmq | 1.1.1 | 3.0.0 | ubuntu (mitaka-9.0)
2 | elasticsearch_kibana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
3 | influxdb_grafana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
4 | lma_collector | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
5 | lma_infrastructure_alerting | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)

Roman Rufanov (rrufanov)
tags: added: customer-found
Revision history for this message
Denis Klepikov (dklepikov) wrote :

I re-check with clean Fuel 9.0 without any plugins
# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+----------+------------------+---------+-------------+-------------------+-------------------+---------------+--------+---------
 8 | discover | Untitled (4b:90) | | 10.220.0.10 | 64:18:82:a0:4b:90 | | | 1 |
10 | discover | Untitled (cb:68) | | 10.220.0.11 | 64:2e:56:93:cb:68 | | | 1 |
 7 | discover | Untitled (e5:15) | | 10.220.0.9 | 64:ef:a5:b7:e5:15 | | | 1 |
 3 | ready | Untitled (f7:cc) | 1 | 10.220.0.7 | 64:fd:a2:04:f7:cc | controller | | 1 | 1
 5 | ready | Untitled (f4:62) | 1 | 10.220.0.4 | 64:2d:0b:9f:f4:62 | ceph-osd, compute | | 1 | 1
 9 | discover | Untitled (84:15) | | 10.220.0.12 | 64:e3:e3:0f:84:15 | | | 1 |
 2 | ready | Untitled (a0:a5) | 1 | 10.220.0.5 | 64:c8:e5:7a:a0:a5 | controller | | 1 | 1
 1 | ready | Untitled (c3:23) | 1 | 10.220.0.3 | 64:ea:13:b3:c3:23 | controller | | 1 | 1
 4 | ready | Untitled (bd:6f) | 1 | 10.220.0.6 | 64:6f:44:7a:bd:6f | ceph-osd, compute | | 1 | 1
 6 | ready | Untitled (0d:2a) | 1 | 10.220.0.8 | 64:8f:e6:ad:0d:2a | ceph-osd, compute | | 1 | 1

# fuel plugins
id | name | version | package_version | releases
---+-----------------------------+---------+-----------------+----------------------------------------------
1 | ceilometer-redis | 1.0.3 | 3.0.0 | ubuntu (mitaka-9.0)
2 | elasticsearch_kibana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
3 | influxdb_grafana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
4 | lma_collector | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)
5 | lma_infrastructure_alerting | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)

Changed in fuel:
milestone: none → 9.1
importance: Undecided → High
description: updated
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

As error message says these plugins are not hot-pluggable and cannot be added to deployed environment.

Changed in fuel:
status: New → Invalid
Changed in lma-toolchain:
status: New → Invalid
Dmitry Pyzhov (dpyzhov)
Changed in lma-toolchain:
status: Invalid → New
Revision history for this message
Simon Pasquier (simon-pasquier) wrote :

@Dmitry for some reason the message is incorrect because the plugins are hot-pluggable [1]...

[1] https://github.com/openstack/fuel-plugin-lma-collector/blob/0254a53cbbbd3ebcb90adf2fe719b036395384df/metadata.yaml#L16

Changed in fuel:
status: Invalid → New
Revision history for this message
Denis Klepikov (dklepikov) wrote :

[root@nailgun ~]# fuel --version
9.0.0
[root@nailgun ~]# cat /var/www/nailgun/plugins/lma_collector-0.10/metadata.yaml
# Plugin name
name: lma_collector
# Human-readable name for your plugin
title: The StackLight Collector Plugin
# Plugin version
version: 0.10.0
# Description
description: The StackLight Collector is the advanced monitoring agent of the so called Logging, Monitoring and Alerting (LMA) Toolchain of Mirantis OpenStack.
# Required fuel version
fuel_version: ['8.0', '9.0']

authors: [Mirantis Inc.]
licenses: [Apache License Version 2.0]
homepage: https://github.com/openstack/fuel-plugin-lma-collector
groups: ['monitoring']
is_hotpluggable: true

# The plugin is compatible with releases in the list
releases:
  - os: ubuntu
    version: liberty-8.0
    mode: ['ha']
    deployment_scripts_path: deployment_scripts/
    repository_path: repositories/ubuntu
  - os: ubuntu
    version: liberty-9.0
    mode: ['ha']
    deployment_scripts_path: deployment_scripts/
    repository_path: repositories/ubuntu
  - os: ubuntu
    version: mitaka-9.0
    mode: ['ha']
    deployment_scripts_path: deployment_scripts/
    repository_path: repositories/ubuntu

# Version of plugin package
package_version: '4.0.0'

tags: added: support
Revision history for this message
Alexey Shtokolov (ashtokolov) wrote :

Denis, re-read the message. It's just a warning and it doesn't block the installation of not hot-pluggable plugins.

Could you please attach the plugins metadata files from /var/www/nailgun/plugins/plugin_name/metadata.yaml from your master node?

Changed in fuel:
assignee: nobody → Denis Klepikov (dklepikov)
status: New → Incomplete
Revision history for this message
Denis Klepikov (dklepikov) wrote :
Revision history for this message
Julia Aranovich (jkirnosova) wrote :

@Denis, thank you for the bug report.
I think the issue priority should be decreased to Medium because plugins are enabled on your screenshot. Message about not hot-pluggable version is a warning only, it doesn't block enabling and deploying the plugin.
Another question is why this message appears. It is displayed on UI in operational environment and when the chosen plugin version has hot_pluggable: false flag in its metadata.

So, we need to check response of GET /api/clusters/<cluster_id>/attributes to triage the issue.
Could you please provide it?

Changed in fuel:
importance: High → Medium
milestone: 9.1 → 9.2
Revision history for this message
Denis Klepikov (dklepikov) wrote :

@Julia,

>>> I think the issue priority should be decreased to Medium because plugins are enabled on your screenshot.
Plugins are enables but you can not save changes due to plugins waiting for some data, all of them have additional options

I move importance to High back

Changed in fuel:
importance: Medium → High
Revision history for this message
Denis Klepikov (dklepikov) wrote :

From my point of view it is a critical one, due to Customers can not use hot-pluggable architecture

Revision history for this message
Denis Klepikov (dklepikov) wrote :

This behavior can be reproduced on HA (3 controller).
To unblock LMA(Stacklight) plugins settings you can additional install NetApp Fuel Plugin, even if you do not plan to use it. After installation of the NetApp Fuel Plugin refresh Fuel UI to get LMA(Stacklight) plugins settings available.

Revision history for this message
Julia Aranovich (jkirnosova) wrote :

Can not reproduce the issue with the plugins installed to HA operational environment (9.0 release ISO). Diagnostic shapshot attached.

Also, it doesn't seem to be UI bug as UI displays controls properly if correct plugin data comes from Nailgun. I would check Nailgun responses for the environment where the issue reproduced.

Swann Croiset (swann-w)
Changed in lma-toolchain:
status: New → Incomplete
Revision history for this message
Denis Klepikov (dklepikov) wrote :
Revision history for this message
Julia Aranovich (jkirnosova) wrote :

The latest research shows that workaround for the issue is to refresh the page in browser.

Cluster settings will be reinitialized and new plugins data will be parsed properly:
https://github.com/openstack/fuel-ui/blob/master/static/models.js#L831-L833

Changed in fuel:
status: Incomplete → Confirmed
assignee: Denis Klepikov (dklepikov) → Fuel UI Team (fuel-ui)
tags: added: area-ui
Changed in fuel:
status: Confirmed → Triaged
Revision history for this message
Denis Klepikov (dklepikov) wrote :
tags: added: release-notes
description: updated
description: updated
tags: added: release-notes-done
removed: release-notes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/391735

Changed in fuel:
status: Triaged → In Progress
assignee: Fuel UI Team (fuel-ui) → Julia Aranovich (jkirnosova)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (master)

Fix proposed to branch: master
Review: https://review.openstack.org/394468

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (stable/mitaka)

Reviewed: https://review.openstack.org/391735
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=e06dbe5b5dd996c72bcaef80a78ab2550aebf250
Submitter: Jenkins
Branch: stable/mitaka

commit e06dbe5b5dd996c72bcaef80a78ab2550aebf250
Author: Julia Aranovich <email address hidden>
Date: Mon Oct 31 10:48:50 2016 +0300

    Process newly installed plugins

    If new plugins appear in settings response,
    their data should be processed and plugin settings
    should be shown immediately on UI.

    Closes-Bug: #1616825

    Change-Id: Iddedd93b11ab22f67af60efaf9f052d8e6f09815

tags: added: in-stable-mitaka
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (master)

Reviewed: https://review.openstack.org/394468
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=278fd74c5a46ced37401b42c3d504d9b2fa073b6
Submitter: Jenkins
Branch: master

commit 278fd74c5a46ced37401b42c3d504d9b2fa073b6
Author: Julia Aranovich <email address hidden>
Date: Mon Oct 31 10:48:50 2016 +0300

    Process newly installed plugins

    If new plugins appear in settings response,
    their data should be processed and plugin settings
    should be shown immediately on UI.

    Closes-Bug: #1616825

    Change-Id: Iddedd93b11ab22f67af60efaf9f052d8e6f09815

tags: added: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ui (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/396172

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ui (stable/newton)

Reviewed: https://review.openstack.org/396172
Committed: https://git.openstack.org/cgit/openstack/fuel-ui/commit/?id=142d8c1abaa859adceb622e340e97a0f5f3f2d12
Submitter: Jenkins
Branch: stable/newton

commit 142d8c1abaa859adceb622e340e97a0f5f3f2d12
Author: Julia Aranovich <email address hidden>
Date: Mon Oct 31 10:48:50 2016 +0300

    Process newly installed plugins

    If new plugins appear in settings response,
    their data should be processed and plugin settings
    should be shown immediately on UI.

    Closes-Bug: #1616825

    Change-Id: Iddedd93b11ab22f67af60efaf9f052d8e6f09815
    (cherry picked from commit 278fd74c5a46ced37401b42c3d504d9b2fa073b6)

Revision history for this message
Alexey. Kalashnikov (akalashnikov) wrote :

Verified on 9.2 snapshot #465 (2016-11-10)

shotgun2 report:
http://paste.openstack.org/show/589041/

Revision history for this message
Alexey. Kalashnikov (akalashnikov) wrote :

Actually, the wrong message about hot-plugging appears with plugin settings. It disappears after refreshing the page. There is no any limits to set up hot pluggable plugin on deployed environment.

tags: removed: on-verification
Revision history for this message
Maria Zlatkova (mzlatkova) wrote :

This bug has been described in 9.1 release notes. Adding the release-notes tag to describe the bug in 9.2 resolved issues.

tags: added: release-notes
removed: release-notes-done
Revision history for this message
Maria Zlatkova (mzlatkova) wrote :

A release note has been added to 9.2 resolved issues: https://review.fuel-infra.org/#/c/29581.

tags: added: release-notes-done
removed: release-notes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-ui 11.0.0.0rc1

This issue was fixed in the openstack/fuel-ui 11.0.0.0rc1 release candidate.

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote :

We don't have any "hot pluggable" plugin for 10.0 release. According to our rules issue was moved to "Fix Released", but we cannot verify the fix on stable/newton branch !

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.