Deployment failure with LMA Infrastructure Alerting, Elasticsearch Kibana, InfluxDB Grafana plugins enabled

[Andrey Lavrentyev]

Deployment failure with LMA Infrastructure Alerting, Elasticsearch Kibana, InfluxDB Grafana, LMA Collector plugins enabled.

Puppet logs refer to inability to connect to MySQL database

Steps to reproduce:

1. Create a new env
2. Build and install 4 plugins for the particular version:
  - Elasticsearch Kibana
  - InfluxDB Grafana
  - LMA Infrastructure Alerting
  - LMA Collector
3. Set network settings and verify network settings
4. Enable and configure plugins with the following:
  collector:
    elasticsearch mode 'none'
    influxdb mode: 'local'
    alerting mode: 'local'
  influx:
    influxdb rootpass: r00tme
    influxdb username: influxdb
    influxdb userpass: influxdbpass
    grafana username: grafana
    grafana userpass: grafanapass
    MySQL mode: local
    mysql dbname: grafanalma
    mysql username: grafanalma
    mysql password: mysqlpass
  lma infra alerting:
    send to: root@localhost
    send from: nagios@localhost
    smtp host: 127.0.0.1
5. Add 1 controller, 1 compute, 1 cinder and 1 LMA Infrastructure Alerting + Elasticsearch Kibana + InfluxDB Grafana
6. Deploy

Expected result:
Deploy is a success

Actual result:
All deployed nodes is a success except 'LMA Infrastructure Alerting + Elasticsearch Kibana + InfluxDB Grafana' node which is failed

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 250
cat /etc/fuel_build_number:
 250
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0

Plugins version:
[root@nailgun ~]# fuel plugins --list

influxdb_grafana | 0.10.0 | 4.0.0 | ubuntu liberty-8.0, liberty-9.0, mitaka-9.0)

lma_infrastructure_alerting | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)

elasticsearch_kibana | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)

lma_collector | 0.10.0 | 4.0.0 | ubuntu (liberty-8.0, liberty-9.0, mitaka-9.0)

Link to the diagnostic snapshot: https://drive.google.com/open?id=0B5HPBFb7K7gXUWhweWxUazE0X1E

[Nastya Urlapova (aurlapova)]

Please, specify here version of plugins.

[Roman Prykhodchenko (romcheg)]

Please allow anonymous access to the attached snapshot.

[Andrey Lavrentyev (alavrentyev)]

@Roman, are you trying to access it using mirantis account? If so there should not be any problem. The snapshot is shared to anyone who is inside mirantis domain. Please use mirantis account and set back status to 'Confirmed'. Thx.

[Roman Prykhodchenko (romcheg)]

If this is a public bug all the links must be world-accessible. Otherwise the bug should be marked as private.

[Andrey Lavrentyev (alavrentyev)]

What the tedious attitude.
Please try one more time, it should be fixed now.

[Roman Prykhodchenko (romcheg)]

Now works fine, thanks. Please keep in mind that we are working on the open-source project together with folks from different companies and individual folks. It's always important to keep everything accessible to everyone in the community.

[Roman Prykhodchenko (romcheg)]

Since the bug is most likely caused by a networking issue, reassigning it to the networking team.

[Aleksey Kasatkin (alekseyk-ru)]

@Roman,

> "bug is most likely caused by a networking issue"

Please provide more details.

[Aleksey Kasatkin (alekseyk-ru)]

A couple of such errors are found in puppet logs of node-6 (lma node):

2016-05-17 06:52:54 +0000 Puppet (err): Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe show databases' returned 1: ERROR 2003 (HY000): Can't connect to MySQL server on '10.109.1.3' (113)

AFAIC, authors of LMA plugin should deal with the issue. Looks like issue with configuration.

[Simon Pasquier (simon-pasquier)]

Please note that LMA bugs should be reported against the lma-toolchain project, not Fuel.

[Simon Pasquier (simon-pasquier)]

@Andrey, if the environment is still up, can you try running this command on node-6 and report the output:

puppet apply --modulepath=/etc/fuel/plugins/influxdb-grafana-0.10/puppet/modules /etc/fuel/plugins/influxdb-grafana-0.10/puppet/manifests/grafana_mysql.pp

[Andrey Lavrentyev (alavrentyev)]

@Simon, unfortunately it's gone. Testing activities assume spinning environment often. I'll see what can be done, will provide output when reproduce it.

[Swann Croiset (swann-w)]

This error is usually a NICs misconfiguration of the LMA node which is not well connected to the mgmt network.

I set invalid status for now since we didn't reproduce it. Feel free to add more details if you reproduce the bug and we'll investigate/fix it.