Deployment of MOS 9.1 with StackLight - InfluxDB/Grafana Fuel Plugin is failed

[Sergii Rizvan]

Summary:
systemd package was recently added to trusty-updates repository. That's why post-install script from influxdb package tries to use systemclt to enable influxdb service. That leads to deployment error.

Env description:
 - compute
 - cinder+controller
 - elasticsearch_kibana+influxdb_grafana

Steps to reproduce:
1. Get Fuel 9.1
2. Configure
 - compute
 - cinder+controller
 - elasticsearch_kibana+influxdb_grafana (stacklight plugin from the oficial mirantis catalog)
3. Deploy environment
4. Deployment ends up with the following error: http://paste.openstack.org/show/593380/

This issue occurs because post-install script tries to use systemcl to enable influxdb service. Here is the parts of the post-install script which lead to the issue:

...
function install_systemd {
  cp -f $SCRIPT_DIR/influxdb.service /lib/systemd/system/influxdb.service
  systemctl enable influxdb
}
...

...
elif [[ -f /etc/debian_version ]]; then
  # Debian/Ubuntu logic
  which systemctl &>/dev/null
  if [[ $? -eq 0 ]]; then
      install_systemd
  else
      # Assuming sysv
      install_init
      install_update_rcd
  fi
...

P.S. As I've figured out from here https://github.com/openstack/fuel-plugin-influxdb-grafana/blob/master/pre_build_hook#L14

pre_build_hook just downloads influxdb deb package from https://s3.amazonaws.com/influxdb/influxdb_0.11.1-1_amd64.deb
So we need to patch that package and place it somewhere else.

[1] https://lists.ubuntu.com/archives/trusty-changes/2016-December/023368.htm

Workaround #1:
=============

Add systemd-shim to "additional packages" in fuel interface before deploying the environment.

Workaround #2:
=============

1. Provision the environment.
2. Install systemd-shim on the influxdb-grafana nodes.
3. Deploy the environment.

[Javier Diaz Jr (javierdiazcharles)]

I believe I am hitting the same issue with Stacklight in MOS 8 w/MU3. Ping me directly via Slack if you need access to lab where deployment is failing. See paste to match above:

http://paste.openstack.org/show/593479/

[Ruslan Khozinov (rkhozinov)]

That issue isn’t related to the https://bugs.launchpad.net/fuel/+bug/1652597

[guillaume thouvenin (guillaume-thouvenin)]

I'm not sure to understand the issue. What you are suggesting is that the version of InfluxDB that is downloaded by the pre_build_hook failed to be installed because now systemd is used? So you mean that InfluxDB 0.11.1 cannot be installed if systemd is detected?

[Simon Pasquier (simon-pasquier)]

IIUC this is the same cause as for https://bugs.launchpad.net/fuel/+bug/1648732. Because systemd package was recently added to trusty-updates repository, it breaks packages that erroneously detect systemd as the init system and try to run 'systemctl enable ...' in the post-inst script. One way to fix it is to install the systemd-shim package in this situation.

[Simon Pasquier (simon-pasquier)]

Ideally systemd-shim should only be installed on Trusty.

[Sergii Rizvan (srizvan)]

systemd-shim package isn't present in the list of initial packages by default. That's why in default configuration systemd package is going to be installed. This leads to the issue described above.
Fix should be relatively easy: we need just to remove systemctl call from postinst script. For example, like this: https://review.fuel-infra.org/#/c/29463/

[Simon Pasquier (simon-pasquier)]

Please re-read my comment: systemd-shim will be available since it will be included in the plugin... I don't see a problem here. Also we don't have the bandwidth/expertise to rebuild the upstream package so your proposal is a no-go for us.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/416276

[Laurent (laurent-tupin)]

I confirm that fix works on our environment with a MOS9.1.

Thanks for your job

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/416276
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=5f57e8dbe81e49ee51e1b022345965a503de6785
Submitter: Jenkins
Branch: master

commit 5f57e8dbe81e49ee51e1b022345965a503de6785
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 3 16:42:32 2017 +0100

    Fix InfluxDB install when systemd is present

    The influxdb package fails to install on Trusty machines because
    while systemd is now installed, it isn't running as the init system.
    This change installs the systemd-shim package to get rid of the issue.

    Change-Id: I571f923c877e723f74a39691e69df8957d21e95d
    Closes-Bug: #1652640

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/1.0
Review: https://review.openstack.org/419945

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419945
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=4d47c04caaf05fb1f66b6352fb4d07cb5f8cbe17
Submitter: Jenkins
Branch: stable/1.0

commit 4d47c04caaf05fb1f66b6352fb4d07cb5f8cbe17
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 3 16:42:32 2017 +0100

    Fix InfluxDB install when systemd is present

    The influxdb package fails to install on Trusty machines because
    while systemd is now installed, it isn't running as the init system.
    This change installs the systemd-shim package to get rid of the issue.

    Change-Id: I571f923c877e723f74a39691e69df8957d21e95d
    Closes-Bug: #1652640
    (cherry picked from commit 5f57e8dbe81e49ee51e1b022345965a503de6785)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.10
Review: https://review.openstack.org/420591

[Ruslan Khozinov (rkhozinov)]

I've builded the fuel-plugin-influxdb-grafana plugin from master

d201f5cf5620787cad2493d276c75c4dbb4d1ecd Prevent co-installation with the Contrail plugin
387ce3ee4c53f76ee854b25ec4d1bbad7d68f9fa Enforce strings in the Hiera YAML file
e8b18384e5dc763d80ece5abd5f80f7d79d73617 Don't rstrip() literal strings from Hiera
5f57e8dbe81e49ee51e1b022345965a503de6785 Fix InfluxDB install when systemd is present
768d468bf8010039fabe1203b27b50f7c763bbae Fix HAproxy dashboard
9a747fb3625e295113b83de344341c48e6c23ce3 Merge "Enforce string type for passwords in hiera data"
c6193a22165d2d69aac447da2ee7150306254054 Enforce string type for passwords in hiera data
6c21061d85787171e37e4d80b809205b5f55a2f7 [Docs] Edits
0e67121b7c9ed9217bd75ca3908763dd8bd7d748 Merge "Fix task dependency with the LMA collector plugin"
5f68b8579acfa6304d4d048dd0361d317dd98534 Update documentation for 1.0

And the redeployed previously failed env with 0.10.2 version.

So an env with the 1.0.0 version of the fuel-plugin-influxdb-grafana plugin has been successfully deployed.

[Ruslan Khozinov (rkhozinov)]

I've checked kibana/grafana services. They work properly.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420591
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=acce42de965e424eaa9f660e14786ee11a46e463
Submitter: Jenkins
Branch: stable/0.10

commit acce42de965e424eaa9f660e14786ee11a46e463
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 3 16:42:32 2017 +0100

    Fix InfluxDB install when systemd is present

    The influxdb package fails to install on Trusty machines because
    while systemd is now installed, it isn't running as the init system.
    This change installs the systemd-shim package to get rid of the issue.

    Change-Id: I571f923c877e723f74a39691e69df8957d21e95d
    Closes-Bug: #1652640
    (cherry picked from commit 5f57e8dbe81e49ee51e1b022345965a503de6785)

[Simon Pasquier (simon-pasquier)]

Lowering to High because there's a workaround.