[9.1][update][LMA] "influxdb-cluster-haproxy" task has failed

[Ruslan Khozinov]

Environment description:

Fuel 9.1

I've deployed the next fuel env:
 - contrail-5.0 fuel plugin
 - murano (additional feature, not plugin)
 - lma_collector 0.10.2
 - elasticsearch_kibana 0.10.2

Role assignment:
 - contrail-controller
 - contrail-controller
 - contrail-analytics, elasticsearch_kibana, influxdb_grafana
 - controller
 - controller
 - controller
 - cinder, compute
 - contrail-controller
 - contrail-analytics, contrail-analytics-db

Detailed bug description:
Deployemnt has failed on the "influxdb-cluster-haproxy" task
with next parameters:
{
  "puppet_modules": "puppet/modules:/etc/puppet/modules",
  "puppet_manifest": "puppet/manifests/cluster-haproxy.pp",
  "timeout": 600,
  "cwd": "/etc/fuel/plugins/influxdb_grafana-0.10/"
}
with the next error:
(/Stage[main]/Main/Service[haproxy]/ensure) change from stopped to running failed: Execution timeout after 1800 seconds!

[Ruslan Khozinov (rkhozinov)]

pacemaker status from the failed node:

root@node-5:~# crm status
Last updated: Thu Dec 1 17:26:33 2016 Last change: Thu Dec 1 15:57:33 2016 by root via crm_resource on node-5.test.domain.local
Stack: corosync
Current DC: node-5.test.domain.local (version 1.1.14-70404b0) - partition with quorum
1 node and 5 resources configured

Online: [ node-5.test.domain.local ]

 vip__grafana (ocf::fuel:ns_IPaddr2): Started node-5.test.domain.local
 vip__es_vip_mgmt (ocf::fuel:ns_IPaddr2): Started node-5.test.domain.local
 vip__kibana (ocf::fuel:ns_IPaddr2): Started node-5.test.domain.local
 vip__influxdb (ocf::fuel:ns_IPaddr2): Started node-5.test.domain.local

Failed Actions:
* p_haproxy_start_0 on node-5.test.domain.local 'unknown error' (1): call=173, status=complete, exitreason='none',
    last-rc-change='Thu Dec 1 17:26:25 2016', queued=0ms, exec=101ms

[Ruslan Khozinov (rkhozinov)]

http://paste.ubuntu.com/23564145/ trace of the p_haproxy resource

++ 17:30:11: ocf_run:383: echo stdin: is not a tty '[ALERT]' 335/173011 '(21947)' : parsing '[/etc/haproxy/haproxy.cfg:31]' : unknown keyword ''\''include'\''' in ''\''defaults'\''' section '[ALERT]' 335/173011 '(21947)' : 'Error(s)' found in configuration file : /etc/haproxy/haproxy.cfg '[ALERT]' 335/173011 '(21947)' : Fatal errors found in configuration.
+ 17:30:11: ocf_run:383: output='stdin: is not a tty [ALERT] 335/173011 (21947) : parsing [/etc/haproxy/haproxy.cfg:31] : unknown keyword '\''include'\'' in '\''defaults'\'' section [ALERT] 335/173011 (21947) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 335/173011 (21947) : Fatal errors found in configuration.'
+ 17:30:11: ocf_run:384: '[' 1 -eq 0 ']'
+ 17:30:11: ocf_run:390: '[' '!' -z 'stdin: is not a tty [ALERT] 335/173011 (21947) : parsing [/etc/haproxy/haproxy.cfg:31] : unknown keyword '\''include'\'' in '\''defaults'\'' section [ALERT] 335/173011 (21947) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 335/173011 (21947) : Fatal errors found in configuration.' ']'
+ 17:30:11: ocf_run:391: ocf_log err 'stdin: is not a tty [ALERT] 335/173011 (21947) : parsing [/etc/haproxy/haproxy.cfg:31] : unknown keyword '\''include'\'' in '\''defaults'\'' section [ALERT] 335/173011 (21947) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg [ALERT] 335/173011 (21947) : Fatal errors found in configuration.'
+ 17:30:11: ocf_log:309: '[' 2 -lt 2 ']'

[Ruslan Khozinov (rkhozinov)]

haproxy:
  Installed: 1.5.4-1ppa1~precise
  Candidate: 1.5.4-1ppa1~precise
  Version table:
     1.6.3-1~u14.04+mos2 0
       1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-updates/main amd64 Packages
 *** 1.5.4-1ppa1~precise 0
       1200 http://10.109.0.2:8080/plugins/contrail-5.0/repositories/ubuntu/ Packages
        100 /var/lib/dpkg/status
     1.5.3-1~u14.04+mos2 0
       1050 http://10.109.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
     1.4.24-2ubuntu0.4 0
       1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-updates/main amd64 Packages
     1.4.24-2 0
       1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty/main amd64 Packages
pacemaker:
  Installed: 1.1.14-2~u14.04+mos1
  Candidate: 1.1.14-2~u14.04+mos1
  Version table:
 *** 1.1.14-2~u14.04+mos1 0
       1050 http://10.109.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
        100 /var/lib/dpkg/status
     1.1.10+git20130802-1ubuntu2.3 0
       1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-updates/main amd64 Packages
     1.1.10+git20130802-1ubuntu2 0
       1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty/main amd64 Packages

[Ruslan Khozinov (rkhozinov)]

haproxy.conf

# This file managed by Puppet
global
  daemon
  group haproxy
  log /dev/log local0
  maxconn 16000
  pidfile /var/run/haproxy.pid
  spread-checks 3
  stats socket /var/lib/haproxy/stats
  tune.bufsize 32768
  tune.maxrewrite 1024
  tune.ssl.default-dh-param 2048
  user haproxy

defaults
  log global
  maxconn 8000
  mode http
  option redispatch
  option http-server-close
  option splice-auto
  option dontlognull
  retries 3
  timeout http-request 20s
  timeout queue 1m
  timeout connect 10s
  timeout client 1m
  timeout server 1m
  timeout check 10s

include conf.d/*.cfg

[Ruslan Khozinov (rkhozinov)]

I've deployed another env only with contrail + murano(core) and I've seen different version of haproxy

root@node-3:~# apt-cache policy haproxy pacemaker
haproxy:
  Installed: 1.6.3-1~u14.04+mos2
  Candidate: 1.6.3-1~u14.04+mos2
  Version table:
 *** 1.6.3-1~u14.04+mos2 0
       1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.5.4-1ppa1~precise 0
        500 http://10.109.0.2:8080/plugins/contrail-5.0/repositories/ubuntu/ Packages
     1.5.3-1~u14.04+mos2 0
       1050 http://10.109.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
     1.4.24-2ubuntu0.4 0
       1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-updates/main amd64 Packages
     1.4.24-2 0
       1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty/main amd64 Packages
pacemaker:
  Installed: 1.1.14-2~u14.04+mos1
  Candidate: 1.1.14-2~u14.04+mos1
  Version table:
 *** 1.1.14-2~u14.04+mos1 0
       1050 http://10.109.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
        100 /var/lib/dpkg/status
     1.1.10+git20130802-1ubuntu2.3 0
       1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-updates/main amd64 Packages
     1.1.10+git20130802-1ubuntu2 0
       1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty/main amd64 Packages

[Ruslan Khozinov (rkhozinov)]

haproxy-1.5.4-1ppa1~precise from pinned contrail repo
haproxy-1.6.3-1~u14.04+mos2 from mos9.0-updates repo

Does it mean that contrail repo pinning was broken?

[Ruslan Khozinov (rkhozinov)]

contrail + stacklight + murano(core)

 100 /var/lib/dpkg/status
     release a=now
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-security/multiverse amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty-security,n=trusty,l=Ubuntu,c=multiverse
     origin mirror.seed-cz1.fuel-infra.org
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-security/universe amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty-security,n=trusty,l=Ubuntu,c=universe
     origin mirror.seed-cz1.fuel-infra.org
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-security/main amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty-security,n=trusty,l=Ubuntu,c=main
     origin mirror.seed-cz1.fuel-infra.org
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-updates/multiverse amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty-updates,n=trusty,l=Ubuntu,c=multiverse
     origin mirror.seed-cz1.fuel-infra.org
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-updates/universe amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty-updates,n=trusty,l=Ubuntu,c=universe
     origin mirror.seed-cz1.fuel-infra.org
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty-updates/main amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty-updates,n=trusty,l=Ubuntu,c=main
     origin mirror.seed-cz1.fuel-infra.org
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty/multiverse amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty,n=trusty,l=Ubuntu,c=multiverse
     origin mirror.seed-cz1.fuel-infra.org
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty/universe amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty,n=trusty,l=Ubuntu,c=universe
     origin mirror.seed-cz1.fuel-infra.org
1001 http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu/ trusty/main amd64 Packages
     release v=14.04,o=Ubuntu,a=trusty,n=trusty,l=Ubuntu,c=main
     origin mirror.seed-cz1.fuel-infra.org
1050 http://10.109.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/restricted amd64 Packages
     release o=Mirantis,a=mos9.0,n=mos9.0,l=mos9.0,c=restricted
     origin 10.109.0.2
1050 http://10.109.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
     release o=Mirantis,a=mos9.0,n=mos9.0,l=mos9.0,c=main
     origin 10.109.0.2
1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-updates/restricted amd64 Packages
     release o=Mirantis,a=mos9.0-updates,n=mos9.0,l=mos9.0,c=restricted
     origin mirror.fuel-infra.org
1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-updates/main amd64 Packages
     release o=Mirantis,a=mos9.0-updates,n=mos9.0,l=mos9.0,c=main
     origin mirror.fuel-infra.org
1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-security/restricted amd64 Packages
     release o=Mirantis,a=mos9.0-security,n=mos9.0-security,l=mos9.0,c=restricted
     origin mirror.fuel-infra.org
1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-security/main amd64 Packages
     release o=Mirantis,a=mos9.0-security,n=mos9.0-security,l=mos9.0,c=main
     origin mirror.fuel-infra.org
1100 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-holdback/restricted amd64 Packages
 ...

[Ruslan Khozinov (rkhozinov)]

For the first case contrail repo has repo priority 1200, but for another case has only 500

[Illia Polliul (ipolliul)]

Contrail plugin doesn't have 'influxdb-cluster-haproxy' task. It is from LMA plugin.

[Patrick Petit (patrick-michel-petit)]

Will address this bug in SL 1.0 for MOS 9.2.
In general co-habitation of plugins on same node is not guaranteed because of potential conflicts including port conflicts.

[Swann Croiset (swann-w)]

The root cause is the HAproxy package used, for some reason apt repository are (re)configured by someone and breaks things.
As Rsulan mentionned in #9, the pin is wrong.

Also, I'm wondering why the contrail-5 apt repo provides ppa package haproxy-1.5.4-1ppa1~precise ?

For recall, the haproxy for MOS has a special patch which add the 'include' directive conf.

To respond to Illia, yes the task which fails belongs to LMA but it only fails because of the wrong haproxy package.

I think the issue occurs because you deploy on same node(s) contrail and LMA and the HAproxy package is pinned here:
https://github.com/openstack/fuel-plugin-contrail/blob/d33c09901773ed473f9d47be3229697ce50cd420/deployment_scripts/puppet/modules/contrail/manifests/compute/override.pp#L18-L35

[Patrick Petit (patrick-michel-petit)]

Illia Polliul who's contrail plugin maintainer mentioned the plugin needs some fixes. That may be brought up to his attention.

[Illia Polliul (ipolliul)]

contrail-analytics requires that version of haproxy due to packages dependencies.
Solution would be to separate contrail-analytics and influxdb_grafana to different nodes.

[Simon Pasquier (simon-pasquier)]

We need to update node_roles.yaml for InfluxDB/Grafana and Elasticsearch/Kibana plugins to specify that they're incompatible with contrail* roles.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/418408

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/418411

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418408
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=d201f5cf5620787cad2493d276c75c4dbb4d1ecd
Submitter: Jenkins
Branch: master

commit d201f5cf5620787cad2493d276c75c4dbb4d1ecd
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 10 15:07:59 2017 +0100

    Prevent co-installation with the Contrail plugin

    Change-Id: I34d2caa7584567d22b9090170bf36093c9860919
    Closes-Bug: #1646550

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418411
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=de4f67b4b09839115a7f7d2e608e5e6e72b33fa7
Submitter: Jenkins
Branch: master

commit de4f67b4b09839115a7f7d2e608e5e6e72b33fa7
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 10 15:07:08 2017 +0100

    Prevent co-installation with the Contrail plugin

    Change-Id: I0ae666b0cef666c387a9b15d6ed71bfa3e75235c
    Closes-Bug: #1646550

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/1.0
Review: https://review.openstack.org/419937

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/0.10
Review: https://review.openstack.org/419938

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419937
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=e436de14c27287c5fb2a4ebb4e5edb1d80166508
Submitter: Jenkins
Branch: stable/1.0

commit e436de14c27287c5fb2a4ebb4e5edb1d80166508
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 10 15:07:08 2017 +0100

    Prevent co-installation with the Contrail plugin

    Change-Id: I0ae666b0cef666c387a9b15d6ed71bfa3e75235c
    Closes-Bug: #1646550
    (cherry picked from commit de4f67b4b09839115a7f7d2e608e5e6e72b33fa7)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/1.0
Review: https://review.openstack.org/419948

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419948
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=ff5abc1daefb51f59e15dd7fa1d7bfeab25f50e1
Submitter: Jenkins
Branch: stable/1.0

commit ff5abc1daefb51f59e15dd7fa1d7bfeab25f50e1
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 10 15:07:59 2017 +0100

    Prevent co-installation with the Contrail plugin

    Change-Id: I34d2caa7584567d22b9090170bf36093c9860919
    Closes-Bug: #1646550
    (cherry picked from commit d201f5cf5620787cad2493d276c75c4dbb4d1ecd)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419938
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=4717f4062d1612099178c31b23d13bcf34c12074
Submitter: Jenkins
Branch: stable/0.10

commit 4717f4062d1612099178c31b23d13bcf34c12074
Author: Simon Pasquier <email address hidden>
Date: Tue Jan 10 15:07:08 2017 +0100

    Prevent co-installation with the Contrail plugin

    Change-Id: I0ae666b0cef666c387a9b15d6ed71bfa3e75235c
    Closes-Bug: #1646550
    (cherry picked from commit de4f67b4b09839115a7f7d2e608e5e6e72b33fa7)