Fuel for OpenStack

Zabbix haproxy configuration dosent set 'source'

Bug #1510115 reported by Bartosz Kupidura
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Maciej Relewicz
Fuel for OpenStack 8.0

Bug Description

Zabbix use haproxy to handle agent traffic for virtual host "Openstack Cluster".

It binds to $zabbix_vip, but inside haproxy namespace we have more than one ip address:
b_management
b_zbx_vip_mgmt

This mean that traffic from haproxy is outcoming from "random" ip address (not accepted by zabbix-agent).

We need to set "source" inside haproxy listen:

listen zabbix-agent
  bind 192.168.0.3:10049
  balance roundrobin
  mode tcp
  option tcpka
  timeout client 48h
  timeout server 48h
  source 192.168.0.3
  server node-1 192.168.0.8:10050 check inter 5000 rise 2 fall 3
  server node-2 192.168.0.7:10050 check inter 5000 rise 2 fall 3
  server node-3 192.168.0.6:10050 check inter 5000 rise 2 fall 3

Maciej Relewicz (rlu)
Changed in fuel:
assignee: Fuel Plugin Zabbix (fuel-plugin-zabbix) → Maciej Relewicz (rlu)
OpenStack Infra (hudson-openstack)
Changed in fuel:
status: New → In Progress
Maciej Relewicz (rlu)
Changed in fuel:
milestone: none → 8.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-external-zabbix (master)

Reviewed: https://review.openstack.org/239876
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-external-zabbix/commit/?id=d78e3f52a5319580bc157835a59e5d795b3080d2
Submitter: Jenkins
Branch: master

commit d78e3f52a5319580bc157835a59e5d795b3080d2
Author: Maciej Relewicz <email address hidden>
Date: Wed Oct 28 10:08:23 2015 +0100

    added source parameter in zabbix haproxy configuration

    Closes-Bug: #1510115
    Change-Id: Id0ee351779ac072c3aefc1d8c5f335edbec3731c

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-external-zabbix (2.0)

Fix proposed to branch: 2.0
Review: https://review.openstack.org/240531

Dmitry Pyzhov (dpyzhov)
tags: added: area-plugins
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-plugin-external-zabbix (2.0)

Change abandoned by Swann Croiset (<email address hidden>) on branch: 2.0
Review: https://review.openstack.org/240531

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-external-zabbix (master)

Fix proposed to branch: master
Review: https://review.openstack.org/249279

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-plugin-external-zabbix (master)

Reviewed: https://review.openstack.org/249226
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-external-zabbix/commit/?id=a2ab0bc0a6ec22d16d54806195711eee16aa13fa
Submitter: Jenkins
Branch: master

commit a2ab0bc0a6ec22d16d54806195711eee16aa13fa
Author: Swann Croiset <email address hidden>
Date: Tue Nov 24 13:59:15 2015 +0000

    Revert "added source parameter in zabbix haproxy configuration"

    With several controllers this patch breaks things.

    haproxy running on other controllers (w/o zabbix vip) try to check
    their backend servers with an invalid address as "source"
    (because the zabbix vip lives only on 1 controller):

    listen zabbix-agent
      bind 10.109.2.3:10049
      balance roundrobin
      mode tcp
      option tcpka
      source 10.109.2.3 <---- this IP doesn't exist on controller and the 'check' below use it'
      timeout client 48h
      timeout server 48h
      server node-31 10.109.2.6:10050 check inter 5000 rise 2 fall 3
      server node-32 10.109.2.5:10050 check inter 5000 rise 2 fall 3
      server node-33 10.109.2.4:10050 check inter 5000 rise 2 fall 3

    This leads to trigger false positive alerts:
    Host Name
    node-31.test.domain.local node-33 backend of zabbix-agent proxy down
    node-31.test.domain.local node-32 backend of zabbix-agent proxy down
    node-31.test.domain.local node-31 backend of zabbix-agent proxy down

    HAproxy log entries:
    <129>Nov 24 13:21:54 node-32 haproxy[31597]: Server zabbix-agent/node-31 is DOWN, reason: Layer4 connection problem, info: "General socket error (Invalid argument)", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    <129>Nov 24 13:21:54 node-32 haproxy[31597]: Server zabbix-agent/node-32 is DOWN, reason: Layer4 connection problem, info: "General socket error (Invalid argument)", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    <129>Nov 24 13:21:54 node-32 haproxy[31597]: Server zabbix-agent/node-33 is DOWN, reason: Layer4 connection problem, info: "General socket error (Invalid argument)", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    <128>Nov 24 13:21:54 node-32 haproxy[31597]: proxy zabbix-agent has no server available!

    This reverts commit d78e3f52a5319580bc157835a59e5d795b3080d2.

    Co-Authored-By: Bartosz Kupidura <email address hidden>

    Related-bug: #1510115

    Change-Id: I5b90b7d7cfb06b0f0631c3fc76cc390e0975623d

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/249279
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-external-zabbix/commit/?id=06b9e4f561c92b56ae11823f43fce3fd57a3efe3
Submitter: Jenkins
Branch: master

commit 06b9e4f561c92b56ae11823f43fce3fd57a3efe3
Author: Swann Croiset <email address hidden>
Date: Tue Nov 24 16:45:33 2015 +0100

    Accept incoming connections from mgmt vip for zabbix agent

    Related-bug: #1510115

    Change-Id: I4a1ff1c47e4c3053ff5fde19f00d539eb1f6f9c3

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-plugin-external-zabbix (2.5)

Related fix proposed to branch: 2.5
Review: https://review.openstack.org/249652

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: 2.5
Review: https://review.openstack.org/249653

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-plugin-external-zabbix (2.5)

Reviewed: https://review.openstack.org/249652
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-external-zabbix/commit/?id=5e01aa238f4a226ffc51b182a259a6acd1abd3b9
Submitter: Jenkins
Branch: 2.5

commit 5e01aa238f4a226ffc51b182a259a6acd1abd3b9
Author: Swann Croiset <email address hidden>
Date: Tue Nov 24 13:59:15 2015 +0000

    Revert "added source parameter in zabbix haproxy configuration"

    With several controllers this patch breaks things.

    haproxy running on other controllers (w/o zabbix vip) try to check
    their backend servers with an invalid address as "source"
    (because the zabbix vip lives only on 1 controller):

    listen zabbix-agent
      bind 10.109.2.3:10049
      balance roundrobin
      mode tcp
      option tcpka
      source 10.109.2.3 <---- this IP doesn't exist on controller and the 'check' below use it'
      timeout client 48h
      timeout server 48h
      server node-31 10.109.2.6:10050 check inter 5000 rise 2 fall 3
      server node-32 10.109.2.5:10050 check inter 5000 rise 2 fall 3
      server node-33 10.109.2.4:10050 check inter 5000 rise 2 fall 3

    This leads to trigger false positive alerts:
    Host Name
    node-31.test.domain.local node-33 backend of zabbix-agent proxy down
    node-31.test.domain.local node-32 backend of zabbix-agent proxy down
    node-31.test.domain.local node-31 backend of zabbix-agent proxy down

    HAproxy log entries:
    <129>Nov 24 13:21:54 node-32 haproxy[31597]: Server zabbix-agent/node-31 is DOWN, reason: Layer4 connection problem, info: "General socket error (Invalid argument)", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    <129>Nov 24 13:21:54 node-32 haproxy[31597]: Server zabbix-agent/node-32 is DOWN, reason: Layer4 connection problem, info: "General socket error (Invalid argument)", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    <129>Nov 24 13:21:54 node-32 haproxy[31597]: Server zabbix-agent/node-33 is DOWN, reason: Layer4 connection problem, info: "General socket error (Invalid argument)", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
    <128>Nov 24 13:21:54 node-32 haproxy[31597]: proxy zabbix-agent has no server available!

    This reverts commit d78e3f52a5319580bc157835a59e5d795b3080d2.

    Co-Authored-By: Bartosz Kupidura <email address hidden>

    Related-bug: #1510115

    Change-Id: I5b90b7d7cfb06b0f0631c3fc76cc390e0975623d
    (cherry picked from commit a2ab0bc0a6ec22d16d54806195711eee16aa13fa)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/249653
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-external-zabbix/commit/?id=f3252b27b5be3ada2f6d86bf58ea74deb8cb0759
Submitter: Jenkins
Branch: 2.5

commit f3252b27b5be3ada2f6d86bf58ea74deb8cb0759
Author: Swann Croiset <email address hidden>
Date: Tue Nov 24 16:45:33 2015 +0100

    Accept incoming connections from mgmt vip for zabbix agent

    Related-bug: #1510115

    Change-Id: I4a1ff1c47e4c3053ff5fde19f00d539eb1f6f9c3
    (cherry picked from commit 06b9e4f561c92b56ae11823f43fce3fd57a3efe3)

Revision history for this message
Andrey Danin (gcon-monolake) wrote :

Bartosz, could you please next time file bugs regarding the plugin here https://bugs.launchpad.net/fuel-plugins/+filebug It helps to understand area of responsibility.

Nastya Urlapova (aurlapova)
tags: added: dev-to-verify
Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :

Fix released

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "589"

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.