LBaas V2: operating_status of 'dead' member is always online with Healthmonitor

Bug #1548774 reported by Cindia-blue on 2016-02-23
32
This bug affects 6 people
Affects Status Importance Assigned to Milestone
neutron
Undecided
Cindia-blue
octavia
Undecided
Nir Magnezi
senlin
Undecided
Unassigned

Bug Description

Expectation:
Lbaas v2 healthmonitor will update status of "bad" member just as it behaves with v1. However, operating_status of pool members will not change no matter it is normal or not.

ENV:
My devstack runs in a single node of ubuntu14.04 and uses master branch code, mysql and rabbitmq. Tenantname is 'demo', username is 'demo'. I am using private-subnet for loadbalancer and member VM. octavia provider.

Steps to reproduce:
create a vm from cirros-0.3.4-x86_64-uec image and create one member accordingly into loadbalancer pool with healthmonitor. Then curl to get the statues of loadbalancer, find member status is online. Then nova stop the member mapped VM, curl again and again. Its operating_status of member keeps 'online' instead of 'error'.

Below comes the curl response. No difference before and after pool member VM turns into SHUTOFF since no status change happens ever.

{"statuses": {"loadbalancer": {"name": "", "listeners": [{"pools": [{"name": "", "provisioning_status": "ACTIVE", "healthmonitor": {"type": "PING", "id": "cb41b4e4-7008-479f-a6d9-4751ac7a1ee4", "name": "", "provisioning_status": "ACTIVE"}, "members": [{"name": "", "provisioning_status": "ACTIVE", "address": "10.0.0.13", "protocol_port": 80, "id": "6d682536-e9fe-4456-ad24-df8521857ee0", "operating_status": "ONLINE"}], "id": "eaef79a9-d5e0-4582-b45b-cd460beea4fc", "operating_status": "ONLINE"}], "name": "", "id": "4e3a7d98-3ab9-4a39-b915-a9651fcada65", "operating_status": "ONLINE", "provisioning_status": "ACTIVE"}], "id": "ef45be96-15e0-42d9-af34-34608dafdb6c", "operating_status": "ONLINE", "provisioning_status": "ACTIVE"}}}

Cindia-blue, thank you for this bug report.
Could you add more information about it, based on this template?
http://docs.openstack.org/developer/neutron/policies/bugs.html#bug-report-template

Changed in neutron:
status: New → Incomplete
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
Yang Yu (yuyangbj) on 2016-02-25
Changed in neutron:
assignee: nobody → Yang Yu (yuyangbj)
Elena Ezhova (eezhova) wrote :

Reproduced for both Octavia and HaproxyOnHostPluginDriver service providers.

Changed in neutron:
status: Incomplete → Confirmed
tags: added: lbaas
cloudbuilders (operations-8) wrote :

We've came across the same problem. Is there any update on this bug? We've tried with Neutron + A10 lbaas driver and with HAProxy. The status is not updated in any case. The troubleshoot we did indicates that Neutron Agent is having a timeout waiting for a response on a rabbit queue. Seems that Neutron Server is either using another rabbit queue, or not responding the message posted by the agent at all.

Cindia-blue (miaoxinhuili) wrote :

Assign the bug to myself again and will try to fix it in next few days.

Changed in neutron:
assignee: Yang Yu (yuyangbj) → Cindia-blue (miaoxinhuili)

Fix proposed to branch: master
Review: https://review.openstack.org/325624

Changed in neutron:
status: Confirmed → In Progress
Changed in neutron:
assignee: Cindia-blue (miaoxinhuili) → KaiLi (damonl1)

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/323645
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Looks like it lost momentum.

Changed in neutron:
status: In Progress → Incomplete
assignee: KaiLi (damonl1) → nobody
tags: added: low-hanging-fruit
Cindia-blue (miaoxinhuili) wrote :

For the bug fix, please review this:
https://review.openstack.org/#/c/325624/

Changed in neutron:
assignee: nobody → Cindia-blue (miaoxinhuili)
Changed in neutron:
assignee: Cindia-blue (miaoxinhuili) → nobody
Cindia-blue (miaoxinhuili) wrote :

release the assignee, and need help to review below related fixes:
https://review.openstack.org/#/c/325624/
https://review.openstack.org/#/c/324197

Changed in neutron:
assignee: nobody → Cindia-blue (miaoxinhuili)
KaiLi (damonl1) on 2016-07-15
Changed in neutron:
status: Incomplete → Opinion
status: Opinion → Incomplete
Changed in neutron:
status: Incomplete → In Progress
Changed in neutron:
assignee: Cindia-blue (miaoxinhuili) → KaiLi (damonl1)
Changed in neutron:
assignee: KaiLi (damonl1) → Cindia-blue (miaoxinhuili)

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/325624
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
status: In Progress → Won't Fix
Qiming Teng (tengqim) wrote :

what does this "won't fix" mean? the bug is invalid? there are other alternatives or workarounds?

We should reassess whether or not a neutron-lbaas fix is worth addressing.

Change abandoned by Michael Johnson (<email address hidden>) on branch: master
Review: https://review.openstack.org/324197
Reason: This is already handled by enabling the event streamer in octavia.
event_streamer_driver = queue_event_streamer

Michael Johnson (johnsom) wrote :

Can you confirm you enabled the existing event streamer in octavia?
event_streamer_driver = queue_event_streamer

Changed in octavia:
status: New → Incomplete

Fix proposed to branch: master
Review: https://review.openstack.org/480933

Changed in octavia:
assignee: nobody → Gary Kotton (garyk)
status: Incomplete → In Progress
Changed in octavia:
assignee: Gary Kotton (garyk) → Nir Magnezi (nmagnezi)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers