neutron

LBAAS - housekeeping serive does not cleanup stale amphora VMs

Bug #1615502 reported by chandrasekaran natarajan on 2016-08-22

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Invalid	High	Ravikumar
	octavia	Invalid	Low	Unassigned

Bug Description

1. Initially there were no spare VMs since the “spare_amphora_pool_size = 0”
.

[house_keeping]
# Pool size for the spare pool
spare_amphora_pool_size = 0

stack@hlm:~/scratch/ansible/next/hos/ansible$ nova list --all
WARNING: Option "--all_tenants" is deprecated; use "--all-tenants"; this option will be removed in novaclient 3.3.0.
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| ID | Name | Tenant ID | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+
| 91eef324-0c51-4b91-8a54-e16abdb64e55 | vm1 | d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running | n1=4.5.6.5 |
| 7d85921c-e7d9-4b70-9023-0478c66b7e7c | vm2 | d15f2abc106740499a453260ae6522f3 | ACTIVE | - | Running | n1=4.5.6.6 |
+--------------------------------------+----------------------------------------------+----------------------------------+--------+------------+-------------+-------------------------------+

2. Change the spare pool size to 1 and restart Octavia-housekeeping service. Spare Amphora VM gets created as below.

3. Now change the spare pool size to 0 and restart Octavia-housekeeping service. Spare Amphora VM does not get deleted.

4. I see the requirement is not satisfied as per the config so hosue keeping should take care of cleaning up the spare amphora VM.But in the logs I see the statement “ requirement is satisfied “.

In the logs

2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating the cleanup of old amphora... db_cleanup /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating spare amphora check... spare_amphora_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
2016-08-22 06:37:57.203 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count : 0 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count : 1 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current spare amphora count satisfies the requirement spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60 2016-08-22 06:37:56.901 63471 DEBUG octavia.cmd.house_keeping [-] Initiating the cleanup of old amphora... db_cleanup /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:64
2016-08-22 06:37:57.194 63471 DEBUG octavia.cmd.house_keeping [-] Initiating spare amphora check... spare_amphora_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/cmd/house_keeping.py:49
2016-08-22 06:37:57.203 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Required Spare Amphora count : 0 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:45
2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current Spare Amphora count : 1 spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:46
2016-08-22 06:37:57.204 63471 DEBUG octavia.controller.housekeeping.house_keeping [-] Current spare amphora count satisfies the requirement spare_check /opt/stack/venv/octavia-20160815T061034Z/lib/python2.7/site-packages/octavia/controller/housekeeping/house_keeping.py:60

Tags:

Revision history for this message

chandrasekaran natarajan (chandrasekaran-natarajan) wrote on 2016-08-22:

Housekeeping log Edit (1.0 MiB, text/plain)

Revision history for this message

chandrasekaran natarajan (chandrasekaran-natarajan) wrote on 2016-08-22:

worker log Edit (554.7 KiB, text/plain)

Ravikumar (ravikumar-vallabhu) on 2016-08-23

Changed in neutron:
assignee:	nobody → Ravikumar (ravikumar-vallabhu)
status:	New → In Progress

Armando Migliaccio (armando-migliaccio) on 2016-08-24

tags:	added: lbaas
Changed in neutron:
importance:	Undecided → High

Revision history for this message

Elena Ezhova (eezhova) wrote on 2016-08-24:

This is a bug in octavia, not in neutron-lbaas.

Changed in neutron:
status:	In Progress → Invalid

Revision history for this message

Michael Johnson (johnsom) wrote on 2016-08-24:

This is correct, we currently do not have a mechanism to delete excess spares pool amphora.
However, they will get consumed as load balancers are created and/or failed over.

Marking this low as there is very little impact by having left over spares available until they are consumed by normal operations.

Changed in octavia:
importance:	Undecided → Low
status:	New → Triaged
tags:	removed: lbaas

Ravikumar (ravikumar-vallabhu) on 2016-08-25

Changed in octavia:
assignee:	nobody → Ravikumar (ravikumar-vallabhu)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-01: Fix proposed to octavia (master)

Fix proposed to branch: master
Review: https://review.openstack.org/364110

Changed in octavia:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-11: Change abandoned on octavia (master)

Change abandoned by Michael Johnson (<email address hidden>) on branch: master
Review: https://review.openstack.org/364110
Reason: This review has been blocked with multiple -1 reviews for more than 4 weeks without comment or updates. We are abandoning this for now.

Feel free to reactivate the review by pressing the restore button and
contacting the reviewers with the -1 on this review to ensure you
address their concerns.

Revision history for this message

Gregory Thiemonge (gthiemonge) wrote on 2023-03-31: auto-abandon-script

Abandoned after re-enabling the Octavia launchpad.