Unable to delete loadbalancer even in ERROR state

Bug #1595269 reported by KRISHNA MOULI. TANKALA
This bug report is a duplicate of:  Bug #1607502: delete load balancers in ERROR state. Edit Remove
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
octavia
Fix Released
High
KRISHNA MOULI. TANKALA

Bug Description

Unable to delete loadbalancer even in ERROR state.
Octavia Load balancer is allowing to delete only in ACTIVE state, it is NOT allowing to delete it even in ERROR state.

When a load balancer is created it will go into PENDING_CREATE. after that if it is successful in spawning Amphora then it may go into ACTIVE state.

But due to some reasons like wrong subnet for amphora, or because some other network issue amphora may become unreachable to octavia controller. In such cases load balancer goes to ERROR state.

When a load balancer is in ERROR state we can issue delete of it.

stack@MouliDev:~$ neutron lbaas-loadbalancer-list
+--------------------------------------+------+-------------+---------------------+----------+
| id | name | vip_address | provisioning_status | provider |
+--------------------------------------+------+-------------+---------------------+----------+
| 4d616fe9-bdb0-49bb-a6c8-eee3317b3f24 | lb1 | 10.0.0.3 | ERROR | octavia |
| 99226592-2381-4b0a-895d-a00f9624b570 | lb2 | 10.0.0.11 | ERROR | octavia |
+--------------------------------------+------+-------------+---------------------+----------+

stack@MouliDev:~$ neutron lbaas-loadbalancer-delete lb2
Deleted loadbalancer: lb2

Though it says deleted we can see error in log as below.

2016-06-21 20:06:31.016 24901 INFO octavia.api.v1.controllers.load_balancer [req-e9254006-ac19-4eba-b54f-a878267eb13e - - - - -] Load Balancer 99226592-2381-4b0a-895d-a00f9624b570 is immutable.
127.0.0.1 - - [21/Jun/2016 20:06:31] "DELETE /v1/loadbalancers/99226592-2381-4b0a-895d-a00f9624b570 HTTP/1.1" 409 147

stack@MouliDev:~$ neutron lbaas-loadbalancer-list
+--------------------------------------+------+-------------+---------------------+----------+
| id | name | vip_address | provisioning_status | provider |
+--------------------------------------+------+-------------+---------------------+----------+
| 4d616fe9-bdb0-49bb-a6c8-eee3317b3f24 | lb1 | 10.0.0.3 | ERROR | octavia |
| 99226592-2381-4b0a-895d-a00f9624b570 | lb2 | 10.0.0.11 | PENDING_DELETE | octavia |
+--------------------------------------+------+-------------+---------------------+----------+

then load balancer will never be deleted as it is immutable in ERROR state.

after some time load balancer goes back to ERROR state again.

stack@MouliDev:~$ neutron lbaas-loadbalancer-list
+--------------------------------------+------+-------------+---------------------+----------+
| id | name | vip_address | provisioning_status | provider |
+--------------------------------------+------+-------------+---------------------+----------+
| 4d616fe9-bdb0-49bb-a6c8-eee3317b3f24 | lb1 | 10.0.0.3 | ERROR | octavia |
| 99226592-2381-4b0a-895d-a00f9624b570 | lb2 | 10.0.0.11 | ERROR | octavia |
+--------------------------------------+------+-------------+---------------------+----------+

Before proceeding to delete a load balancer, Octavia checks the provisioning status as below. Which allows to delete only if it is in ACTIVE state, But we should delete load balancer even if it is in ERROR state as well.

def test_and_set_lb_and_listeners_prov_status(self, session, lb_id,
                                                  lb_prov_status,
                                                  listener_prov_status,
                                                  listener_ids=None):
        """Tests and sets a load balancer and listener provisioning status.
        Puts a lock on the load balancer table to check the status of a
        load balancer. If the status is ACTIVE then the status of the load
        balancer and listener is updated and the method returns True. If the
        status is not ACTIVE, then nothing is done and False is returned.
        :param session: A Sql Alchemy database session.
        :param lb_id: id of Load Balancer
        :param lb_prov_status: Status to set Load Balancer and Listener if
                               check passes.
        :param listener_prov_status: Status to set Listeners if check passes
        :param listener_ids: List of ids of a listeners (can be empty)
        :returns: bool

octavia/common/constants.py
64
PENDING_UPDATE, DELETED, ERROR)
65
MUTABLE_STATUSES = (ACTIVE,)
66

Expected results:

User should be able to delete a load balancer when it is ERROR state.
By mistake user would have given wrong subnet which caused this issue, but user should be able to recover from his mistake.

Changed in octavia:
assignee: nobody → KRISHNA MOULI. TANKALA (mouli-tankala)
Revision history for this message
Michael Johnson (johnsom) wrote :

Hi, thank you for your bug report.

Can you please clarify for me. From the output included in the bug report it appears that the deleted load balancer has transitioned into the "PENDING_DELETE" state as opposed to the "ERROR" state as you mention. Did you mean that it gets stuck in "PENDING_DELETE" and cannot be deleted again?

Thanks,
Michael

Changed in octavia:
status: New → Incomplete
Revision history for this message
KRISHNA MOULI. TANKALA (mouli-tankala) wrote :

after some time load balancer goes back to ERROR state again.

stack@MouliDev:~$ neutron lbaas-loadbalancer-list
+--------------------------------------+------+-------------+---------------------+----------+
| id | name | vip_address | provisioning_status | provider |
+--------------------------------------+------+-------------+---------------------+----------+
| 4d616fe9-bdb0-49bb-a6c8-eee3317b3f24 | lb1 | 10.0.0.3 | ERROR | octavia |
| 99226592-2381-4b0a-895d-a00f9624b570 | lb2 | 10.0.0.11 | ERROR | octavia |
+--------------------------------------+------+-------------+---------------------+----------+

then load balancer will never be deleted as it is immutable in ERROR state.

description: updated
Revision history for this message
KRISHNA MOULI. TANKALA (mouli-tankala) wrote :

By Octavia design unless Load balancer is in ACTIVE state, it is NOT allowing to delete it.

I think, we should be able to DELETE it, if it is in ERROR state too.

octavia/common/constants.py
64
PENDING_UPDATE, DELETED, ERROR)
65
MUTABLE_STATUSES = (ACTIVE,)
66

67
SUPPORTED_AMPHORA_STATUSES = (AMPHORA_ALLOCATED, AMPHORA_BOOTING,

octavia/db/repositories.py

        """Tests and sets a load balancer and provisioning status.

         Puts a lock on the load balancer table to check the status of a
         load balancer. If the status is ACTIVE then the status of the load
         balancer is updated and the method returns True. If the
         status is not ACTIVE, then nothing is done and False is returned.

         :param session: A Sql Alchemy database session.
         :param id: id of Load Balancer
         :param status: Status to set Load Balancer if check passes.
         :returns: bool
         """

  323
lb = session.query(self.model_class).with_for_update().filter_by(
  324
id=id).one()
  325
if lb.provisioning_status not in constants.MUTABLE_STATUSES:
return False
             lb.provisioning_status = status
             session.add(lb)
            return True

description: updated
Revision history for this message
li,chen (chen-li) wrote :

I have met the same issue, with the log in o-api:

2016-06-23 03:35:16.658 31148 DEBUG oslo_db.sqlalchemy.engines [req-24afe2a3-f94d-4c7c-88e3-0ef9ed1a0244 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:256
2016-06-23 03:35:16.734 31148 INFO octavia.api.v1.controllers.load_balancer [req-24afe2a3-f94d-4c7c-88e3-0ef9ed1a0244 - - - - -] Load Balancer 55912c67-6a33-42d5-b1b3-407a3e6b3e43 is immutable.
2016-06-23 03:35:16.749 31148 DEBUG wsme.api [req-24afe2a3-f94d-4c7c-88e3-0ef9ed1a0244 - - - - -] Client-side error: Load Balancer 55912c67-6a33-42d5-b1b3-407a3e6b3e43 is immutable and cannot be updated. format_exception /usr/lib/python2.7/site-packages/wsme/api.py:221
127.0.0.1 - - [23/Jun/2016 03:35:16] "DELETE /v1/loadbalancers/55912c67-6a33-42d5-b1b3-407a3e6b3e43 HTTP/1.1" 409 147
127.0.0.1 - - [23/Jun/2016 03:35:16] "GET /v1/loadbalancers/55912c67-6a33-42d5-b1b3-407a3e6b3e43 HTTP/1.1" 200 429

Changed in octavia:
status: Incomplete → Confirmed
Revision history for this message
Michael Johnson (johnsom) wrote :

Hi,

Thank you for clarifying your bug. It might be easier for people to follow the discussion on this bug if you do not change the originally submitted description, but clarify by adding comments.

I see that this is in fact an issue in the current code.

Thank you for your bug report!

Changed in octavia:
importance: Undecided → High
status: Confirmed → Triaged
Revision history for this message
Adam Harwell (adam-harwell) wrote :

Krishna, I see you assigned this to yourself, but it is not marked "In Progress" yet. Are you actively working on this? If not, I can take a look.

Changed in octavia:
status: Triaged → In Progress
Revision history for this message
KRISHNA MOULI. TANKALA (mouli-tankala) wrote :

Yes Adam, currently i am working on this.

Revision history for this message
Zoltan Arnold Nagy (zoltan) wrote :

Any progress updates?

Revision history for this message
KRISHNA MOULI. TANKALA (mouli-tankala) wrote :

yes zoltan, identified fix for this issue.

while testing fix for this, identified another issue which is stopping Loadbalancer delete itself.

Issue: Loadbalancer provisioning_status is inconsistent between Octavia and Neutron Database during LB create failure. https://bugs.launchpad.net/octavia/+bug/1606842

when an amphora is unreachable to controller worker because of network connectivity, loadbalancer provisioning_state will goes into ERROR state, same is status available in neutron DB.

where as in Octavia DB provisioning_state is stuck to PENDING_CREATE.

since we check provisioning_state in Octavia DB to be ERROR while delete, delete is unsuccessful.

for this issue, i raised another bug as https://bugs.launchpad.net/octavia/+bug/1606842

waiting for this to be triage.

Revision history for this message
li,chen (chen-li) wrote :

Looks like this one is the same as https://bugs.launchpad.net/octavia/+bug/1607502.
And already been fixed by https://review.openstack.org/#/c/348529/

Revision history for this message
Adam Harwell (adam-harwell) wrote :

Yes, I believe this has been resolved. Need to confirm.

Revision history for this message
Adam Harwell (adam-harwell) wrote :

This is very old, and myself and others have verified it is working now. Marking it as fixed.

Changed in octavia:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.