Trying to create a load balancer while no avaliable compute node ends with a non-deletable loadbalancer

[Miguel Angel Ajo]

1) kill all nova computes in your environment

2) create a load balancer,

After load balancer creation timeout, you see in o-cw.log

2016-04-07 13:54:30.256 8241 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/controller/worker/tasks/compute_tasks.py", line 184, in execute
2016-04-07 13:54:30.256 8241 ERROR oslo_messaging.rpc.dispatcher raise exceptions.ComputeWaitTimeoutException()

Then, when you try to delete the loadbalancer:

vagrant@devstack:~$ neutron lbaas-loadbalancer-list
+--------------------------------------+------+-------------+---------------------+----------+
| id | name | vip_address | provisioning_status | provider |
+--------------------------------------+------+-------------+---------------------+----------+
| 9263305b-d4a2-4e27-b31f-c9e687bbed90 | | 10.0.0.24 | PENDING_CREATE | octavia |
+--------------------------------------+------+-------------+---------------------+----------+

vagrant@devstack:~$ neutron lbaas-loadbalancer-delete 9263305b-d4a2-4e27-b31f-c9e687bbed90
Invalid state PENDING_CREATE of loadbalancer resource 9263305b-d4a2-4e27-b31f-c9e687bbed90

loadbalancer is left in PENDING_CREATE forever.

[Lingxian Kong (kong)]

I encountered the same problem, create a loadbalancer, something error occured in o-cw process(e.g. kill the process unexpectedly), after query timeout in octavia driver side, loadbalancer status in neutron db is ERROR, loadbalancer status in octavia is PENDING_CREATE, then delete that lb, exception will be raised in octavia side.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/309253

[Miguel Angel Ajo (mangelajo)]

@kong, how's https://review.openstack.org/#/c/309253 supposed to help here?

Cheers, and thank you.

[Lingxian Kong (kong)]

Hi, Miguel, that patch doesn't intend to fix this bug totally, but give a way to solve it. I answered the same question that German asked, paste it here:

Currently, it may stuck for the resourses we created because of the asynchronous communication between neutron and octiava-api, and octavia-worker, then we would see the inconsistency of resource status between neutron and octavia. If that happens, we could do nothing according to current design, because there are not any hints for operators to decide which resource in the temporary status in octavia(e.g. PENDING_CREATE) will never be 'active'. Adding timestamp to resources at least give a way to support operators make a dicision, e.g. they could say, 'This load balancer has been in PENDING_CREATE status for an more than one hour, let's delete it.'

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309253
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=1c67df237014c2fc503d448b90b87504019d3af1
Submitter: Jenkins
Branch: master

commit 1c67df237014c2fc503d448b90b87504019d3af1
Author: Lingxian Kong <email address hidden>
Date: Thu Apr 21 16:19:33 2016 +1200

    Add timestamp to octavia resources

    Add 'created_at' and 'updated_at' columns to octavia database resources,
    oslo.db will be used to implement that. Also add 'created_at' and
    'updated_at' to the API response of loadbalancer/listener/pool/member.

    Closes-Bug: #1573287
    Related-Bug: #1496646
    Related-Bug: #1567463

    Change-Id: I422805e49b9658d7eaa7d5b3b18a30a0f38d7050

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/332704
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=bc5b2d0e4676437b19c72d9b9385d12e4c1f4d52
Submitter: Jenkins
Branch: master

commit bc5b2d0e4676437b19c72d9b9385d12e4c1f4d52
Author: chen-li <email address hidden>
Date: Thu Jun 23 03:19:23 2016 -0500

    Add revert method in database_tasks MapLoadbalancerToAmphora

    Partial-Bug: #1567463

    Change-Id: I8c79d418d9523f8c3ccdabed9a89a40982c7b25a

[OpenStack Infra (hudson-openstack)]

Change abandoned by li,chen (<email address hidden>) on branch: master
Review: https://review.openstack.org/333800
Reason: In the discussion of the lbaas weekly meeting(8/3), we decide to abandon this change.

For these related bugs, we need:
1. Make sure tasks in flow will have their own timeout.
2. use the job board: http://docs.openstack.org/developer/taskflow/persistence.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/372008

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/372008
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=58ca8187ddd8537e3af48042d5d4573009d40414
Submitter: Jenkins
Branch: stable/mitaka

commit 58ca8187ddd8537e3af48042d5d4573009d40414
Author: chen-li <email address hidden>
Date: Thu Jun 23 03:19:23 2016 -0500

    Add revert method in database_tasks MapLoadbalancerToAmphora

    Partial-Bug: #1567463

    Change-Id: I8c79d418d9523f8c3ccdabed9a89a40982c7b25a
    (cherry picked from commit bc5b2d0e4676437b19c72d9b9385d12e4c1f4d52)

[Miguel Angel Ajo (mangelajo)]

li,chen, are you still looking into this bug, is it complete?

[Michael Johnson (johnsom)]

This should be fully fixed now. The LB should go to ERROR state, which can be deleted.

https://review.openstack.org/#/c/372791/