Failover fails when the amphora instance is stoped

[li,chen]

Steps to re-produce

1. Configure octvia with Active-Standby
2. Create a loadbalancer
3. stop one amphora with command 'nova stop'

Get error log:

2016-08-22 00:13:54.197 19904 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X POST http://10.110.67.85:8774/v2.1/servers/364b639e-c7df-4d7f-97da-e75b6a60e1c6/os-interface -H "User-Agent: python-novaclient" -
H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.1" -H "X-Auth-Token: {SHA1}d5cbbd21825f4b25a107e0e5fedb07707ab43b2a" -d '{"interfaceAttachment": {"port_id":
 "e1e953ad-14fc-4f8d-a7c3-927c0abfa983"}}' _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:337
2016-08-22 00:13:55.271 19904 DEBUG novaclient.v2.client [-] RESP: [200] Content-Length: 283 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-
API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-013c11de-cf2b-4a87-88b5-6ea53fa39410 Date: Mon, 22 Aug 2016 05:13:55 GMT Connection: keep-alive
RESP BODY: {"interfaceAttachment": {"port_state": "ACTIVE", "fixed_ips": [{"subnet_id": "52aa4161-34e0-42a3-981e-aeae350f1f65", "ip_address": "192.168.2.14"}], "port_id": "e1e953ad-14fc-4f8d-a7c3-927c0abfa983", "net_id": "54ee02e1-f429-4f01-938a-e81386138cfa", "mac_addr": "fa:16:3e:56:bf:07"}}
 _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:366
2016-08-22 00:13:55.272 19904 DEBUG novaclient.v2.client [-] POST call to compute for http://10.110.67.85:8774/v2.1/servers/364b639e-c7df-4d7f-97da-e75b6a60e1c6/os-interface used request id req-013c11de-cf2b-4a87-88b5-6ea53fa39410 _log_request_id /usr/lib/python2.7/site-packages/novaclient/client.py:85
2016-08-22 00:13:55.272 19904 DEBUG octavia.controller.worker.tasks.network_tasks [-] Plugging VIP port ID: e17b6d7b-a43d-447f-b680-712e87e284e6 into compute instance: 364b639e-c7df-4d7f-97da-e75b6a60e1c6. execute /opt/stack/octavia/octavia/controller/worker/tasks/network_tasks.py:420
2016-08-22 00:13:55.273 19904 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X POST http://10.110.67.85:8774/v2.1/servers/364b639e-c7df-4d7f-97da-e75b6a60e1c6/os-interface -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.1" -H "X-Auth-Token: {SHA1}d5cbbd21825f4b25a107e0e5fedb07707ab43b2a" -d '{"interfaceAttachment": {"port_id": "e17b6d7b-a43d-447f-b680-712e87e284e6"}}' _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:337
2016-08-22 00:13:55.416 19904 DEBUG novaclient.v2.client [-] RESP: [400] Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 147 X-Compute-Request-Id: req-09f394a5-89b4-47b3-843f-4b4e4604d043 Date: Mon, 22 Aug 2016 05:13:55 GMT Connection: keep-alive
RESP BODY: {"badRequest": {"message": "Port e17b6d7b-a43d-447f-b680-712e87e284e6 not usable for instance 364b639e-c7df-4d7f-97da-e75b6a60e1c6.", "code": 400}}
 _http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:366
2016-08-22 00:13:55.416 19904 DEBUG novaclient.v2.client [-] POST call to compute for http://10.110.67.85:8774/v2.1/servers/364b639e-c7df-4d7f-97da-e75b6a60e1c6/os-interface used request id req-09f394a5-89b4-47b3-843f-4b4e4604d043 _log_request_id /usr/lib/python2.7/site-packages/novaclient/client.py:85
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] Error plugging amphora (compute_id: 364b639e-c7df-4d7f-97da-e75b6a60e1c6) into port e17b6d7b-a43d-447f-b680-712e87e284e6.
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs Traceback (most recent call last):
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/opt/stack/octavia/octavia/network/drivers/neutron/allowed_address_pairs.py", line 482, in plug_port
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs fixed_ip=None, port_id=port.id)
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1799, in interface_attach
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs body, 'interfaceAttachment')
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 356, in _create
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs resp, body = self.api.client.post(url, body=body)
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 193, in post
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs return self.request(url, 'POST', **kwargs)
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 117, in request
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs raise exceptions.from_response(resp, body, url, method)
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs BadRequest: Port e17b6d7b-a43d-447f-b680-712e87e284e6 not usable for instance 364b639e-c7df-4d7f-97da-e75b6a60e1c6. (HTTP 400) (Request-ID: req-09f394a5-89b4-47b3-843f-4b4e4604d043)
2016-08-22 00:13:55.416 19904 ERROR octavia.network.drivers.neutron.allowed_address_pairs
2016-08-22 00:13:55.431 19904 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.network_tasks.PlugVIPPort' (a2653d70-0c10-4970-8e82-5aa4ffee2947) transitioned into state 'FAILURE' from state 'RUNNING'
41 predecessors (most recent first):

[li,chen (chen-li)]

Did a little more debug, found error in nova-compute:

http://paste.openstack.org/show/562160/

The error message in nova-compute indicate that port['tenant_id'] != instance.project_id.
And port['tenant_id'] is actually the project I used for creating lb while the instance.project_id is the project 'admin'.

And, based on the above log, octavia is trying to attach port 'e17b6d7b-a43d-447f-b680-712e87e284e6'. Show the port by neutron command, and we get the port name is:

 neutron port-show e17b6d7b-a43d-447f-b680-712e87e284e6

| name | loadbalancer-df14f332-deac-43a4-b433-968324a3d3ae |

This indicate this is the port for VIP address.
Based on current logic, this port should not be attached to amphora directly.
Another port should be created, and the vip address should be added as the address pair.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/358450

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/358450
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=d7ac8b10ac77c0b229608aba52d66eeb679f73e8
Submitter: Jenkins
Branch: master

commit d7ac8b10ac77c0b229608aba52d66eeb679f73e8
Author: chen-li <email address hidden>
Date: Mon Aug 22 02:08:08 2016 -0500

    Fix failover

    The vip port is added to the vrrp port as address_pair, so
    we do not need to plug the vip port to amphora.

    Change-Id: I37bf247e9ea94d4efd6fbce6dea6f82d139bd36a
    Closes-Bug: 1615471

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/octavia 0.9.0 release.