Creation of loadbalancer fails with plug vip exception

[Preethi Dsilva]

Here is the scenario:
I have two compete nodes with following bridge mappings:
Compute node 1:
1. physnet3:br-hed0 (This is the octavia-mgt-network)
2. physnet2: br-hed2
Compute node 2:
1. physnet3:br-hed0 (This is the octavia-mgt-network)
2. physnet1:br-hed1
3. physnet2:br-hed2
Now if I create a loadbalancer with VIP in physnet1, the NOVA is scheduling the amphora image on compute node1. However as there is no physnet1 mapping in compute node 1, the octavia is failing to plug the amphora image into VIP network.
Expected result:
Octavia should internally check if the availability zone on which nova is scheduling the amphora image has the mapping for the required VIP network or not.
Here is the VIP network details:
stack@padawan-cp1-c1-m1-mgmt:~/scratch/ansible/next/hos/ansible$ neutron net-show net1
---------------------------------------------------------------+
Field Value
---------------------------------------------------------------+
admin_state_up True
availability_zone_hints
availability_zones nova
created_at 2016-07-29T03:45:02
description
id cd5a5e69-f810-4f08-ad9f-72f6184754af
ipv4_address_scope
ipv6_address_scope
mtu 1500
name net1
provider:network_type vlan
provider:physical_network physnet1
provider:segmentation_id 1442
router:external False
shared False
status ACTIVE
subnets 115f7f23-68e2-4cba-9209-97d362612a7f
tags
tenant_id 6b192dcb6a704f72b039d0552bec5e11
updated_at 2016-07-29T03:45:02
---------------------------------------------------------------+
stack@padawan-cp1-c1-m1-mgmt:~/scratch/ansible/next/hos/ansible$
Here is the exception from octavia-worker.log:
"/var/log/octavia/octavia-worker.log" [readonly] 2554L, 591063C 1,1 Top
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/octavia/controller/queue/endpoint.py", line 45, in create_load_balancer
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher self.worker.create_load_balancer(load_balancer_id)
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/octavia/controller/worker/controller_worker.py", line 322, in create_load_balancer
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher post_lb_amp_assoc.run()
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 230, in run
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher for _state in self.run_iter(timeout=timeout):
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 308, in run_iter
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher failure.Failure.reraise_if_any(fails)
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher failures[0].reraise()
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher six.reraise(*self._exc_info)
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher result = task.execute(**arguments)
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/octavia/controller/worker/tasks/network_tasks.py", line 279, in execute
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher loadbalancer.vip)
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 278, in plug_vip
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher subnet.network_id)
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/venv/octavia-20160727T193208Z/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 93, in _plug_amphora_vip
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher raise base.PlugVIPException(message)
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher PlugVIPException: Error plugging amphora (compute_id: 85e75aeb-4ce8-4f26-89ed-6b6a9a006b12) into vip network cd5a5e69-f810-4f08-ad9f-72f6184754af.
2016-07-29 03:46:39.043 10434 ERROR oslo_messaging.rpc.dispatcher

The same issue will arise w.r.t spare pools.

This issue is seen in stable/mitaka

[Michael Johnson (johnsom)]

This is working as designed.

Octavia is a load balancing driver that hot-plugs networks requested by users into the load balancer (amphora). This allows the user flexibility in what networks they use for their VIP network and/or their backend member networks. Because of this we cannot predict what networks a user may request plugged into the amphora at nova boot time.

If we took your recommendation and plugged the VIP network in at Nova boot time, the user would run into the same error when they added backend member networks.

To support this end user flexibility, we require the operator to use Nova flavors that allow the amphora access to the user networks such that Octavia can fulfill the user requests for load balancing on those networks.

For this use case, I would recommend that you create a nova flavor that is restricted to hosts that have the required physical networks available. You can specify the nova flavor for Octavia to use in the octavia.conf configuration file.

[Preethi Dsilva (prdsilva)]

Michael Lets consider a scenario, I have
compute1->physnet1
compute2->physnet2
compute3->physnet3

Can we create nova flavor for each of this combination,if so how will amphora map flavor(with multiple flavors) to compute at the time of boot.

With SRIOV/PCIPT feature enabled on the setup there is a very high probability that the user might end up in above situation, as when the NIC is configured for SRIOV/PCIPT then that nic will not be configured for OVS.
So there is a high probability that the amphora image might get spin up on the compute which has the NIC connected on the said physnet but that NIC happens to be SRIOV/PCIPT enabled and not for OVS.In such case can we have multiple falvors in mapped in octavia.conf so that SRIOV/PCIPT computes will not be used for amphora VM boot.

Can you please elaborate how having vip nic being plugged at the time of boot will result in user running into the same error when they add backend member networks.

[Preethi Dsilva (prdsilva)]

Adding member from different subnet can be done using routing to amphora subnet .However at the moment in our setup loadbalancer creation itself fails.This issue may arise in scalability setups as well

[Preethi Dsilva (prdsilva)]

Please ignore the comment3.

We tried adding member with subnet different from that of a LB VIP network ,in which case member subnet port is created in the amphora NS

so we can Still control member addition w.r.t physnets present in the compute where amphoraVM is booted.However issue occurs when booting amphora itself.when there are multiple computes there is no way we can control where the amphora VM gets booted

one more scenario is we have multiple computes(all the physnets are present in all the computes),however when they are used for PCIPT/SRIOV they will not be avaialable for OVS.hence if the amphora lands on these computes Loadbalancer creation fails as it can not plug the VIP .This issue may arise in scalability setups

[Preethi Dsilva (prdsilva)]

Please dont move the bug to invalid without giving proper reason.
This is very valid scenario.We have setups as mentioned below
Compute1-->physnet2-->OVS
compute2-->physnet1-->OVS
compute3-->physnet2-->PCIPT
all three computes carrying OCTAVIA MGMT network.

In this scenario Please let me know how will amphora Plumb the second network?
If LB is created for physnet2 and if amphora is already up in compute2 .we will face VIP related issue.
We can not mandate customer to have same set of physnets on all computes.However if Nova is made aware of physnet in which LB needs to be plumbed at the time of LB creation(not port update later) is what we are asking .