No port found <two uuids> when loadbalancer member shares IP with another instance in Openstack

[Drew Freiberger]

When troubleshooting an issue related to loadbalancer creation for kubernetes-master via the openstack-integrator:loadbalancer relation interface, the following traceback occurred:

unit-openstack-integrator-0: 17:07:19 INFO unit.openstack-integrator/0.juju-log status-set: maintenance: Managing load balancers
unit-openstack-integrator-0: 17:07:19 INFO unit.openstack-integrator/0.juju-log Managing load balancer for kubernetes-master
unit-openstack-integrator-0: 17:07:29 INFO unit.openstack-integrator/0.juju-log Found existing security group openstack-integrator-82ea31e06743-kubernetes-master-members (ec6f1673-638b-4da2-a849-4e61716ab5bd)
unit-openstack-integrator-0: 17:07:34 WARNING unit.openstack-integrator/0.update-status No Port found for 6c6267ab-4eb1-4fb3-b7dc-a1de33b00350 81e2a001-803d-45f8-a5c0-be43e652b436
unit-openstack-integrator-0: 17:07:34 ERROR unit.openstack-integrator/0.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/reactive/openstack.py", line 142, in create_or_update_loadbalancers
    lb = layer.openstack.manage_loadbalancer(request.application_name,
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 172, in manage_loadbalancer
    lb_manager = LoadBalancer.get_or_create(
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 369, in get_or_create
    lb = cls(app_name, port, subnet, algorithm, fip_net, manage_secgrps)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 404, in __init__
    self._try_load_cached_info()
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 663, in _try_load_cached_info
    self._add_member_sg(member)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 632, in _add_member_sg
    if self.member_sg_id not in _openstack(
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 307, in _openstack
    output = _run_with_creds('openstack', *args, '--format=yaml')
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 298, in _run_with_creds
    result = subprocess.run(args,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('openstack', 'port', 'show', '6c6267ab-4eb1-4fb3-b7dc-a1de33b00350 81e2a001-803d-45f8-a5c0-be43e652b436', '--format=yaml')' returned non-zero exit status 1.

This was tracked down by George Kraft as occurring in find_port of the LB Implementation class for Octavia.

https://github.com/juju-solutions/charm-openstack-integrator/blob/a0363d0d103764418e6cf93fbbdbaa0b2b02e55a/lib/charms/layer/openstack.py#L735-L738

The presence of two ports in the above traceback is due to the port query matching any ports with a fixed-ip of one of the LB members. If two separate tenant network subnets share a CIDR, such as 192.168.0.0/24 and both tenant subnets contain a VM on the same IP as one of the kubernetes-master or kubernetes-worker units, there's potential IP space collision.

I recommend limiting the query in this function to the subnet of the LB:

Current filter is: --fixed-ip ip-address={address}
This can be extended to be more accurate with: --fixed-ip subnet={self.subnet},ip-address={address}

To reproduce, deploy kubernetes on top of openstack in a project.
In another project, create a network and subnet with the same same CIDR as the kubernetes project's subnet.
In that other project, create a port on the subnet with the same IP address of one of the kubernetes-master units, then add-relation kubernetes-master:loadbalancer to openstack-integrator:loadbalancer, and you will likely witness the above traceback.

PR incoming.

[Drew Freiberger (afreiberger)]

https://github.com/juju-solutions/charm-openstack-integrator/pull/57