Ran into this with the new nova-multi-cell job:
http://logs.openstack.org/22/655222/3/check/nova-multi-cell/c1ba7e0/controller/logs/screen-n-api.txt.gz#_Apr_25_20_35_15_249786
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips [None req-3b6da167-cd81-4f88-a27b-05876cf4e6f4 tempest-FloatingIPsAssociationTestJSON-110201909 tempest-FloatingIPsAssociationTestJSON-110201909] Unable to associate floating IP 172.24.5.15 to fixed IP 10.1.0.10 for instance be0ea845-25df-4528-8fcf-c27835f41636. Error: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.: nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips Traceback (most recent call last):
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/api/openstack/compute/floating_ips.py", line 266, in _add_floating_ip
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips fixed_address=fixed_address)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/network/base_api.py", line 83, in wrapper
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips res = f(self, context, *args, **kwargs)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2401, in associate_floating_ip
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips orig_instance_uuid)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips result = fn(cls, context, *args, **kwargs)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/objects/instance.py", line 505, in get_by_uuid
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips use_slave=use_slave)
Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 213, in wrapper
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips return f(*args, **kwargs)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/objects/instance.py", line 497, in _db_instance_get_by_uuid
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips columns_to_join=columns_to_join)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/api.py", line 758, in instance_get_by_uuid
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 171, in wrapper
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips return f(*args, **kwargs)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 258, in wrapped
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips return f(context, *args, **kwargs)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1827, in instance_get_by_uuid
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips columns_to_join=columns_to_join)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1836, in _instance_get_by_uuid
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips raise exception.InstanceNotFound(instance_id=uuid)
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.
Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 <email address hidden>[19571]: ERROR nova.api.openstack.compute.floating_ips
Due to this old code:
https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L2400
https://review.opendev.org/#/c/33054/
The idea there is if you're associating a floating IP from one server to another, we try to refresh the network info cache on the instance that previously had the floating IP so it's not stale and you have 2 instances reporting they have the same floating IP until the cache is cleaned up.
The problem is in a multi-cell environment the servers could be in different cells and the context is targeted at the cell we're assigning the floating IP *to* but will fail to lookup the instance in the other cell that the floating IP is coming *from* (the original instance).
This results in a 400 error in the API (I'm surprised it's not returning a 404 or 500 actually):
https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/api/openstack/compute/floating_ips.py#L282
I'm not really sure what we should do in this case. We could capture the InstanceNotFound here:
https:/ /github. com/openstack/ nova/blob/ a991980863f0563 23c1ee9fd6a46db c4cb899eca/ nova/network/ neutronv2/ api.py# L2400
And arguably should have been doing that anyway, but then do we ignore it as a best-effort kind of thing? That would mean the info cache on the original instance (if it does exist in another cell) would be stale until the _heal_instance_ info_cache periodic runs in that other cell.
We could handle the InstanceNotFound and attempt to lookup the InstanceMapping for that instance and if found, target the cell it's in to get the instance and refresh it's cache. It looks like that should work in the neutron case because associate_ floating_ ip is only called from the API so it'd have access to the API DB to lookup the InstanceMapping.