Displaying of floating IPs takes a lot of time

[Alexander Rubtsov]

Release: Mirantis OpenStack 4.0

Steps to reproduce:
Launch instances (as many as you can). Assign floating IPs to these instances.
Check how much time is needed for displaying all of them in Horizon

Expected result:
Information about floating IPs appears immediately after assignment to an instance

Actual result:
There is a delay between floating IPs assignment and displaying they both in Horizon and "nova list"

There is the more detailed description:
Network information is placed in "InstanceInfoCache" entry in the database
Nova compute runs periodic task for updating network information every 60 seconds (by default)
But due to implementation of task, only one instance can be processed in each run.
So every new instance on node will increase update period for specific instance by 1 minute

There is a snippet from nova-compute.log
2014-05-23 02:04:47 DEBUG nova.compute.manager [req-fb3efb94-0ebf-4633-b10a-160ed4914847 None None] [instance: 3de0cbfd-b92b-412c-bd4c-d8d59d617055] Updated the info_cache for instance _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4364
2014-05-23 02:05:50 DEBUG nova.compute.manager [-] [instance: 0d6fd7ae-a84a-4ed2-9c14-0342531f10f2] Updated the info_cache for instance _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4364
2014-05-23 02:06:53 DEBUG nova.compute.manager [-] [instance: 4b63b037-c617-436c-9522-a812e80b8026] Updated the info_cache for instance _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4364

The configuration option "heal_instance_info_cache_interval" in the nova.conf is intended to changing frequency of this periodic task. But this option doesn't work due to this bug: https://bugs.launchpad.net/nova/+bug/1274317
Anyway, applying of patch to fix this bug is just workaround that is able to reduce delay.
Also, if the interval is too small, it can lead to some nova-compute service are sporadically going down (It's not confirmed that there is the root cause, but issue appeared after changing the value of "heal_instance_info_cache_interval" to 20 seconds)

I verified how it works in Mirantis OpenStack 5.0
There are the following messages are writed in the nova-compute.log every minute:
<183>May 30 17:40:28 node-1 nova-nova.openstack.common.periodic_task DEBUG: Running periodic task ComputeManager._heal_instance_info_cache
<183>May 30 17:40:28 node-1 nova-nova.compute.manager DEBUG: Starting heal instance info cache
<183>May 30 17:40:28 node-1 nova-nova.compute.manager DEBUG: Rebuilding the list of instances to heal
It seems this logic eliminates subsequent info cache update for each instance
Information about Floating IPs appears in Horizon IMMEDIATELY (before next info cache update happens)
Also I've checked that the value of "heal_instance_info_cache_interval" can be configured out-of-box (without any patches)

Would you please backport code from 5.0 to the previous versions (4.x)

[Alexander Dobdin (sanek-dobdin)]

Also, backport to 4.0 is required

[Timur Sufiev (tsufiev-x)]

Folks, that seems to me like a Nova changes backport, am I right? The reason I'm asking is that since the creation of Core Components team I'm responsible for Horizon component in MOS. So if it's a Horizon backport, it's my task then.

[Mike Durnosvistov (glacierrdev)]

I've tried to reproduce that bug on OS Icehouse and MOS 5.0.1 but `nova list` displayed `floating-ip` normally.

[Alexander Rubtsov (arubtsov)]

Mikhail,
Regarding MOS 5.0.x - you are right
I mentioned it in the initial message, but I tested only dashboard (... Information about Floating IPs appears in Horizon immediately ...)

[Roman Podoliaka (rpodolyaka)]

This is fixed in icehouse, so 5.0+ releases are not affected. Marked invalid for those releases. Keep this opened just in case we decide to release 4.1.2 (for some reason I can't nominate the bug for that series).

[Dmitry Mescheryakov (dmitrymex)]

Moving to confirmed for 4.1.2 judging by Roman's comment