Comment 5 for bug 1491307

I've found the root cause of this issue.

The call chain goes like this...

In "./nova/compute/api.py"
add_rules() AND remove_rules() calls
 trigger_rules_refresh() which in turn calls refresh_instance_security_rules() [RPC]

This ends up in ./nova/compute/manager.py
[RPC] _ComputeV4Proxy::refresh_instance_security_rules() which calls ComputeManager::refresh_instance_security_rules(). But the manager's function decorated with '@object_compat' calls decorated_function()
  which calls _load_instance(kwargs['instance']). This fails since there is no 'instance' key there and once again calls _load_instance from the except catch...
        except KeyError:
            args = (_load_instance(args[0]),) + args[1:]

From there it calls objects.Instance._from_db_object() with argument 'expected_attrs=metas' which is initialized to
metas = ['metadata', 'system_metadata']

The db instance dict doesn't have the keys in 'metas' because in trigger_rules_refresh() the sec groups are got from db by joining on the instances column, but it doesn't join on the metadata/system_metadata fields.

This again causes 'KeyError' because when db instance dict is converted to the Instance object, it expects fields that aren't in the dict. So the manager's function do not call refresh_instance_security_rules() on the LibvirtDriver and thereby IptablesFirewallDriver.

This same issue is mentioned in bug 1484738, although the end problems they cause differ. I've verified the fix proposed by Matt Riedemann and it resolves the issue.

More details: https://bugs.launchpad.net/nova/+bug/1484738/comments/11

Why this is not repro'ed in Liberty and why can't those changes be backported: https://bugs.launchpad.net/nova/+bug/1484738/comments/13

More details in commit msg: https://bugs.launchpad.net/nova/+bug/1484738/comments/17

So once the fix for bug 1484738 is merged to stable/kilo, it will resolve this issue as well.