Neutron loading policy file too early

[Vincent Untz]

From time to time, I could see that "neutron net-external-list" was returning something empty for a non-admin user while this is not true on my setup.

After quite some debugging, it turned out that it was failing because of the policy (using the default policy.json, btw):

  "get_network": "rule:admin_or_owner or rule:shared or rule:external"

This was not returning true. Clearly, "rule:external" should have (as a reminder, this rule is: "external": "field:networks:router:external=True")

After more debugging, I found out that the field check was comparing u'True' with True. That is, the "True" that is in rule:external was a unicode, and not a boolean.

Looking at extensions/external_net.py, though, I could see that convert_to was set to attr.convert_to_boolean, so it should have worked.

Adding more debug output, I found out that the policies were loaded before the external-net extension was loaded. Therefore, the convert_to field wasn't available yet, and so the conversion to false wasn't true.

This all happened because there was some handling of an event before the external-net extension was loaded, causing the policy file to be loaded a bit too early.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/73866

[Eugene Nikanorov (enikanorov)]

Could you specify what path leads to early loading of the policy?
This particular issue was fixed several times by fixing code paths that caused policy loading before loading of the extensions.

[Vincent Untz (vuntz)]

Eugene: I'm attaching a log with debug where you can see this happening. I didn't look for the full code path. I cut the log when the net-external extension is loaded. And you can see that there's a log "Loading policies from file" earlier than that.

[Eugene Nikanorov (enikanorov)]

Thanks for the log, Vincent, however it's not clear, which part of the code triggers policy loading before the extensions.
I'm asking because I thought I've fixed these code paths, and it appears that there is some that I have missed.
So It's not clear to me, if your fix solves the issue.

[Vincent Untz (vuntz)]

Eugene: I printed the traceback from the first time the policy is loaded:

  File "/usr/lib64/python2.6/site-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)

  File "/usr/lib64/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
    **args)

  File "/usr/lib64/python2.6/site-packages/neutron/common/rpc.py", line 42, in dispatch
    neutron_ctxt = context.Context(user_id, tenant_id, **rpc_ctxt_dict)

  File "/usr/lib64/python2.6/site-packages/neutron/context.py", line 63, in __init__
    admin_roles = policy.get_admin_roles()

  File "/usr/lib64/python2.6/site-packages/neutron/policy.py", line 417, in get_admin_roles
    init()

  File "/usr/lib64/python2.6/site-packages/neutron/policy.py", line 78, in init
    reload_func=_set_rules)

  File "/usr/lib64/python2.6/site-packages/neutron/common/utils.py", line 60, in read_cached_file
    reload_func(cache_info['data'])

  File "/usr/lib64/python2.6/site-packages/neutron/policy.py", line 90, in _set_rules
    LOG.debug(repr(traceback.format_stack()))

[Vincent Untz (vuntz)]

FWIW, even if we manage to fix this other case of things not happening in the right order, I still believe that resetting the policy is correct (and will protect from weird bugs like this in the future, which can be hard to diagnose).

[Vincent Untz (vuntz)]

Hrm, I see that a recent commit for bug 1254555 might fix this. Will test it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/73866
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d8f8d3af706ded5d6d16fc86e0de89f5c7db581c
Submitter: Jenkins
Branch: master

commit d8f8d3af706ded5d6d16fc86e0de89f5c7db581c
Author: Vincent Untz <email address hidden>
Date: Sun Feb 16 10:33:44 2014 +0100

    Reset the policy after loading extensions

    The loading of extensions might impact how rules are interpreted in the
    policies; for instance, the external-net extension specifies how to
    convert the router:external field of a network (to a boolean). So we
    need to make sure that the policy is recreated afterwards.

    We also need to fix a unit test that assumes that the policy is loaded.

    Change-Id: Ifde13323b1435fc84c6cb24ab6a71de5d67f6b9f
    Closes-Bug: #1280738