ML2 doesn't return fixed_ips on a port update with binding

[Carl Baldwin]

I found this yesterday while working on deferred IP allocation for routed networks. However, it isn't unique to deferred port binding. With my deferred IP allocation patch [2], I need to be able to make a port create call [1] without binding information that doesn't allocate an IP address. Then, I need to follow it up with a port update which sends host binding information and allocates an IP address. But, when I do that, the response doesn't contain the IP addresses that were allocated [3]. However, immediately following it with a GET on the same port shows the allocation [4].

This doesn't happen in other plugins besides ML2. Only with ML2. I've put up a patch to run unit tests with ML2 that expose this problem [5]. The problem can be reproduced on master [6]. I can get it to happen by creating a network without a subnet, creating a port on the network (with no IP address), and then calling port update to allocate an IP address.

If this goes unaddressed, Nova will have to make a GET call after doing a port update with binding information when working with a port with deferred IP allocation.

[1] http://paste.openstack.org/show/505419/
[2] https://review.openstack.org/#/c/320631/
[3] http://paste.openstack.org/show/505420/
[4] http://paste.openstack.org/show/505421/
[5] http://logs.openstack.org/57/320657/2/check/gate-neutron-python27/153a619/testr_results.html.gz
[6] https://review.openstack.org/321152

[Carl Baldwin (carl-baldwin)]

I marked this High because we need to start doing deferred IP allocation when host binding information is provided on port update. This is for routed networks.

[Carl Baldwin (carl-baldwin)]

Related patches:

https://review.openstack.org/#/c/320657
   -> deferred IP allocation with notes about what is going wrong
https://review.openstack.org/#/c/321152
   -> minimal unit test exposing the problem on master

[Hong Hui Xiao (xiaohhui)]

I will look into it today.

[Hong Hui Xiao (xiaohhui)]

The UT at [1] fails because there is no mechanism driver is registered in the test case, so the _bind_port_level and [2] return None, which cause the bind_port fail.

I can eliminate the failed UT by adding appropriate mechanism driver to the test code. Also, I can't reproduce the problem in my local env[3]. So I guess the problem here is the fixed_ip is not returned when ml2 fail to bind port.

But should it be returned? I will continue working on it.

[1] https://review.openstack.org/321152
[2] https://github.com/openstack/neutron/blob/cd71dd0eac675faf7bf3895a5fe9734e740322c6/neutron/plugins/ml2/managers.py#L720
[3] http://paste.openstack.org/show/505514/

[Hong Hui Xiao (xiaohhui)]

I think there is a issue, but I don't know the root cause.

When bind port fail, the code will hit [1] -> [2], but at [2], the port_db will have empty fixed_ips. I debug and found the corresponding IPAllocation is in DB at [2]. So, the problem here is that the object has relationship with Port(IPAllocation in this case) is not returned by db.

Why NeutronDbPluginV2 won't have the same problem? I debug and set breakpoint at [3]. The same problem will occur. I try to get port at [3], the port will be returned without fixed_ips. However, corresponding IPAllocation could be found. NeutronDbPluginV2 will not appear to have this problem is that fixed_ips is push to the port_dict at [4].

I guess it might be because the db update transaction should be alone(just wild guess). So I make some change to the code at [5]. At least the UT success in my local env.

[1] https://github.com/openstack/neutron/blob/cd71dd0eac675faf7bf3895a5fe9734e740322c6/neutron/plugins/ml2/plugin.py#L353
[2] https://github.com/openstack/neutron/blob/cd71dd0eac675faf7bf3895a5fe9734e740322c6/neutron/plugins/ml2/plugin.py#L426
[3] https://github.com/openstack/neutron/blob/cd71dd0eac675faf7bf3895a5fe9734e740322c6/neutron/db/db_base_plugin_v2.py#L1306
[4] https://github.com/openstack/neutron/blob/cd71dd0eac675faf7bf3895a5fe9734e740322c6/neutron/db/db_base_plugin_v2.py#L1308-L1310
[5] https://review.openstack.org/#/c/321152/3//COMMIT_MSG

[Carl Baldwin (carl-baldwin)]

Hong Hui Xiao, you're probably right about binding failing in the unit test. However, when I first discovered this issue, it was on a system with a registered mechanism driver where binding succeeded. The original PUT that I referenced in the description showed this [3]. You can see in that response that binding succeeded and the IP allocation was not returned in the response.

So, if you register a mechanism driver, the unit test passes? That is odd. It might mean that I need to tweak the unit test a bit more to expose the original issue.

Even when port binding fails and an IP allocation is made, it should be returned. You might argue that the IP shouldn't be allocated in the first place, but it is. That would be a different bug, I think.

[3] http://paste.openstack.org/show/505420/

[Carl Baldwin (carl-baldwin)]

I played with this a little bit more. It is definitely reproducible on master [1]. I did most of this with the neutron CLI but I couldn't figure out how to do the port update with the CLI so I did it with a REST client and captured the HTTP conversation and pasted it. You can clearly see that binding succeeded and the fixed_ips are not returned with the result. But, they are available using port show immediately after.

The fixed_ips need to be returned with port update response.

[1] http://paste.openstack.org/show/507056/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/321152
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fcd33b31282848c5853cebc37e1e8e0a06e878f2
Submitter: Jenkins
Branch: master

commit fcd33b31282848c5853cebc37e1e8e0a06e878f2
Author: Hong Hui Xiao <email address hidden>
Date: Tue Jun 7 15:35:32 2016 -0600

    Return fixed_ips from port update

    fixed_ips has a join relationship with port. During a DB transaction,
    it will be acquired once. If the IPAllocation is updated after
    acquiring fixed_ips, the port DB model object will still have the
    stale value of fixed_ips, which is empty in the case of reported bug.

    Expire the fixed_ips of the port can make the next query of port have
    the latest fixed_ips, and thus, has the latest fixed_ips in the result
    of update port.

    Change-Id: I8ca52caf661daa2975cf53212d008eb953d83cc0
    Closes-Bug: #1585738

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 9.0.0.0b2 development milestone.