Neutron fails to create external network gateway when gateway's IP in different subnet with br-ex

[Hoang Do]

Hi guys,

I encountered a problem with neutron when trying to create external network with gateway in different subnet:

neutron subnet-create ext-net --name ext-subnet \
  --allocation-pool start=46.105.252.216,end=46.105.252.219\
  --disable-dhcp --gateway 176.31.105.254 46.105.252.0/24

The external network has gateway in different subnet: 46.105.252.216/24 and 176.31.105.254
I need something like this due to the router configuration in DC.

The problem is neutron show no error, and on dashboard, the ext-net also shows its gateway 176.31.105.254. However, packets are not routed because in IP routing table of the router, no default gateway entry is added:

sudo ip netns exec qrouter-f918cbb7-dc0c-4713-a6f5-3c66b46e12cf route -n

Destination Gateway Genmask Flags Metric Ref Use Iface

46.105.252.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-0103d6fa-31
192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-343ab2cb-f5

I can work around by manually adding two line in routing table:

Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 176.31.105.254 0.0.0.0 UG 0 0 0 qg-0103d6fa-31
46.105.252.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-0103d6fa-31
176.31.105.254 0.0.0.0 255.255.255.255 UH 0 0 0 qg-0103d6fa-31
192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-343ab2cb-f5

Then it worked fine!
I believe this is a bug, due to adding gateway with different subnet in routing table will be rejected. In this case, we need to add this line first before adding gateway:

176.31.105.254 0.0.0.0 255.255.255.255 UH 0 0 0 qg-0103d6fa-31

So either we need to show users an error "not allow to add gateway in different subnet", or we should support adding gateway properly.

[Assaf Muller (amuller)]

Check out this patch:
https://review.openstack.org/#/c/92619/

neutron.conf has an option that validates that the gateway is in the subnet range. It's turned off by default and the patch above actually turns it on.

[Akihiro Motoki (amotoki)]

What I am not sure is a usecase of gateway IP out of the subnet. The use case described in this bug report is a possible case but I am not sure it is a special case or not. I don't have a good answer to this question so far and I don't see this use case in my personal experience. It is very special case, it sounds reasonable that neutron does not allow to specify a gateway IP out of the subnet. Does anyone have an answer?

[Assaf Muller (amuller)]

The reason I made Neutron enforce the 'gateway in subnet' rule the patch in comment 1 is that our agents don't actually support that operation, and the server/API layer didn't state as such. So you could configure a gateway out of the subnet, and just as this bug complains, the server happily accepted it but the agent failed to make the configuration.

However, it's quite easy to fix this issue in the agents. Just add a route to the gateway subnet before adding the gateway route, and there you go.

I think that a patch that fixes the issue in the agents can then change the default value for force_gateway_on_subnet back to False.

[Hoang Do (dohuyhoang88)]

Hi Akihiro, I just want to provide my use case. We deploy OpenStack on OVH dedicated server (in France). Then we have to purchase a range of public IPs (in different subnet range with the host) as floating IP. Indeed we need this feature.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114185

[OpenStack Infra (hudson-openstack)]

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/114185
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/233287

[Sreekumar S (sreesiv)]

Can someone please review the changes? Some CI tests are failing, but I've verified that for other open requests also they've failed, and the their logs doesn't suggest any issue related to the current changes being submitted.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/277303

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/233287
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b6126bc0f17f348ff6303b9bd6041cb018479ef9
Submitter: Jenkins
Branch: master

commit b6126bc0f17f348ff6303b9bd6041cb018479ef9
Author: Sreekumar S <email address hidden>
Date: Sat Oct 10 03:18:00 2015 +0530

    Fix for adding gateway with IP outside subnet

    Currently 'force_gateway_on_subnet' configuration is set to True
    by default and enforces the subnet on to the gateway. With this
    fix 'force_gateway_on_subnet' can be changed to False, and
    gateway outside the subnet can be added.
    Before adding the default route, a route to the gateway IP is
    added. This applies to both external and internal networks.

    This configuration option is deprecated, and should be removed
    in a future release. It should always allow gateway outside the
    subnet. This is done as a separate patch
    https://review.openstack.org/#/c/277303/

    Change-Id: I3a942cf98d263681802729cf09527f06c80fab2b
    Closes-Bug: #1335023
    Closes-Bug: #1398768

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/283030

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277303
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3e6d602f542fc97e64cd5503bf1ea4e71d648abf
Submitter: Jenkins
Branch: master

commit 3e6d602f542fc97e64cd5503bf1ea4e71d648abf
Author: Sreekumar S <email address hidden>
Date: Mon Feb 8 12:58:01 2016 +0530

    Deprecate 'force_gateway_on_subnet' configuration option

    Currently 'force_gateway_on_subnet' configuration is set to True
    by default and enforces the subnet on to the gateway. With the
    fix in https://review.openstack.org/#/c/233287/, gateway outside
    the subnet can be added, and the configuration option now has
    lost its significance.

    With this patch, the configuration option is deprecated.
    It should be removed in Newton release, and the system should
    always allow gateway outside the subnet.
    This patch is dependent on the fix for adding gateway outside
    the subnet, mentioned above.

    DocImpact: 'force_gateway_on_subnet' description should be
    updated in the docs and marked as deprecated to be removed in
    the Newton release.

    Change-Id: I28b3d7add303ee479fc071c1de142b0f7811e4e5
    Closes-Bug: #1335023
    Closes-Bug: #1398768

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/283415

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/283030
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9e11c5b8516c42000d5409c0f44f852acb5fed3d
Submitter: Jenkins
Branch: stable/liberty

commit 9e11c5b8516c42000d5409c0f44f852acb5fed3d
Author: Sreekumar S <email address hidden>
Date: Sat Oct 10 03:18:00 2015 +0530

    Fix for adding gateway with IP outside subnet

    Currently 'force_gateway_on_subnet' configuration is set to True
    by default and enforces the subnet on to the gateway. With this
    fix 'force_gateway_on_subnet' can be changed to False, and
    gateway outside the subnet can be added.
    Before adding the default route, a route to the gateway IP is
    added. This applies to both external and internal networks.

    Change-Id: I3a942cf98d263681802729cf09527f06c80fab2b
    Closes-Bug: #1335023
    Closes-Bug: #1398768
    (cherry picked from commit b6126bc0f17f348ff6303b9bd6041cb018479ef9)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.

[Armando Migliaccio (armando-migliaccio)]

The patch that introduced the deprecation: https://review.openstack.org/#/c/277303/

[Miguel Angel Ajo (mangelajo)]

Thanks for working on the on-link route support.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.0.4 release.