Unable to update external network subnet's gateway-ip

[Vishal Agarwal]

I am trying below scenario:-
1. Create one external network i.e. with router:external=True option.
2. Create one subnet under the above network with gateway-ip provided.
3. Create one router.
4. Issue command “neutron router-gateway-set <routerID-point3 above> <networkID-point1 above>”
5. Update the subnet in point2 above with new gateway IP i.e “neutron subnet-update <subnetID-point2> --gateway-ip <newIP>”
6. I can see success-full subnet updated response on cli.
7. For validating the changed gateway-ip I verified router namespace present on Network node by using command “ip netns exec <router-namespace-point3> route -n”. But in the output the new gateway-ip is not updated it is still showing the old one.

Brief about my setup:-
1. It has one controller node, one Network node and 2 Compute nodes.
2. I am on Icehouse-GA.

If the Subnet update API does not have a constraint to prevent the change then the L3 agent should affect it properly.

[Salvatore Orlando (salvatore-orlando)]

Assigned for triaging.

[Salvatore Orlando (salvatore-orlando)]

The root cause of this bug is fairly straightforward.
Subnet updates don't trigger updates on router. A similar situation happens when a subnet's gateway is changed.

As a workaround, removing and then adding the external network for the gateway should fix the configuration.
I would advice against short term fixes such as notifying routers on updates for subnet attached to external networks.
It is perhaps better to rework how neutron handles router ports to ensure they are always in sync with the configuration of the subnet they're dependent on.

There is already work in progress in this direction.

[Carl Baldwin (carl-baldwin)]

Salvatore, I noticed the lack of notifications when I was working on adding subnets to external gateway [1]. I have a task in my backlog to add this notification. It is a bit different but maybe there is enough in common that both problems could be fixed together.

[1] https://review.openstack.org/#/c/90250/

[Eugene Nikanorov (enikanorov)]

Looks like it worth increasing importance to make this bug more visible

[KaiLin (linkai3)]

Salvatore,I check the bug in Juno,and when I execute the command: neutron subnet-update <subnetID-point2> --gateway-ip <newIP>, the cli response is: "Invalid input for operation: Gateway is not valid on subnet. (HTTP 400) ", I can't see success-full subnet updated response on cli.

[Felipe Alfaro Solana (felipe-alfaro-gmail)]

As far as I know, the CLI error "Gateway is not valid on subnet" usually happens when Neutron is configured to force the gateway to be on the same network it's serving but the gateway specified when creating or updating a subnet lies outside that IP network.

This behavior is controlled by means of the force_gateway_on_subnet setting in /etc/neutron/neutron.conf. In Havana, I think the default used to be False, but the default has been changed to True in Icehouse, I believe.

[bharaththiruveedula (bharath-ves)]

I am unable to reproduce this bug. The steps I have done to reproduce:
1)I have created with network with external = True
2)Created a subnet for the above network with CIDR=172.24.4.0/24 with gateway-ip =172.24.4.5
3)Created the router
4)Set the gateway interface to the router
5)Tried to change subnet gateway-ip but got this error"Gateway ip 172.24.4.7 conflicts with allocation pool 172.24.4.6-172.24.4.254"
I used this command for that"neutron subnet-update ff9fe828-9ca2-42c4-9997-3743d8fc0b0c --gateway-ip 172.24.4.7"
Can you please help me with this issue?

[Andreas Scheuring (andreas-scheuring)]

The error messages says, that the 24.7 IP adress is within the allocation pool (24.6 - 24.254) that is used for floating ip and so on. Just pick a new gateway adress that is not in this range. (e.g. 24.4)

[bharaththiruveedula (bharath-ves)]

I have tried that, but now it says "Gateway ip 172.24.4.4 conflicts with allocation pool 172.24.4.1-172.24.4.4"

[Itsuro Oda (oda-g)]

bharaththiruveedula, gateway-ip must be outside of allocation pools.

I confirmed this bug report still valid. reproduction example is as follows:
---
$ neutron net-create test-g --router:external True
...
$ neutron subnet-create --name test-g --allocation-pool start=120.0.0.10,end=120.0.0.254 test-g 120.0.0.0/24
...
| gateway_ip | 120.0.0.1 |
...
$ neutron router-create test-r
...
| id | 8cd87e18-6da6-4cbb-a341-f9846cc7c446 |
...
$ neutron router-gateway-set test-r test-g
Set gateway for router test-r
$ sudo ip netns exec qrouter-8cd87e18-6da6-4cbb-a341-f9846cc7c446 route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 120.0.0.1 0.0.0.0 UG 0 0 0 qg-ad805ab5-3b
120.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-ad805ab5-3b
$ neutron subnet-update test-g --gateway_ip 120.0.0.2
Updated subnet: test-g
$ neutron subnet-show test-g
...
| gateway_ip | 120.0.0.2 |
...
$ sudo ip netns exec qrouter-8cd87e18-6da6-4cbb-a341-f9846cc7c446 route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 120.0.0.1 0.0.0.0 UG 0 0 0 qg-ad805ab5-3b
120.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-ad805ab5-3b
$
---

This is because subnet update does not trigger any notification to l3-agent. So l3-agent never notice the gateway ip is changed.

[Tomoko Inoue (inoue-tomoko)]

bharaththiruveedula,

we can't specify duplicate IP address with one subnet-update CLI.

For example)

precondition:
 gateway_ip 172.24.1.1
 allocation_pools {"start": "172.24.1.2", "end": "172.24.1.254"}

$neutron subnet-update public-subnet --allocation-pool start=172.24.1.30,end=172.24.1.254 --gateway 172.24.1.5

->
"Gateway ip 172.24.1.5 conflicts with allocation pool 172.24.1.2-172.24.1.254"

But we can specify the gateway-ip in the following steps(two steps).

$neutron subnet-update public-subnet --allocation-pool start=172.24.1.30,end=172.24.1.254
Updated subnet: public-subnet

$neutron subnet-update public-subnet --gateway 172.24.1.5
Updated subnet: public-subnet

As a result, we can reproduce this problem(Please see note-10).

[bharaththiruveedula (bharath-ves)]

Tomoko Inoue, Itsuro Oda

After updating the subnet with new gateway IP, we can't see the changes in ip r. But if we do router set-gateway again we can see the changes in ip r.

[Tomoko Inoue (inoue-tomoko)]

bharaththiruveedula,
thank you for your comments.
I know that workaround(note-2 and your note-12), but...
Even though we can specify gateway_ip with subnet-update API(20x OK), ip r is not updated.
Operator/user can not detect it until checking ip r or packet loss.
I think that's a problem.

[Hirofumi Ichihara (ichihara-hirofumi)]

I caught the bug, then I confirmed it. The problem hasn't been solved yet.

[Kyle Mestery (mestery)]

Moving to medium priority and targeting Liberty-1.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/194939
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d02bcb9c3917028948b08c319d1443d487c36846
Submitter: Jenkins
Branch: master

commit d02bcb9c3917028948b08c319d1443d487c36846
Author: Hirofumi Ichihara <email address hidden>
Date: Tue Aug 25 09:10:00 2015 +0900

    Enable to update external network subnet's gateway-ip

    This patch enables users to update gateway_ip of a subnet even if
    the subnet is in use for an external network of a router.

    Change-Id: I78d2b024c99b1af0001bd454465d2fc02692cbf2
    Closes-Bug: #1317363

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224334

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

[OpenStack Infra (hudson-openstack)]

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334