If we set a gateway outside subnet neutron-l3-agent fails when trying to set the route in the virtual router.

Bug #1398768 reported by Miguel Angel Ajo
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Low
Sreekumar S

Bug Description

014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent Traceback (most recent call last):
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 341, in call
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent return func(*args, **kwargs)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 938, in process_router
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent self.external_gateway_added(ri, ex_gw_port, interface_name)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 1318, in external_gateway_added
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent ri.ns_name, preserve_ips)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 1362, in _external_gateway_added
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent preserve_ips=preserve_ips)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 120, in init_l3
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent device.route.add_gateway(gateway)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 395, in add_gateway
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent self._as_root(*args)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 242, in _as_root
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent kwargs.get('use_root_namespace', False))
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 74, in _as_root
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent log_fail_as_error=self.log_fail_as_error)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 86, in _execute
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent log_fail_as_error=log_fail_as_error)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 84, in execute
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent raise RuntimeError(m)
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent RuntimeError:
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-bcfe22ff-049c-4eb9-9b57-235d903ee52f', 'ip', 'route', 'replace', 'default', 'via', '37.187.128.254', 'dev', 'qg-eaa2de59-95']
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent Exit code: 2
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent Stdout: ''
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent Stderr: 'RTNETLINK answers: Network is unreachable\n'
2014-11-30 16:25:10.113 8086 TRACE neutron.agent.l3_agent

This happens because we should set an on-link route to the gateway IP
on the external network interface first.

Once this works, we can remove the "force_gateway_on_subnet" deprecation,
to allow this not to be enforced, as it's a valid use case in many data centers.

Revision history for this message
Miguel Angel Ajo (mangelajo) wrote :

This is a POC patch, which must be updated to check if gateway belongs to
subnet , or if it doesn't:

diff --git a/neutron/agent/linux/interface.py b/neutron/agent/linux/interface.py
index 538527b..5a9f186 100644
--- a/neutron/agent/linux/interface.py
+++ b/neutron/agent/linux/interface.py
@@ -116,15 +116,16 @@ class LinuxInterfaceDriver(object):
                                             namespace=namespace,
                                             ip=ip_cidr)

- if gateway:
- device.route.add_gateway(gateway)
-
         new_onlink_routes = set(s['cidr'] for s in extra_subnets)
+ if gateway:
+ new_onlink_routes.update([gateway])
         existing_onlink_routes = set(device.route.list_onlink_routes())
         for route in new_onlink_routes - existing_onlink_routes:
             device.route.add_onlink_route(route)
         for route in existing_onlink_routes - new_onlink_routes:
             device.route.delete_onlink_route(route)
+ if gateway:
+ device.route.add_gateway(gateway)

     def delete_conntrack_state(self, root_helper, namespace, ip):
         """Delete conntrack state associated with an IP address.

Changed in neutron:
assignee: nobody → Miguel Angel Ajo (mangelajo)
description: updated
tags: added: l3-ipam-dhcp
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

I'm not sure if this needs to be fixed.
The trace indicates of misconfiguration.

Changed in neutron:
status: New → Incomplete
Changed in neutron:
importance: Undecided → Low
status: Incomplete → Confirmed
Revision history for this message
Miguel Angel Ajo (mangelajo) wrote :

This needs to be fixed, it's not missconfiguration.

Just that neutron-l3-agent is not supporting on-link route setup on the l3 agent.
That's necessary when the router IP is outside the subnet range.

goocher (farmerworking)
Changed in neutron:
status: Confirmed → In Progress
Changed in neutron:
status: In Progress → Confirmed
Revision history for this message
Sreekumar S (sreesiv) wrote :

This seems duplicate of the same issues mentioned in 1335023. I've already submitted a patch for this.

Revision history for this message
Sreekumar S (sreesiv) wrote :
Changed in neutron:
assignee: Miguel Angel Ajo (mangelajo) → Sreekumar S (sreesiv)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/277303

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/233287
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b6126bc0f17f348ff6303b9bd6041cb018479ef9
Submitter: Jenkins
Branch: master

commit b6126bc0f17f348ff6303b9bd6041cb018479ef9
Author: Sreekumar S <email address hidden>
Date: Sat Oct 10 03:18:00 2015 +0530

    Fix for adding gateway with IP outside subnet

    Currently 'force_gateway_on_subnet' configuration is set to True
    by default and enforces the subnet on to the gateway. With this
    fix 'force_gateway_on_subnet' can be changed to False, and
    gateway outside the subnet can be added.
    Before adding the default route, a route to the gateway IP is
    added. This applies to both external and internal networks.

    This configuration option is deprecated, and should be removed
    in a future release. It should always allow gateway outside the
    subnet. This is done as a separate patch
    https://review.openstack.org/#/c/277303/

    Change-Id: I3a942cf98d263681802729cf09527f06c80fab2b
    Closes-Bug: #1335023
    Closes-Bug: #1398768

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/283030

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/277303
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3e6d602f542fc97e64cd5503bf1ea4e71d648abf
Submitter: Jenkins
Branch: master

commit 3e6d602f542fc97e64cd5503bf1ea4e71d648abf
Author: Sreekumar S <email address hidden>
Date: Mon Feb 8 12:58:01 2016 +0530

    Deprecate 'force_gateway_on_subnet' configuration option

    Currently 'force_gateway_on_subnet' configuration is set to True
    by default and enforces the subnet on to the gateway. With the
    fix in https://review.openstack.org/#/c/233287/, gateway outside
    the subnet can be added, and the configuration option now has
    lost its significance.

    With this patch, the configuration option is deprecated.
    It should be removed in Newton release, and the system should
    always allow gateway outside the subnet.
    This patch is dependent on the fix for adding gateway outside
    the subnet, mentioned above.

    DocImpact: 'force_gateway_on_subnet' description should be
    updated in the docs and marked as deprecated to be removed in
    the Newton release.

    Change-Id: I28b3d7add303ee479fc071c1de142b0f7811e4e5
    Closes-Bug: #1335023
    Closes-Bug: #1398768

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/283415

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/283030
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9e11c5b8516c42000d5409c0f44f852acb5fed3d
Submitter: Jenkins
Branch: stable/liberty

commit 9e11c5b8516c42000d5409c0f44f852acb5fed3d
Author: Sreekumar S <email address hidden>
Date: Sat Oct 10 03:18:00 2015 +0530

    Fix for adding gateway with IP outside subnet

    Currently 'force_gateway_on_subnet' configuration is set to True
    by default and enforces the subnet on to the gateway. With this
    fix 'force_gateway_on_subnet' can be changed to False, and
    gateway outside the subnet can be added.
    Before adding the default route, a route to the gateway IP is
    added. This applies to both external and internal networks.

    Change-Id: I3a942cf98d263681802729cf09527f06c80fab2b
    Closes-Bug: #1335023
    Closes-Bug: #1398768
    (cherry picked from commit b6126bc0f17f348ff6303b9bd6041cb018479ef9)

tags: added: in-stable-liberty
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/neutron 8.0.0.0b3

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.

Revision history for this message
Miguel Angel Ajo (mangelajo) wrote :

awesome! :)

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/neutron 7.0.4

This issue was fixed in the openstack/neutron 7.0.4 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.