Support external gateways in VLAN mode

[Lorin Hochstein]

Currently, there isn't a good way to configure an external gateway when running in VLAN mode because each network requires a separate gateway.

Note that dnsmasq can support multiple gateways by using tagging.

See http://paste.openstack.org/show/18471/ by Nate Burton for a solution that uses network labels as dnsmasq tags.

[Vish Ishaya (vishvananda)]

If this is just as simple as adding the network label from the paste, we should definitely propose that one line fix in.

[Nathanael Burton (mathrock)]

Obviously the external gateway needs to be configured to be in the correct vlan, etc on the managed switch/router, but it pretty much is as simple as the one line patch, updating the Nova DB to set the label (this gets used as the tag in dnsmasq and can be set during the network create), make sure nova-network doesn't collide with whatever addresses your physical network is using and then the custom configured dnsmasq.conf file.

Maybe one addition would be a parameter to nova-manage network create to specify that the network didn't need a nova gateway (something like '--no-gateway' on the create).

The last issue is I have no idea how any of this meshes with the long term, Quantum plans.

Thanks,

Nate

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/10468

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/10468
Committed: http://github.com/openstack/nova/commit/cefb87511c4eae22c46f25da7335b4beb4631c4a
Submitter: Jenkins
Branch: master

commit cefb87511c4eae22c46f25da7335b4beb4631c4a
Author: mathrock <email address hidden>
Date: Sat Jul 28 14:38:43 2012 -0400

    Support external gateways in VLAN mode.

    Add network label when starting dnsmasq. Fixes bug 1012443.

    To allow for per-network dnsmasq customization we can use the network
    label from the database as the dnsmasq network tag. A user can then
    specify a custom dnsmasq config file using the 'dnsmasq_config_file'
    configuration flag.

    An example dnsmasq config file showing how to set a custom external
    gateway or ntp server for particular networks based on their tag/label
    follows:

    dhcp-option=tag:'private',option:router,10.0.0.254
    dhcp-option=tag:'private',option:ntp-server,10.0.0.253
    dhcp-option=tag:'private',option:dns-server,8.8.8.8

    dhcp-option=tag:'blue',option:router,10.10.0.254
    dhcp-option=tag:'blue',option:ntp-server,10.10.0.253
    dhcp-option=tag:'blue',option:dns-server,8.8.4.4

    Change-Id: I56cb175a968e33b803dfcb965f90f3a5ca4983ca

[Michael S. Moody (michael-sykosoft)]

Has this been backported to Essex release? We currently manually patch the files in question in Essex (and are not ready to move to Folsom release due to some significant show-stopping bugs that affect us).

[Nathanael Burton (mathrock)]

@Michael This was a feature change and as such is not a candidate for back porting to the stable branch. Can you describe or reference the show stopping bugs you refer to that's holding you back from going to Folsom?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/23928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/23928
Committed: http://github.com/openstack/nova/commit/d8188ef32293659c5a12d99f82ac19b452184967
Submitter: Jenkins
Branch: master

commit d8188ef32293659c5a12d99f82ac19b452184967
Author: Davanum Srinivas <email address hidden>
Date: Fri Mar 8 10:26:57 2013 -0500

    Fix Wrong syntax for set:tag in dnsmasq startup option

    Looks like the introduction of quotes in fix for previous
    bug 1012443 was erroneous.

    Fixes LP# 1152504

    Change-Id: Ic84739e38e3b3567f1617302a33623de03196f85

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/25586

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25586
Committed: http://github.com/openstack/nova/commit/50dece6ff2cc94a0ecd0fe9a26c3917e70fe3434
Submitter: Jenkins
Branch: stable/folsom

commit 50dece6ff2cc94a0ecd0fe9a26c3917e70fe3434
Author: Davanum Srinivas <email address hidden>
Date: Fri Mar 8 10:26:57 2013 -0500

    Fix Wrong syntax for set:tag in dnsmasq startup option

    Looks like the introduction of quotes in fix for previous
    bug 1012443 was erroneous.

    Fixes LP# 1152504

    Change-Id: Ic84739e38e3b3567f1617302a33623de03196f85
    (cherry picked from commit d8188ef32293659c5a12d99f82ac19b452184967)

[Michael S. Moody (michael-sykosoft)]

How does this feature work with quantum? We're big users of this patch (in essex), and are looking to move to grizzly in the short term. However, this is critical functionality that we cannot do without. Is this functionality still available in Grizzly, and if so, does it have a comparable feature in quantum?

[Nathanael Burton (mathrock)]

Michael, the feature is still available in Grizzly. I'm not up to date with
Quantum to know if/how to do this with Quantum. I would recommend asking on
the mailing list.

Nate
On Apr 5, 2013 6:55 PM, "Michael S. Moody" <email address hidden>
wrote:

> How does this feature work with quantum? We're big users of this patch
> (in essex), and are looking to move to grizzly in the short term.
> However, this is critical functionality that we cannot do without. Is
> this functionality still available in Grizzly, and if so, does it have a
> comparable feature in quantum?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1012443
>
> Title:
> Support external gateways in VLAN mode
>
> Status in OpenStack Compute (Nova):
> Fix Released
>
> Bug description:
> Currently, there isn't a good way to configure an external gateway
> when running in VLAN mode because each network requires a separate
> gateway.
>
> Note that dnsmasq can support multiple gateways by using tagging.
>
> See http://paste.openstack.org/show/18471/ by Nate Burton for a
> solution that uses network labels as dnsmasq tags.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1012443/+subscriptions
>

[Lorin Hochstein (lorinh)]

Michael:

I would also recommend opening a quantum bug as a feature request if it isn't in quantum yet.

On Fri, Apr 5, 2013 at 7:40 PM, Nathanael Burton <<email address hidden>="mailto:<email address hidden>">> wrote:
Michael, the feature is still available in Grizzly. I'm not up to date with
Quantum to know if/how to do this with Quantum. I would recommend asking on
the mailing list.

Nate
On Apr 5, 2013 6:55 PM, "Michael S. Moody" <email address hidden>
wrote:

> How does this feature work with quantum? We're big users of this patch
> (in essex), and are looking to move to grizzly in the short term.
> However, this is critical functionality that we cannot do without. Is
> this functionality still available in Grizzly, and if so, does it have a
> comparable feature in quantum?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1012443
>
> Title:
> Support external gateways in VLAN mode
>
> Status in OpenStack Compute (Nova):
> Fix Released
>
> Bug description:
> Currently, there isn't a good way to configure an external gateway
> when running in VLAN mode because each network requires a separate
> gateway.
>
> Note that dnsmasq can support multiple gateways by using tagging.
>
> See http://paste.openstack.org/show/18471/ by Nate Burton for a
> solution that uses network labels as dnsmasq tags.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1012443/+subscriptions
>

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1012443

Title:
  Support external gateways in VLAN mode

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  Currently, there isn't a good way to configure an external gateway
  when running in VLAN mode because each network requires a separate
  gateway.

  Note that dnsmasq can support multiple gateways by using tagging.

  See http://paste.openstack.org/show/18471/ by Nate Burton for a
  solution that uses network labels as dnsmasq tags.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1012443/+subscriptions

[Michael S. Moody (michael-sykosoft)]

Thank you, I will ask on the mailing list.

Michael

On Fri, Apr 5, 2013 at 5:30 PM, Nathanael Burton <
<email address hidden>> wrote:

> Michael, the feature is still available in Grizzly. I'm not up to date with
> Quantum to know if/how to do this with Quantum. I would recommend asking on
> the mailing list.
>
> Nate
> On Apr 5, 2013 6:55 PM, "Michael S. Moody" <email address hidden>
> wrote:
>
> > How does this feature work with quantum? We're big users of this patch
> > (in essex), and are looking to move to grizzly in the short term.
> > However, this is critical functionality that we cannot do without. Is
> > this functionality still available in Grizzly, and if so, does it have a
> > comparable feature in quantum?
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1012443
> >
> > Title:
> > Support external gateways in VLAN mode
> >
> > Status in OpenStack Compute (Nova):
> > Fix Released
> >
> > Bug description:
> > Currently, there isn't a good way to configure an external gateway
> > when running in VLAN mode because each network requires a separate
> > gateway.
> >
> > Note that dnsmasq can support multiple gateways by using tagging.
> >
> > See http://paste.openstack.org/show/18471/ by Nate Burton for a
> > solution that uses network labels as dnsmasq tags.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/nova/+bug/1012443/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1012443
>
> Title:
> Support external gateways in VLAN mode
>
> Status in OpenStack Compute (Nova):
> Fix Released
>
> Bug description:
> Currently, there isn't a good way to configure an external gateway
> when running in VLAN mode because each network requires a separate
> gateway.
>
> Note that dnsmasq can support multiple gateways by using tagging.
>
> See http://paste.openstack.org/show/18471/ by Nate Burton for a
> solution that uses network labels as dnsmasq tags.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1012443/+subscriptions
>

[Salvatore Orlando (salvatore-orlando)]

Hi, for Quantum support, are you just referring to the fix on gerrit for this bug?
If yes, this should not affect quantum: https://github.com/openstack/quantum/blob/master/quantum/agent/linux/dhcp.py#L271

If instead your question is about external gateways, Quantum has a concept of 'external network' which can be used to allocate gateways for routers and floating IPs. The externa gateway at the moment implicitly SNATs all the traffic, but this is becoming configurable for havana.
Quantum does not support however multiple external gateways per router. Is that the feature you are looking for?

[Michael S. Moody (michael-sykosoft)]

Allow me to be more specific:

Our layout looks like this: Physical Hardware Firewalls (supporting
802.1q). These firewalls are attached to switches which also support
802.1q, and then of course we use tag interfaces on the compute nodes
(which double as network nodes, for availability), and we pre-configure
tenants with a subnet (say, a /24), a vlan tag (say, vlan 4), and various
firewall policies (and perhaps also gateway-to-gateway IPSec VPN
configurations). Then, cloud instances are configured to use these hardware
firewalls as their default gateway (let's say, 172.27.4.1/24) using the
dnsmasq patch. In some cases, the tenant may also have physical dedicated
systems on the same subnet (vlan 4), and use the openstack instances to
provide dynamic scaling for certain parts of their workload, while using
the dedicated physical hardware for other purposes (highly available
database servers for instance, which for security couldn't be on a shared
architecture, or for various other reasons).

This, when instances are started, they all use 172.27.4.1 as a default
gateway, and the nova-network is configured to tag all traffic.

There are some interesting SDN (software defined networking) features that
we want to use with Quantum, but they are (or at least seem) mutually
exclusive with our current (unchangeable) setup.

Any public IP addressing is done through our hardware firewall stack, and
is one-to-one NAT mapping generally speaking. The firewalls also serve as
IPS/IDS, and VPN gateways, this they are needed.

Does this explanation help?

Michael

On Sun, Apr 7, 2013 at 5:51 AM, Salvatore Orlando <
<email address hidden>> wrote:

> Hi, for Quantum support, are you just referring to the fix on gerrit for
> this bug?
> If yes, this should not affect quantum:
> https://github.com/openstack/quantum/blob/master/quantum/agent/linux/dhcp.py#L271
>
> If instead your question is about external gateways, Quantum has a concept
> of 'external network' which can be used to allocate gateways for routers
> and floating IPs. The externa gateway at the moment implicitly SNATs all
> the traffic, but this is becoming configurable for havana.
> Quantum does not support however multiple external gateways per router. Is
> that the feature you are looking for?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1012443
>
> Title:
> Support external gateways in VLAN mode
>
> Status in OpenStack Compute (Nova):
> Fix Released
>
> Bug description:
> Currently, there isn't a good way to configure an external gateway
> when running in VLAN mode because each network requires a separate
> gateway.
>
> Note that dnsmasq can support multiple gateways by using tagging.
>
> See http://paste.openstack.org/show/18471/ by Nate Burton for a
> solution that uses network labels as dnsmasq tags.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1012443/+subscriptions
>

[Jian Wen (wenjianhn)]

These options are *NOT AVAILABLE* in dnsmasq versions lower than 2.53.

https://www.redhat.com/archives/rhos-list/2012-December/msg00053.html

[Jian Wen (wenjianhn)]

The following is an example shows how to setup a network with an external gateway and
disable the gateway for another network.

[root@c ~]# nova net-list
+--------------------------------------+--------+-----------------+
| ID | Label | CIDR |
+--------------------------------------+--------+-----------------+
| d97ad30e-9375-4ceb-b5b0-d98b9de77390 | vmnet0 | 11.15.24.0/24 |
| fe2c0cb4-2f5c-4fdc-9e67-a67a86dc8df1 | vmnet1 | 10.15.24.0/16 |
+--------------------------------------+--------+-----------------+

[root@c ~]# cat /etc/nova/dnsmasq.conf
dhcp-option=tag:vmnet1,option:router,10.15.0.254
dhcp-option=tag:vmnet0,option:router