openvswitch gre tunnels not working in quantal

[Serge Hallyn]

[Impact]
Quantal currently only has the openvswitch module provided via the kernel; this does not support gre tunnels between switches which is used extensively by OpenStack Quantum. The DKMS module which is currently disabled in quantal provides these and other features. This is a regression compared to precise which ships the dkms module.

[Test Case]
Serges charm is a excellent way to test this - see lp:~serge-hallyn/charms/quantal/ovs-lxc/trunk

juju deploy local:ovs-lxc master
juju deploy local:ovs-lxc slave
juju add-relation master:master slave:slave
juju ssh slave/0
sudo lxc-start -n quantal-amd64-pristine

LXC instances will not get an IP address from the DHCP server running on the master.

sudo ovs-dpctl show -s
Will not show a gre entry for the configured pipe (its non-functional)

[Regression Potential]
The fix pull in 8 cherry picked commits from upstream trunk which are predominately targetted at the datapath component (currently disabled) so are well contained.

These picks fixup support for the 3.5 kernel and rename the the dkms provided modules to remove the _mod suffix (aligning to the kernel module naming).

There are some changes in the ovs scripts which are used during init control of openvswitch.

[Original Bug]
I use the charm at lp:~serge-hallyn/charms/quantal/ovs-lxc to fire up a pair of lxc nodes. The hooks/install file will show how the nodes are set up. Each node has an openvswitch bridge with a gre tunnel with the other node as remote_ip, and a dnsmasq on the master lxc host. It creates containers which have eth1 slaved to the openvswitch bridge.

On precise (using lp:~serge-hallyn/charms/precise/ovs-lxc) this works. Containers on slave nodes get ip addresses from the dnsmasq on the master, and can communicate to each other over the private 192.168.100.x subnet. Precise is using the openvswitch_mod kernel module.

On quantal, which uses the upstream openvswitch kernel module, it does not work. When I create a container on the slave, and run dhclient on its 'eth1' which is slaved to the openvswitch bridge, tcpdump on the slave host shows the queries on ovbr0 (the openvswitch bridge) but not on eth0.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1068365

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

Does this issue go away if you boot into the Precise kernel on a Quantal install?

[Stefan Bader (smb)]

Looks like deeply hidden in here (http://openvswitch.org/pipermail/discuss/2012-June/007413.html) it say the kernel openvswitch module has no implicit gre support.

[Serge Hallyn (serge-hallyn)]

Quoting Stefan Bader (<email address hidden>):
> Looks like deeply hidden in here
> (http://openvswitch.org/pipermail/discuss/2012-June/007413.html) it say
> the kernel openvswitch module has no implicit gre support.

Boy that is a tough thread to follow. At first I thought it was
saying there is a different way to accomplish it, but the redhat
bugzilla explicitly says there is not.

If this is the case, I think this is a huge regression, and we need
the openvswitch-datapath-dkms package back as it is the only way to
achieve the tunnels which worked in precise.

[James Page (james-page)]

The dkms package was disabled as its not compatible with the 3.5 kernel; this decision was taken under the assumption that the upstream kernel module was feature comparable with the dkms provided kernel module.

As this appears not to be the case then we need to focus on:

1) Making the dkms package work again with the 3.5 kernel and re-introducing it under SRU
2) Fixing the kernel openvswitch module to support gre.

This bug will potentially impact openstack installations using quantum on quantal (its OK on precise which reflects Serge's finding).

[vivekraghuwanshi (vivekraghuwanshi)]

It may be possible but we are facing same issue even in vlan and gre

https://answers.launchpad.net/quantum/+question/211395

[James Page (james-page)]

I cut a new package based on trunk of openvswitch (1.8.90) and other than a high latency on the first connection over the bridge, things worked OK (i.e. the dkms package built and installed OK).

[James Page (james-page)]

Branch linked with cherry picked patches from upstream master branch which enable the DKMS package on 3.5 kernel for 1.4.3 release.

[Ben Pfaff (blp-nicira)]

> The dkms package was disabled as its not compatible with the 3.5 kernel; this decision was taken under the assumption that the upstream kernel module was feature comparable with the dkms provided kernel module.

That's right, the Open vSwitch FAQ says:

Q: What features are not available in the Open vSwitch kernel datapath
   that ships as part of the upstream Linux kernel?

A: The kernel module in upstream Linux 3.3 and later does not include
   the following features:

       - Bridge compatibility, that is, support for the ovs-brcompatd
         daemon that (if you enable it) lets "brctl" and other Linux
         bridge tools transparently work with Open vSwitch instead.

         We do not expect bridge compatibility to ever be available in
         upstream Linux. If you need bridge compatibility, use the
         kernel module from the Open vSwitch distribution instead of the
         upstream Linux kernel module.

       - Tunnel virtual ports, that is, interfaces with type "gre",
         "ipsec_gre", "capwap". It is possible to create tunnels in
         Linux and attach them to Open vSwitch as system devices.
         However, they cannot be dynamically created through the OVSDB
         protocol or set the tunnel ids as a flow action.

         Work is in progress in adding these features to the upstream
         Linux version of the Open vSwitch kernel module. For now, if
         you need these features, use the kernel module from the Open
         vSwitch distribution instead of the upstream Linux kernel
         module.

       - Patch virtual ports, that is, interfaces with type "patch".
         You can use Linux "veth" devices as a substitute.

         We don't have any plans to add patch ports upstream.

[James Page (james-page)]

Thanks Ben - found that a bit earlier today...

Any chance you could take a look at the proposed cherry pick patches in the branch to see if they look sane?

They are testing OK and performance looks comparable with that in 12.04 on 1.4.0 of openvswitch.

[Ben Pfaff (blp-nicira)]

> Any chance you could take a look at the proposed cherry pick patches in the branch to see if they look sane?

I'd be glad to do so, but I'm not having any luck finding the proposed cherry pick patches. (I always have a terrible time finding my way around launchpad.)

[Ben Pfaff (blp-nicira)]

I managed to find the patches.

All of those patches seem reasonable to me.

[James Page (james-page)]

Thanks for the feedback Ben; I'll get the SRU moving.

[Serge Hallyn (serge-hallyn)]

(the package in ppa:james-page/junk works for me; i'll verify it in quantal-proposed when available)

[James Page (james-page)]

Uploaded to -proposed for SRU team review.

[Dave Walker (davewalker)]

Hello Serge, or anyone else affected,

Accepted into quantal-proposed. The package will build now and be available in a few hours in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvswitch - 1.4.3-0ubuntu2

---------------
openvswitch (1.4.3-0ubuntu2) quantal-proposed; urgency=low

  * Re-enable the openvswitch-datapath-dkms package to enable support
    for gre tunnels between virtual switches which is not supported
    in the kernel provided openvswitch module (LP: #1068365).
    - d/patches/0001->0008*.patch: Cherry picked patches from upstream
      trunk which enable support for the 3.5 linux kernel and align
      dkms module naming with kernel module naming.
    - d/dkms.conf.in: Drop _mod postfix from dkms module names.
 -- James Page <email address hidden> Tue, 23 Oct 2012 17:24:17 +0200

[Serge Hallyn (serge-hallyn)]

Verified with my juju charm (updating to -proposed at start of hooks/install). Thanks!

[Steve Langasek (vorlon)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvswitch - 1.4.3-0ubuntu2

---------------
openvswitch (1.4.3-0ubuntu2) quantal-proposed; urgency=low

  * Re-enable the openvswitch-datapath-dkms package to enable support
    for gre tunnels between virtual switches which is not supported
    in the kernel provided openvswitch module (LP: #1068365).
    - d/patches/0001->0008*.patch: Cherry picked patches from upstream
      trunk which enable support for the 3.5 linux kernel and align
      dkms module naming with kernel module naming.
    - d/dkms.conf.in: Drop _mod postfix from dkms module names.
 -- James Page <email address hidden> Tue, 23 Oct 2012 17:24:17 +0200

[Thiago Martins (martinx)]

Hi!

 I'm using Ubuntu Quantal 12.10 64 bits, with "openvswitch-switch 1.4.3-0ubuntu2" and...:

tail /var/log/quantum/openvswitch-agent.log

2012-12-31 00:49:10 ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Failed to create OVS patch port. Cannot have tunneling enabled on this agent, since this version of OVS does not support tunnels or patch ports. Agent terminated!

 So, not solved... I think...

Tks,
Thiago

[Serge Hallyn (serge-hallyn)]

Thiago,

I *think* you misunderstood.

There is no claim that this is fixed in the upstream driver.

Rather the openvswitch-dkms package, which has the non-upstream kernel driver, was re-added to the archive. You can install and use it to use this feature until the feature is added to the upstream kernel driver.

[Thiago Martins (martinx)]

Thank you Serge! I'll try that... Sorry for the confusion.