OVS with GRE doesn't work in 3.10

[Artem Panchenko]

Fuel version info (6.1 iso # 376): http://paste.openstack.org/show/214942/

Health checks fail on environment with Neutron+GRE (CentOS), because instances can't get IP address after boot:

http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.thread_7/116/testReport/junit/(root)/deploy_ceph_ha_nodegroups/deploy_ceph_ha_nodegroups/
http://paste.openstack.org/show/214945/

Steps to reproduce:

1. Create cluster with Neutron GRE and Ceph on CentOS
2. Add 3 controller + ceph nodes
3. Add 2 compute + ceph nodes
4. Select Fedora 3.10 kernel on 'Settings' tab
5. Deploy cluster
6. Run health checks (OSTF)

Expected result:

 - environment passes health checks

Actual result:

 - 2 checks fail ('Check network connectivity from instance via floating IP' and 'Launch instance with file injection') because after creating instance it's not reachable by internal/floating IP address

Here are errors I've found in neutron logs on controller (node-1):

http://paste.openstack.org/show/214940/

The same errors ('Failed to set-up gre tunnel port') present in neutron logs on all slave nodes. I tried to sniff traffic of management network for GRE packets, but tcpdump showed nothing. Also I was able to setup GRE tunnel between controller and compute manually and it worked fine, so I believe GRE isn't blocked by something. Ubuntu deployments aren't affected. Diagnostic snapshot is attached.

[Artem Panchenko (apanchenko-8)]

This issue affects all deployments with NeutronGre on CentOS:

http://jenkins-product.srt.mirantis.net:8080/job/6.1.system_test.centos.thread_1/122/testReport/(root)/deploy_neutron_gre/deploy_neutron_gre/?

[Oleksiy Molchanov (omolchanov)]

Seems that needs to build a module for the 3.10 kernel support.

[Aleksander Mogylchenko (amogylchenko)]

We have prepared kmod for openvwitch:
https://review.fuel-infra.org/#/admin/projects/packages/centos6/openvswitch-kmod

It should be compiled upon installation for specific kernel (requires kernel-headers).

P.S. There is a separate issue whey openvswitch module fails to load if kernel was built with CONFIG_NET_IPGRE_DEMUX=y, but in our case that is not a problem:
https://review.fuel-infra.org/gitweb?p=packages/centos6/kernel-lt.git;a=blob;f=config-3.10.55-x86_64;h=4c87411115eb7543582e8bacc0be74c0f28953cd;hb=refs/heads/6.1#l715

tl;dr: Fuel should make sure that openvswitch-kmod is compiled upon installation for specific kernel version. Another alternative is to build module for all kernel versions in advance.

[Timur Nurlygayanov (tnurlygayanov)]

25 Tempest tests failed because of the issue on configurations with CentOS+NeutronGRE.

[Roman Vyalov (r0mikiam)]

I understand that this bug requires to build new package with ovs for 3.10 kernel. I reassign this bug to the MOS-linux team

[Aleksander Mogylchenko (amogylchenko)]

@Roman, why? We are not dealing with building packages, and existing kmod package is all that you need to build the module for any required kernel version:
rpmbuild -bb --define 'kversion 3.10.0-123.el7.x86_64' openvswitch-kmod.spec

There is another problem: 3.10 as provided by CentOS uses -fstack-protector-strong, which is not supported by gcc 4.4. So you will need to update build infrastructure to build modules for new kernel.

[Albert Syriy (asyriy)]

The package has been built for 3.10.55-1.mira4.x86_64 kernel with the spec file
https://review.fuel-infra.org/#/c/6474/5/openvswitch-kmod.spec

The disadvantage of the spec file is the kernel version was hardcoded into the spec.

There is a possibility to pass kernel version as a parameter kversion (see the previous post #7) and remove the hardcoded version, but I am not very familiar with OBS.

The second (potential) issue is the outgoing package name built for kernel 2.6.32-504.1.3.el6.x86_64 and for 3.10.55-1.mira4.x86_64 is the same, so IMHO package name should be formed with kernel version (or kernel/kernel-lt) sufix on the fly.

[Aleksander Mogylchenko (amogylchenko)]

Naming can be solved with custom RPM filename:
$ grep rpmfilename openvswitch-kmod.spec
%define _rpmfilename %%{NAME}-%{kversion}.%%{ARCH}.rpm

which will produce 'kmod-openvswitch-3.10.55-1.mira4.x86_64.x86_64.rpm'. Such approach also requires changing fuel-library manifests.

[Aleksander Mogylchenko (amogylchenko)]

I'm building a spec that will be capable of producing RPMs for both kernels, but this is an ugly hack.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/6543

[Aleksander Mogylchenko (amogylchenko)]

@library, packages are ready. Since nothing explicitly depends on kmod-openvswitch-3.10.55-1, it should be installed manually from fuel-library. I'm not merging them yet, because they should be reviewed by other teams.

P.S. I have no idea why OBS made such a strange name for the package "kmod-openvswitch-3.10.55-1.mira4-2.3.1-mira1.mira2.x86_64.rpm" :)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/182261

[Aleksander Mogylchenko (amogylchenko)]

I've updated module names as requested:
ls rpmbuild/RPMS/x86_64/ -1
kmod-openvswitch-2.6.32-504.16.2.el6.x86_64-2.3.1-mira1.x86_64.rpm
kmod-openvswitch-3.10.55-1.mira4-2.3.1-mira1.x86_64.rpm

Please review the CR and let me know (by setting +1) that it works for you.

[Aleksander Mogylchenko (amogylchenko)]

Please also note that something should depend on it, or it should be added to requirements.txt file. In other case it won't be included in the image.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/182261
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=848528d5f802f50679f72435b80e9d28bd8f684c
Submitter: Jenkins
Branch: master

commit 848528d5f802f50679f72435b80e9d28bd8f684c
Author: Stanislaw Bogatkin <email address hidden>
Date: Tue May 12 15:25:17 2015 +0300

    Explicitly declared kmod-openvswitch for 3.10 kernel

    If using kernel 3.10, we should install newer kmod package.

    Change-Id: I0dff6809c71f5da1e85bb0ea0515fa2d664d3516
    Closes-Bug: #1451756

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6543
Submitter: Artem Silenkov <email address hidden>
Branch: 6.1

Commit: 774b29bd87b02983b4a6196fba5575ee0ede7ce5
Author: Aleksandr Mogylchenko <email address hidden>
Date: Wed May 13 22:27:00 2015

Build kernel module for two CentOS kernels

This commit modifies the spec so that 2 kernel modules are produces:
- OVS module for default kernel;
- OVS module for kernel-devel set in the spec;

By default, RedHat and Novell rely on %kernel_module_packag macro to build
modules for different kernel flavors:
http://www.novell.com/developer/Kmpm-code11.pdf

But they are only suitable for building modules for *one* kernel version.

Producing several modules required a hack (hence this commit), but in future it
should be migrated to dkms (added already to 6.5 in different commit).

3rd, 4th and so on module can be added.

Change-Id: I6b1da70a069e997ba7e0b603c2139684e31f55ab
Closes-Bug: #1451756

[Artem Panchenko (apanchenko-8)]

Verified on iso #432:

api: '1.0'
astute_sha: cb655a9a9ad26848bcd9d9ace91857b6f4a0ec15
auth_required: true
build_id: 2015-05-18_03-43-53
build_number: '432'
feature_groups:
- mirantis
fuel-library_sha: 1621cb350af744f497c35f2b3bb889c2041465d8
fuel-ostf_sha: 9ce1800749081780b8b2a4a7eab6586583ffaf33
fuelmain_sha: 0e970647a83d9a7d336c4cc253606d4dd0d59a60
nailgun_sha: 076566b5df37f681c3fd5b139c966d680d81e0a5
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 38765563e1a7f14f45201fd47cf507393ff5d673
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: cb655a9a9ad26848bcd9d9ace91857b6f4a0ec15
      build_id: 2015-05-18_03-43-53
      build_number: '432'
      feature_groups:
      - mirantis
      fuel-library_sha: 1621cb350af744f497c35f2b3bb889c2041465d8
      fuel-ostf_sha: 9ce1800749081780b8b2a4a7eab6586583ffaf33
      fuelmain_sha: 0e970647a83d9a7d336c4cc253606d4dd0d59a60
      nailgun_sha: 076566b5df37f681c3fd5b139c966d680d81e0a5
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 38765563e1a7f14f45201fd47cf507393ff5d673
      release: '6.1'

[Artem Panchenko (apanchenko-8)]

Seems I missed something when verified the fix on iso #432, because on iso #446 the issue still exists:

2015-05-21 08:44:08 ERR (/Stage[main]/L23network::L2/Package[openvswitch-datapath]/ensure) change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install kmod-openvswitch-3.10.55-1' returned 1: Error: Nothing to do

[root@nailgun ~]# find /var/www/nailgun/ -name kmod-openvswitch*
/var/www/nailgun/2014.2.2-6.1/centos/x86_64/Packages/kmod-openvswitch-2.3.1-1.mira2.x86_64.rpm
/var/www/nailgun/2014.2.2-6.1/centos/x86_64/Packages/kmod-openvswitch-lt-2.3.1-1.mira2.x86_64.rpm

[Igor Zinovik (izinovik)]

Artem, seems that your ISO does not contain change for fuel-library package:
https://review.openstack.org/#/c/184220/3/deployment/puppet/l23network/manifests/params.pp,cm

[Igor Zinovik (izinovik)]

I checked revision of fuel-library/ repository that was in your ISO and it does not contain
aforementioned change.

[Artem Panchenko (apanchenko-8)]

@Igor,

thanks! I re-check the fix when the next iso is assembled.

[Artem Panchenko (apanchenko-8)]

Guys,

I checked iso #447 (which includes https://review.openstack.org/#/c/184220/) and deployment still fails due to dependency hell:

http://paste.openstack.org/show/230802/

[Artem Silenkov (asilenkov)]

Correct openvswitch-kmod-lt is not ready yet. It will be ready tomorrow so
stay tuned.
21 мая 2015 г. 8:05 PM пользователь "Artem Panchenko" <
<email address hidden>> написал:

> Guys,
>
> I checked iso #447 (which includes
> https://review.openstack.org/#/c/184220/) and deployment still fails due
> to dependency hell:
>
> http://paste.openstack.org/show/230802/
>
>
> ** Attachment added: "fuel-snapshot-2015-05-21_16-43-55.tar.xz"
>
> https://bugs.launchpad.net/fuel/+bug/1451756/+attachment/4401866/+files/fuel-snapshot-2015-05-21_16-43-55.tar.xz
>
> ** Changed in: fuel
> Status: Fix Committed => Confirmed
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1451756
>
> Title:
> OVS with GRE doesn't work in 3.10
>
> Status in Fuel: OpenStack installer that works:
> Confirmed
>
> Bug description:
> Fuel version info (6.1 iso # 376):
> http://paste.openstack.org/show/214942/
>
> Health checks fail on environment with Neutron+GRE (CentOS), because
> instances can't get IP address after boot:
>
>
> http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.thread_7/116/testReport/junit/(root)/deploy_ceph_ha_nodegroups/deploy_ceph_ha_nodegroups/
> http://paste.openstack.org/show/214945/
>
> Steps to reproduce:
>
> 1. Create cluster with Neutron GRE and Ceph on CentOS
> 2. Add 3 controller + ceph nodes
> 3. Add 2 compute + ceph nodes
> 4. Select Fedora 3.10 kernel on 'Settings' tab
> 5. Deploy cluster
> 6. Run health checks (OSTF)
>
> Expected result:
>
> - environment passes health checks
>
> Actual result:
>
> - 2 checks fail ('Check network connectivity from instance via
> floating IP' and 'Launch instance with file injection') because after
> creating instance it's not reachable by internal/floating IP address
>
> Here are errors I've found in neutron logs on controller (node-1):
>
> http://paste.openstack.org/show/214940/
>
> The same errors ('Failed to set-up gre tunnel port') present in
> neutron logs on all slave nodes. I tried to sniff traffic of
> management network for GRE packets, but tcpdump showed nothing. Also I
> was able to setup GRE tunnel between controller and compute manually
> and it worked fine, so I believe GRE isn't blocked by something.
> Ubuntu deployments aren't affected. Diagnostic snapshot is attached.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1451756/+subscriptions
>

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/6897

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6897
Submitter: Dmitry Burmistrov <email address hidden>
Branch: master

Commit: eec949f26bb3799c06a9dc426c788d6da9d4d353
Author: Artem Silenkov <email address hidden>
Date: Fri May 22 14:19:45 2015

openvswitch refactoring.

We need to build separate modules for each kernel.

Change-Id: I11826ac8365c746367ba298e4cf490d28fd12deb
Related-Bug: #1451756

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/6899

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/6900

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/6901

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/6902

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6901
Submitter: Artem Silenkov <email address hidden>
Branch: 6.1

Commit: e209c4d3c0b1776d5dbfd50a2ddca3b223e9bc26
Author: Artem Silenkov <email address hidden>
Date: Fri May 22 14:43:56 2015

Openvswitch kmod powered module for 2.6.32 kernel.

Change-Id: Ia920bb6e1bc1ca4c341d54f33beccf027ec0f8b7
Related-Bug: #1451756

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6902
Submitter: Artem Silenkov <email address hidden>
Branch: 6.1

Commit: f388a4d3dc5b4712b9bd247a9c2623a76c6ace29
Author: Artem Silenkov <email address hidden>
Date: Fri May 22 14:44:04 2015

Openvswitch kmod powered module for 3.10 kernel

Change-Id: Id1e28df2ee61915ec668264fcf8ec02520ca3c40
Related-Bug: #1451756

[Artem Silenkov (asilenkov)]

Could you please test that new packages located in mirrors?

[Artem Panchenko (apanchenko-8)]

Tested using iso #464 and the issue with GRE tunnels still exists. Here is info about kernel and openvswitch packages installed on slaves:

http://paste.openstack.org/show/235911/

Nova instance still doesn't get IP address via DHCP, there are the same logs (see bug description) in openvswitch-agent.log. Attached diagnostic snapshot.

[Artem Silenkov (asilenkov)]

Module is ok on slave

[root@node-1 ~]# modinfo openvswitch
filename: /lib/modules/3.10.55-1.mira6.x86_64/kernel/net/openvswitch/openvswitch.ko
license: GPL
description: Open vSwitch switching datapath
srcversion: 6C96BB877D9210EC3EF3824
depends:
intree: Y
vermagic: 3.10.55-1.mira6.x86_64 SMP mod_unload modversions

[Artem Silenkov (asilenkov)]

Some logging and useful info

http://paste.openstack.org/show/235952/

[Artem Silenkov (asilenkov)]

For some reason "depends:" field for this new module is empty.
It must be depends: libcrc32c,vxlan,gre

Neither vxlan nor gre modules are loaded inside VM. So it could be a root but still not sure.

[Artem Silenkov (asilenkov)]

Yes, depmod postinst macros is doing wrong.

depmod 3.10.55-1.mira6
must be
depmod 3.10.55-1.mira6.x86_64

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/6945

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/6946

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6945
Submitter: Artem Silenkov <email address hidden>
Branch: 6.1

Commit: cb49971cb5129c716fe0051c170dbe0d2a725c5c
Author: Artem Silenkov <email address hidden>
Date: Mon May 25 15:45:06 2015

Fixed incorrect depmod params

Change-Id: Ie1060a54edd23e510a46870bf8b51dbb42a0dced
Related-Bug: #1451756

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6946
Submitter: Artem Silenkov <email address hidden>
Branch: 6.1

Commit: 26cb33dfbb0a3057dd6980fb025461acc890f40d
Author: Artem Silenkov <email address hidden>
Date: Mon May 25 15:46:16 2015

Fixed incorrect depmod params

Change-Id: I96d8ae1bed11b20e1e30f1204db3d44ad2760379
Related-Bug: #1451756

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Artem Silenkov <email address hidden>
Review: https://review.fuel-infra.org/6992

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6992
Submitter: Artem Silenkov <email address hidden>
Branch: 6.1

Commit: 2fe689c2cbe97c0f58f7447eb3b28d7d0adf0b03
Author: Artem Silenkov <email address hidden>
Date: Tue May 26 13:23:04 2015

Revert to original kmod-openvswitch

Change-Id: I7ded0de218ef5426956c526f0e66540421b2288f
Related-Bug: #1458764
Related-Bug: #1451756

[Artem Silenkov (asilenkov)]

What's the latest status? Could you please verify with latest ISO and give me working env if not working?

[Artem Panchenko (apanchenko-8)]

Verified on iso #477, environment on CentOS with 3.10 kernel passes all OSTF tests, connectivity through GRE tunnels works fine:

api: '1.0'
astute_sha: 5d570ae5e03909182db8e284fbe6e4468c0a4e3e
auth_required: true
build_id: 2015-05-28_06-58-38
build_number: '477'
feature_groups:
- mirantis
fuel-library_sha: 7eb54c7b30c3668e3d282e9302a04265112e2582
fuel-ostf_sha: 4cd2fef040ae9e7645a6b17a7cb44d3cd8fbe0be
fuelmain_sha: 6b5712a7197672d588801a1816f56f321cbceebd
nailgun_sha: 2f5ffd9f238cd2b0428937bc31580182a5e1d2c1
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 5d570ae5e03909182db8e284fbe6e4468c0a4e3e
      build_id: 2015-05-28_06-58-38
      build_number: '477'
      feature_groups:
      - mirantis
      fuel-library_sha: 7eb54c7b30c3668e3d282e9302a04265112e2582
      fuel-ostf_sha: 4cd2fef040ae9e7645a6b17a7cb44d3cd8fbe0be
      fuelmain_sha: 6b5712a7197672d588801a1816f56f321cbceebd
      nailgun_sha: 2f5ffd9f238cd2b0428937bc31580182a5e1d2c1
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: e19f1b65792f84c4a18b5a9473f85ef3ba172fce
      release: '6.1'

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Artem Silenkov <email address hidden> on branch: master
Review: https://review.fuel-infra.org/6899

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Artem Silenkov <email address hidden> on branch: master
Review: https://review.fuel-infra.org/6900