secgroup ofperror flooding in the fullstack ci

[Li Ma]

2016-08-31 07:55:37.351 9430 ERROR dragonflow.controller.ryu_base_app [-] OFPErrorMsg received: type=0x02 code=0x00 message=0x04 0x0e 0x00 0x68 0xc3 0xf7 0x47 0x0f 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x4d 0x00 0x00 0x00 0x00 0x00 0x00 0x06 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0x00 0x00 0x00 0x00 0x00 0x01 0x00 0x12 0x00 0x01 0x4a 0x04 0x00 0x00 0x00 0x04 0x80 0x00 0x0a 0x02
2016-08-31 07:55:37.352 9430 DEBUG RyuDFAdapter [-] EventOFPErrorMsg received.
version=0x4, msg_type=0x1, msg_len=0x4c, xid=0xc3f7470f
 `-- msg_type: OFPT_ERROR(1)
OFPErrorMsg(type=0x2, code=0x0, data=b'\x04\x0e\x00\x68\xc3\xf7\x47\x0f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x4d\x00\x00\x00\x00\x00\x00\x06\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x00\x00\x01\x00\x12\x00\x01\x4a\x04\x00\x00\x00\x04\x80\x00\x0a\x02')
 |-- type: OFPET_BAD_ACTION(2)
 |-- code: OFPBAC_BAD_TYPE(0) error_msg_handler /usr/local/lib/python2.7/dist-packages/ryu/controller/ofp_handler.py:282
2016-08-31 07:55:37.352 9430 DEBUG RyuDFAdapter [-] `-- data: version=0x4, msg_type=0xe, msg_len=0x68, xid=0xc3f7470f
     `-- msg_type: OFPT_FLOW_MOD(14) error_msg_handler /usr/local/lib/python2.7/dist-
packages/ryu/controller/ofp_handler.py:289

Link:
http://logs.openstack.org/41/363441/1/check/gate-dragonflow-dsvm-fullstack-nv/1fd015f/logs/screen-df-controller.txt.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/364869

[Hong Hui Xiao (xiaohhui)]

Actually, this error is also observed in the log, which makes bug 1589761 related

2016-08-31 07:51:46.247 9430 ERROR dragonflow.controller.ryu_base_app [-] OFPErrorMsg received: type=0x04 code=0x08 message=0x04 0x0e 0x00 0x68 0xc3 0xf7 0x46 0x21 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x4d 0x00 0x00 0x00 0x00 0x00 0xff 0xfe 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0x00 0x00 0x00 0x00 0x00 0x01 0x00 0x16 0x00 0x01 0xd3 0x08 0x00 0x00 0x00 0x25 0x00 0x00 0x00 0x35
2016-08-31 07:51:46.248 9430 DEBUG RyuDFAdapter [-] EventOFPErrorMsg received.
version=0x4, msg_type=0x1, msg_len=0x4c, xid=0xc3f74621
 `-- msg_type: OFPT_ERROR(1)
OFPErrorMsg(type=0x4, code=0x8, data=b'\x04\x0e\x00\x68\xc3\xf7\x46\x21\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x4d\x00\x00\x00\x00\x00\xff\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x00\x00\x01\x00\x16\x00\x01\xd3\x08\x00\x00\x00\x25\x00\x00\x00\x35')
 |-- type: OFPET_BAD_MATCH(4)
 |-- code: OFPBMC_BAD_MASK(8) error_msg_handler /usr/local/lib/python2.7/dist-packages/ryu/controller/ofp_handler.py:282

[OpenStack Infra (hudson-openstack)]

Change abandoned by Li Ma (<email address hidden>) on branch: master
Review: https://review.openstack.org/365207
Reason: seems it has nothing to do with ovs and ryu. it needs more investigation.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Hong Hui Xiao (<email address hidden>) on branch: master
Review: https://review.openstack.org/364869
Reason: This might not be the perfect solution as you have already pointed out.

[Hong Hui Xiao (xiaohhui)]

I spent some time investigate this bug today. I have 3 envs: fedora24(kernal 4.5.0), ubuntu16(kernal 4.4.0), ubuntu14(kernal 4.2.0).

This bug only happens in ubuntu14. Because devstack will use apt to install openvswitch at first, an old version openvswitch(2.0.0) is installed. At this time, openvswitch can't load nf_conntrack in the kernal modules. Even if the script install openvswitch from git then, the kernal modules still can't update. However, the output of dpkg -l does update. Removing it from kernal modules by using "modprobe -r" is a workaround, which is used by neutron, which is also not perfect.

openvswitch will not come with operation system. So, we should avoid install it in devstack if we want to install it in dragonflow. This should be the first step to partial resolve the bug.

What if user install ovs and then install dragonflow? The feasible solution might clean the ovs before install it from git, just like the unstack script do.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/367028

[Hong Hui Xiao (xiaohhui)]

I have talked with wei-yuan and with agreement from wei-yuan, I would take over this bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366824
Committed: https://git.openstack.org/cgit/openstack/dragonflow/commit/?id=36407cf6025561aeaab724ef614805dc9e1e2f04
Submitter: Jenkins
Branch: master

commit 36407cf6025561aeaab724ef614805dc9e1e2f04
Author: Hong Hui Xiao <email address hidden>
Date: Wed Sep 7 09:00:51 2016 -0500

    Don't install ovs from devstack

    Since dragonflow will install it, devstack don't need to install it.
    And sometimes, an old version will be installed, which will cause issue.

    Change-Id: I3cd62b18b2573dc79bbf1c4f1bfc1d257f5b5b44
    Partial-Bug: #1619101

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/367028
Committed: https://git.openstack.org/cgit/openstack/dragonflow/commit/?id=aa227f2148dbc20747edd99189e17f2c6ba5ec69
Submitter: Jenkins
Branch: master

commit aa227f2148dbc20747edd99189e17f2c6ba5ec69
Author: Hong Hui Xiao <email address hidden>
Date: Wed Sep 7 19:18:24 2016 -0500

    Install necessary packages before compiling ovs

    Not all packages which are needed to compile ovs will come along with
    OS. For example, in fedora 24, autoconf and automake are not installed
    by default. This will cause issue when using OVS_INSTALL_FROM_GIT=True.

    Change-Id: I9e6677720d0f21ff43b6ed8a267e20c4660c5f6c
    Related-Bug: #1619101

[Hong Hui Xiao (xiaohhui)]

With https://review.openstack.org/#/c/366824/ merged, this issue has already gone in normal env.

I think we could downgrade the importance.

[Li Ma (nick-ma-z)]

Could we close this bug as it is gone in CI?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/367070
Committed: https://git.openstack.org/cgit/openstack/dragonflow/commit/?id=ef06ad338883116c269f75471a2326f69728b29e
Submitter: Jenkins
Branch: master

commit ef06ad338883116c269f75471a2326f69728b29e
Author: Hong Hui Xiao <email address hidden>
Date: Wed Sep 7 22:51:22 2016 -0500

    Clean ovs if it has been installed

    ovs will not come along with OS, but sometimes user may have already install
    ovs and then install dragonflow by devstack. If the ovs is an old version,
    and dragonflow tries to install a new version ovs, this may cause issue.
    For example, the kernal module can't be updated to new version.

    This patch just cleans ovs if it has been installed.

    Change-Id: I357d22b4cc0c7d6f56e22850471ce75ec5debbd6
    Closes-Bug: #1619101