OVS agent should fail if can't get DVR mac address

[Oleg Bondarev]

If ovs agent is configured to run in dvr mode it has to get it's unique mac address from server on startup . In case it cannot get it after several attempts (commit 51303b5fe4785d0cda76f095c95eb4d746d7d783) due to some error, it falls back to non-dvr mode.

The question is what is the purpoise of ovs agent to be running in non-dvr mode while it was configured for dvr? Server code does not handle ovs agent 'in_distributed_mode' flag in any way and will continue scheduling dvr routers to such nodes. This may lead to connectivity issues which are hard to debug.

Example:

2016-01-12 11:29:15.186 16238 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-e3b3643d-6976-4656-b247-ab291e6a4b27 - - - - -] L2 agent could not get DVR MAC address at startup due to RPC error. It happens when the server does not support this RPC API. Detailed message: Remote error: DBConnectionError (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0")

There were some issues with mysql on startup which led to half of ovs agents running in non-dvr mode silently.

The proposal is to fail in case agent cannot operate in the mode it was configured to.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/270130

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270130
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e2a37be73da7bbc0ab562eabef69bfe69dc7cfa1
Submitter: Jenkins
Branch: master

commit e2a37be73da7bbc0ab562eabef69bfe69dc7cfa1
Author: Oleg Bondarev <email address hidden>
Date: Wed Jan 20 13:53:20 2016 +0300

    OVS agent should fail if it can't get DVR mac address

    Currently agent will fall back to non-dvr mode in case it can't.
    However neutron server does not check dvr mode of ovs agents when
    scheduling routers. So in a DVR enabled cluster all ovs agents
    should run in DVR mode. Otherwise it will lead to undefined
    behavior which is hard to debug.

    Closes-Bug: #1536110
    Change-Id: I6c31aabf1852c688e9c27fc1859d3fdd830caa68

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/271755

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/276297

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276297
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8211113eb6387e6deb6fdc7a38821eada20c414c
Submitter: Jenkins
Branch: master

commit 8211113eb6387e6deb6fdc7a38821eada20c414c
Author: Oleg Bondarev <email address hidden>
Date: Thu Feb 4 18:08:36 2016 +0300

    Reno note regarding OVS DVR agent failure on startup

    Commit e2a37be73da7bbc0ab562eabef69bfe69dc7cfa1 changed the behavior
    of the OVS DVR agent so it will fail if can't get DVR config on start-up.
    This patch adds a reno note describing this new behavior.

    Related-Bug: #1536110
    Change-Id: Idcdcdad9eba78ef5fbdae420e114a7ed793f3e43

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/271755
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e6a58a43993643abe47918590b84672593ef5a47
Submitter: Jenkins
Branch: stable/liberty

commit e6a58a43993643abe47918590b84672593ef5a47
Author: Oleg Bondarev <email address hidden>
Date: Wed Jan 20 13:53:20 2016 +0300

    OVS agent should fail if it can't get DVR mac address

    Currently agent will fall back to non-dvr mode in case it can't.
    However neutron server does not check dvr mode of ovs agents when
    scheduling routers. So in a DVR enabled cluster all ovs agents
    should run in DVR mode. Otherwise it will lead to undefined
    behavior which is hard to debug.

    Closes-Bug: #1536110
    Change-Id: I6c31aabf1852c688e9c27fc1859d3fdd830caa68
    (cherry picked from commit e2a37be73da7bbc0ab562eabef69bfe69dc7cfa1)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.0.3 release.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.