DHCP not leasing IP after instance reboot with Open vSwitch Firewall Driver configured.

[Javier Diaz Jr]

Detailed bug description:

DHCP works fine during initial boot. Instance is able to acquire DHCP lease and works as expected. However, if instance is either soft/hard rebooted via nova the instance is then unable to obtain DHCP lease and fails at discover.

Steps to reproduce:

1. Create MOS9.2 environment with the Open vSwitch Firewall Driver configured.
2. Launch instance (i.e. testvm) using internal network.
3. Reboot the instance from within the instance (i.e. sudo reboot) and DHCP works fine.
4. Reboot the instance using soft/hard reboot from nova and DHCP fails.

Expected results:

DHCP should be leasing IP to instance.

Actual result:

DHCP fails to lease IP if nova reboot commands issued.

Reproducibility:

Tried it on several instances and reproducibility is always.

Impact:

No DHCP can affect production level instances. High impact.

Description of the environment:
 Operation system: Ubuntu 14.04 Kernel 4.4
 Versions of components: MOS 9.2
 Network model: VLAN
Additional information:

If you require access to lab where issue is present please let me know.

[Inessa Vasilevskaya (ivasilevskaya)]

I noticed that the port number in ovs flows that correspond ro the vm rebooted via 'nova reboot' doesn't match the port number assigned to vm's tap device in 'ovs-ofctl show'.
Another observation - things get back to normal once you shelve/unshelve the vm.

Broken state after reboot (tapdb04e855-b6, port number is 29 and 30) - http://paste.openstack.org/show/602726/
Fixed state after shelve/unshelve, port numbers match - http://paste.openstack.org/show/602729/

[Inessa Vasilevskaya (ivasilevskaya)]

I managed to reproduce the problem on multinode devstack - https://bugs.launchpad.net/neutron/+bug/1673027 (upstream bug reported).

Mitaka and newton releases seem to be affected, didn't test with Ocata.

[Javier Diaz Jr (javierdiazcharles)]

Upstream bug fix released:

https://bugs.launchpad.net/neutron/+bug/1645655

ETA on MOS Fix?

[Inessa Vasilevskaya (ivasilevskaya)]

In order to fix that in MOS 2 patches need to be backported - https://review.openstack.org/#/c/448531/ and https://review.openstack.org/#/c/448052/

If everything goes as planned the fix should be ready by the end of the week, in case of any extraordinaries - at the beginning of next week.

[Javier Diaz Jr (javierdiazcharles)]

Any updates? :)

[Inessa Vasilevskaya (ivasilevskaya)]

Javier, unfortunately the fix didn't make it into the maintenance update due to high code freeze, so it's gonna be released only with the next update.

[Inessa Vasilevskaya (ivasilevskaya)]

Patches:

https://review.fuel-infra.org/#/c/32580/

https://review.fuel-infra.org/#/c/32579/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/32579
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 2fae3da8eb45a8fed6a8771ac2c342f5fbfb9f97
Author: Inessa Vasilevskaya <email address hidden>
Date: Tue Mar 28 13:12:43 2017

ovsfw: Raise exception if tag cannot be found in other_config

Previously, if tag was not present in other_config obtained from ovsdb
for any reason, DEAD VLAN tag was used. This is not smart at all as it
puts all conntrack entries to one point. Also tag is mandatory and if
other_config doesn't contain it, it's a huge mistake that should never
happen.

Related-bug: 1564947
Related-bug: 1670185

(cherry-picked from commit a66c27193573ce015c6c1234b0f2a1d86fb85a22)

Change-Id: I91ab75b52b70dbba4c7823550bfdfe0ab9396336

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/32580
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 1fd86e66237401ab3ab5f73ab6f7416125ce0b02
Author: Inessa Vasilevskaya <email address hidden>
Date: Tue Mar 28 13:12:43 2017

ovsfw: Refresh OFPort when necessary

Events like server reboots change ofport numbers.
In such cases, cached ofports need to be refreshed.

Closes-Bug: #1645655
Closes-Bug: 1670185

(cherry-picked from commit 829b39b1ebdb96d87323852e2e099a955e402b63)

Change-Id: If4acf61736b8f1e9707efc409509e1f557d5f886

[Alexey Stupnikov (astupnikov)]

A fix for upstream bug #1645655 was backported to upstream stable/newton branch. We will get this fix after next sync. https://review.openstack.org/#/c/447772/

Moving to Confirmed for MOS10, will close the bug after sync.

[Alexey Stupnikov (astupnikov)]

Fixes were merged yesterday: https://review.fuel-infra.org/#/q/topic:bug/1645655 . Moving to Fix Committed and assigning back to Inessa.

[Ilya Bumarskov (ibumarskov)]

Verified on Fuel 9.2 MU2 (MOS_UBUNTU_MIRROR_ID=9.0-2017-06-27-104421, MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2017-06-27-104421)

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/1.0/mitaka
Change author: Inessa Vasilevskaya <email address hidden>
Review: https://review.fuel-infra.org/38214