Network interface allocation corrupts instance info cache

[Roman Podoliaka]

Upstream bug: https://bugs.launchpad.net/nova/+bug/1501735

Allocation of network interfaces for an instance can result in corruption of the instance info cache in Nova. The result is that the cache may contain duplicate entries for network interfaces. This can cause failure to boot nodes, as seen with the Libvirt driver.

[Roman Podoliaka (rpodolyaka)]

Should be fixed in https://review.openstack.org/230919

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Mark Goddard <email address hidden>
Review: https://review.fuel-infra.org/16126

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/16126
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: e8c1faa318f9f26d1a50f869381c6cbebbae2d0b
Author: Mark Goddard <email address hidden>
Date: Thu Jan 14 15:26:40 2016

network: Don't repopulate instance info cache from Neutron ports

Allocation of network interfaces for an instance can result in
corruption of the instance info cache. The result is that the cache
may contain duplicate entries for network interfaces. This can cause
instance boot failure. This bug appears to be attributable to the
combined effects of the fixes for bugs #1467581 and #1407664.

This change reverts the fix for bug #1407664, whilst keeping a
modified version of the unit test that was added with it. It also
adds a second unit test.

Closes-Bug: #1505124

Change-Id: I53d5284907d44ae8b5546993f8fd461b385c39e6

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Mark Goddard <email address hidden>
Review: https://review.fuel-infra.org/16691

[Sergii Rizvan (srizvan)]

Invalid for MOS 6.1, because nova in 6.1 doesn't contain changes for upstream bug #1407664, which were reverted to fix this issue (see upstream bug #1501735).

[Timur Nurlygayanov (tnurlygayanov)]

Ok, it looks like all works fine, moved to Fix Released.

[Roman Podoliaka (rpodolyaka)]

This has finally been merged to Mitaka: https://review.openstack.org/#/c/230919/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/12734
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 6e97f7b778dd234e112682a874332c11456b5bf1
Author: Mark Goddard <email address hidden>
Date: Thu Feb 4 14:56:08 2016

Refresh instance info cache within lock

Fix interface attachment bug where multiple concurrent attachment
requests can cause corruption of the nova instance info cache. This
change refreshes the info cache object from the database whilst
holding the refresh-cache lock, ensuring that changes are
synchronised.

Change-Id: I6ea2eda8a61f418b0c32f13a7ed6904352712857
Closes-Bug: #1541838
Related-Bug: #1505124

[Anna Babich (ababich)]

It has been verified on (3 controllers+2 computes, vlan, cinder lvm):
[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 154
cat /etc/fuel_build_number:
 154
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-misc-9.0.0-1.mos8259.noarch
 python-fuelclient-9.0.0-1.mos301.noarch
 fuel-9.0.0-1.mos6325.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8611.noarch
 fuelmenu-9.0.0-1.mos263.noarch
 fuel-ui-9.0.0-1.mos2641.noarch
 fuel-migrate-9.0.0-1.mos8259.noarch
 fuel-ostf-9.0.0-1.mos921.noarch
 shotgun-9.0.0-1.mos87.noarch
 python-packetary-9.0.0-1.mos130.noarch
 fuel-bootstrap-cli-9.0.0-1.mos272.noarch
 fuel-nailgun-9.0.0-1.mos8611.noarch
 fuel-setup-9.0.0-1.mos6325.noarch
 rubygem-astute-9.0.0-1.mos733.noarch
 fuel-mirror-9.0.0-1.mos130.noarch
 fuel-openstack-metadata-9.0.0-1.mos8611.noarch
 fuel-notify-9.0.0-1.mos8259.noarch
 fuel-release-9.0.0-1.mos6325.noarch
 nailgun-mcagents-9.0.0-1.mos733.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-utils-9.0.0-1.mos8259.noarch
 fuel-library9.0-9.0.0-1.mos8259.noarch
 fuel-agent-9.0.0-1.mos272.noarch
[root@nailgun ~]#

The scenario proposed in upstream https://bugs.launchpad.net/nova/+bug/1501735 description has been used for verification (the time.sleep(2) has been added to nova code and the script allowing to create an instance with multiple network interfaces has been run 20 times). All test iterations were OK: instances with multiple network interfaces were created and launched successfully and there were no duplicate interfaces in the instance info cache.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19384
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 91532a49fd4d240f6022b86d9e9ae63fd45731a2
Author: Sergii Rizvan <email address hidden>
Date: Mon Apr 11 13:38:24 2016

Revert "Refresh instance info cache within lock"

This reverts commit 6e97f7b778dd234e112682a874332c11456b5bf1.

Change-Id: I60dd9b0a5b73b5dcc8f20e8ce6374caacaa5aff9
Related-Bug: #1541838
Related-Bug: #1505124

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/16691
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 606daa93060d971e378e6cb55c2f43434bfe4ad2
Author: Mark Goddard <email address hidden>
Date: Mon Apr 25 13:08:27 2016

network: Don't repopulate instance info cache from Neutron ports

Allocation of network interfaces for an instance can result in
corruption of the instance info cache. The result is that the cache
may contain duplicate entries for network interfaces. This can cause
instance boot failure. This bug appears to be attributable to the
combined effects of the fixes for bugs #1467581 and #1407664.

This change reverts the fix for bug #1407664, whilst keeping a
modified version of the unit test that was added with it. It also
adds a second unit test.

Closes-bug: #1505124
Related-bug: #1541838
Change-Id: I53d5284907d44ae8b5546993f8fd461b385c39e6
(cherry picked from upstream commit 8694c1619d774bb8a6c23ed4c0f33df2084849bc)

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/20750

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20750
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: ad4c76814f5940a7abebbdc77cbcd6d1139d8ac6
Author: Sergii Rizvan <email address hidden>
Date: Tue May 17 13:05:59 2016

Revert "Revert "Refresh instance info cache within lock""

This reverts commit 91532a49fd4d240f6022b86d9e9ae63fd45731a2.

Change-Id: I3fab22a9fe140164b159716e2667522450ab71ab
Closes-Bug: #1541838
Related-Bug: #1505124

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 7.0 mu4 updates.
Scenario for verification is described in upstream https://bugs.launchpad.net/nova/+bug/1501735
(the time.sleep(2) has been added to nova code and the script from comment #4 has been run 20 times).

Actual result:
All 20 times the correct instance info cache was observed - every time instance with multiple network interfaces was created and launched successfully and there was no duplicate interfaces in the instance info cache.