[Backport 1418786] more than one port got created for VM

[Alexander Ignatov]

This is a backport of https://bugs.launchpad.net/neutron/+bug/1418786

Original description
================

If server with neutron-server service have not enough resources for fast processing of requests then there is a high risk of multiple port created for VM during scheduling/networking process.

How to reproduce:

Just get some environment with not so fast neutron-server service node and/or mysql server. Try to spawn bunch of VMs.
Some of them will got two ports created (in neutron DB they will have same device_id). If the system is very slow they could get three of them. If VMs would be spawned it will have only last one which nova got from neutron and this port will be the only active one.

[Oleg Bondarev (obondarev)]

It is recommended to increase neutron.url_timeout on envs with "slow" neutron server

[Aleksandr Shaposhnikov (alashai8)]

If we're talking about 25 nodes environment the problem will be actively seen if you're already have around 1000-1200 vm's created and trying to spawn a bunch of 10-20 vm's.

[Dmitry Borodaenko (angdraug)]

I'm assuming that this bug exists across our Juno releases, so marked Confirmed for 6.0.x; please set to Confirmed for 4.1.x-5.1.x if it also applies to pre-Juno releases.

[Oleg Bondarev (obondarev)]

Investigations on the lab (6.0 release, with oslo_messaging heartbeats disabled for neutron) showed that the root cause is keystone timeouting authorization requests for neutron when nova tries to fetch port that was just created - so nova thinks network creation failed on a compute and reschedules instance to another compute which creates new port in neutron for the same instance.
So we think that this might a manifestation of https://bugs.launchpad.net/mos/+bug/1413341
However this should be proven on a lab with repro.

Meanwhile the bug was filed for nova: https://bugs.launchpad.net/nova/+bug/1423845
I'm going to backport fix once it merged in upstream.
This one should also be verified on the lab with repro.

[OSCI Robot (oscirobot)]

package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.1.mira8.git.c32419c.7c8043a

Changeset: https://review.fuel-infra.org/4840
project: openstack/nova
branch: openstack-ci/fuel-6.1/2014.2
author: Oleg Bondarev
committer: Oleg Bondarev
subject: Fix orphaned ports on build failure
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-4840/

[OSCI Robot (oscirobot)]

RPM package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.0.1.mira6.git.f5eb235.1009d65

Changeset: https://review.fuel-infra.org/4841
project: openstack/nova
branch: openstack-ci/fuel-6.0.1/2014.2
author: Oleg Bondarev
committer: Oleg Bondarev
subject: Fix orphaned ports on build failure
status: patchset-created

Files placed on repository:
openstack-nova-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-api-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-cells-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-cert-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-common-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-compute-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-conductor-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-console-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-doc-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-network-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-novncproxy-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-objectstore-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
openstack-nova-scheduler-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm
python-nova-2014.2.2-fuel6.0.1.mira6.git.f5eb235.1009d65.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4841/centos

[OSCI Robot (oscirobot)]

package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.1~mira6+git.c32419c.7c8043a

Changeset: https://review.fuel-infra.org/4840
project: openstack/nova
branch: openstack-ci/fuel-6.1/2014.2
author: Oleg Bondarev
committer: Oleg Bondarev
subject: Fix orphaned ports on build failure
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /trusty-fuel-6.1-stable-4840/

[OSCI Robot (oscirobot)]

DEB package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.0.1~mira6+git.f5eb235.1009d65

Changeset: https://review.fuel-infra.org/4841
project: openstack/nova
branch: openstack-ci/fuel-6.0.1/2014.2
author: Oleg Bondarev
committer: Oleg Bondarev
subject: Fix orphaned ports on build failure
status: patchset-created

Files placed on repository:
nova-ajax-console-proxy_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-api-ec2_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-api-metadata_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-api-os-compute_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-api-os-volume_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-api_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-baremetal_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-cells_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-cert_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-common_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-compute-kvm_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-compute-libvirt_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-compute-lxc_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-compute-qemu_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-compute-vmware_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-compute-xen_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-compute_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-conductor_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-console_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-consoleauth_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-doc_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-network_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-novncproxy_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-objectstore_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-scheduler_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-spiceproxy_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-volume_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
nova-xvpvncproxy_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb
python-nova_2014.2.2-fuel6.0.1~mira6+git.f5eb235.1009d65_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4841/ubuntu

[OSCI Robot (oscirobot)]

package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.1.mira8

Changeset: https://review.fuel-infra.org/4840
project: openstack/nova
branch: openstack-ci/fuel-6.1/2014.2
author: Oleg Bondarev
committer: Oleg Bondarev
subject: Fix orphaned ports on build failure
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /centos-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

package nova has been built for project openstack/nova
Package version == 2014.2.2, package release == fuel6.1~mira6

Changeset: https://review.fuel-infra.org/4840
project: openstack/nova
branch: openstack-ci/fuel-6.1/2014.2
author: Oleg Bondarev
committer: Oleg Bondarev
subject: Fix orphaned ports on build failure
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /trusty-fuel-6.1-stable/

[Alexander Ignatov (aignatov)]

Made it Won't Fix for previous versions of MOS like 5.x since it likly happen in big scale (>2K VMs) which is not certified for those releases

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/5618

[OSCI Robot (oscirobot)]

RPM package nova has been built for project openstack/nova
Package version == 2014.2, package release == fuel6.0.mira22.git.a42ecf0.bea7645

Changeset: https://review.fuel-infra.org/5618
project: openstack/nova
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Vitaly Sedelnik
committer: Vitaly Sedelnik
subject: Fix orphaned ports on build failure
status: patchset-created

Files placed on repository:
openstack-nova-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-api-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-cells-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-cert-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-common-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-compute-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-conductor-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-console-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-doc-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-network-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-novncproxy-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-objectstore-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
openstack-nova-scheduler-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm
python-nova-2014.2-fuel6.0.mira22.git.a42ecf0.bea7645.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable-5618/centos

[OSCI Robot (oscirobot)]

DEB package nova has been built for project openstack/nova
Package version == 2014.2, package release == fuel6.0~mira22+git.a42ecf0.bea7645

Changeset: https://review.fuel-infra.org/5618
project: openstack/nova
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Vitaly Sedelnik
committer: Vitaly Sedelnik
subject: Fix orphaned ports on build failure
status: patchset-created

Files placed on repository:
nova-ajax-console-proxy_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-api-ec2_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-api-metadata_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-api-os-compute_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-api-os-volume_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-api_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-baremetal_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-cells_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-cert_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-common_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-compute-kvm_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-compute-libvirt_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-compute-lxc_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-compute-qemu_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-compute-vmware_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-compute-xen_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-compute_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-conductor_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-console_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-consoleauth_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-doc_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-network_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-novncproxy_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-objectstore_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-scheduler_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-spiceproxy_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-volume_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
nova-xvpvncproxy_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb
python-nova_2014.2-fuel6.0~mira22+git.a42ecf0.bea7645_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-5618/ubuntu

[Leontii Istomin (listomin)]

Will try to reproduce using rally test, when rally doesn't delete items.
For checking will use the follwoing sql request:
select * from ports where device_owner not like '%network:router%' and device_owner not like '%network:dhcp%' group by device_id having count(*)>1;

[Leontii Istomin (listomin)]

Used 6.1-425 build
I've tried to run 3000 instances with concurrency=5. Actually only 834 VMs are in ACTIVE state. Also 166 VMs aren't in ACTIVE state.
The issue wasn't reproduced:
root@node-53:~# mysql -e "use neutron; select * from ports where device_owner not like '%network:router%' and device_owner not like '%network:dhcp%' group by device_id having count(*)>1;"
root@node-53:~#

rally logs and results are attached.