ports duplication in the VM XML when using heat and multiple networks

[David Edery]

On latest devstack with the following (part of) local.conf:
REGION_NAME=RegionOne
Q_PLUGIN=ml2
ENABLE_TENANT_VLANS=True
ML2_VLAN_RANGES=RegionOne:2:4090

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service h-eng h-api h-api-cfn h-api-cw

After a successful stack.sh, run:
neutron quota-update --network 100 --subnet 100 --port 100

Run the following script to create networks:
#!/bin/bash

NET_NUM=20

while [ $NET_NUM -gt 0 ]; do
    neutron net-create NET_${NET_NUM}
    neutron subnet-create --name 10.21.${NET_NUM}.0/24 --gateway 10.21.${NET_NUM}.1 NET_${NET_NUM} 10.21.${NET_NUM}.0/24
    NET_NUM=$[$NET_NUM-1]
done

Create an instance with 20 ports using the following HEAT template:
heat_template_version: 2013-05-23

description: >
    HOT template that just defines a single compute instance.
    Contains just base features.

resources:
    ovs_port_1:
        type: OS::Neutron::Port
        properties:
            network: NET_1

    ovs_port_2:
        type: OS::Neutron::Port
        properties:
            network: NET_2

    ovs_port_3:
        type: OS::Neutron::Port
        properties:
            network: NET_3

    ovs_port_4:
        type: OS::Neutron::Port
        properties:
            network: NET_4

    ovs_port_5:
        type: OS::Neutron::Port
        properties:
            network: NET_5

    ovs_port_6:
        type: OS::Neutron::Port
        properties:
            network: NET_6

    ovs_port_7:
        type: OS::Neutron::Port
        properties:
            network: NET_7

    ovs_port_8:
        type: OS::Neutron::Port
        properties:
            network: NET_8

    ovs_port_9:
        type: OS::Neutron::Port
        properties:
            network: NET_9

    ovs_port_10:
        type: OS::Neutron::Port
        properties:
            network: NET_10

    ovs_port_11:
        type: OS::Neutron::Port
        properties:
            network: NET_11

    ovs_port_12:
        type: OS::Neutron::Port
        properties:
            network: NET_12

    ovs_port_13:
        type: OS::Neutron::Port
        properties:
            network: NET_13

    ovs_port_14:
        type: OS::Neutron::Port
        properties:
            network: NET_14

    ovs_port_15:
        type: OS::Neutron::Port
        properties:
            network: NET_15

    ovs_port_16:
        type: OS::Neutron::Port
        properties:
            network: NET_16

    ovs_port_17:
        type: OS::Neutron::Port
        properties:
            network: NET_17

    ovs_port_18:
        type: OS::Neutron::Port
        properties:
            network: NET_18

    ovs_port_19:
        type: OS::Neutron::Port
        properties:
            network: NET_19

    ovs_port_20:
        type: OS::Neutron::Port
        properties:
            network: NET_20

    ovs_instance:
        type: OS::Nova::Server
        properties:
            image: cirros-0.3.4-x86_64-uec
            flavor: m1.nano
            networks:
                - port: { get_resource: ovs_port_1 }
                - port: { get_resource: ovs_port_2 }
                - port: { get_resource: ovs_port_3 }
                - port: { get_resource: ovs_port_4 }
                - port: { get_resource: ovs_port_5 }
                - port: { get_resource: ovs_port_6 }
                - port: { get_resource: ovs_port_7 }
                - port: { get_resource: ovs_port_8 }
                - port: { get_resource: ovs_port_9 }
                - port: { get_resource: ovs_port_10 }
                - port: { get_resource: ovs_port_11 }
                - port: { get_resource: ovs_port_12 }
                - port: { get_resource: ovs_port_13 }
                - port: { get_resource: ovs_port_14 }
                - port: { get_resource: ovs_port_15 }
                - port: { get_resource: ovs_port_16 }
                - port: { get_resource: ovs_port_17 }
                - port: { get_resource: ovs_port_18 }
                - port: { get_resource: ovs_port_19 }
                - port: { get_resource: ovs_port_20 }

The stack creation fails and nova conductor (n-cond) shows:
2015-12-02 20:07:26.944 ERROR nova.scheduler.utils [req-9ddafa19-5281-4873-8d08-ecee6e6ad0f0 admin admin] [instance: a8c3f7a2-4298-45a6-81c1-3eea81541ca4] Error from last host: devstack-Box (node devstack-Box): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1902, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2066, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance a8c3f7a2-4298-45a6-81c1-3eea81541ca4 was re-scheduled: Unable to create tap device tap0739f4dd-d0: Device or resource busy\n']

Looking at the generated VM XML file there are duplications of (some of) the ports which causes the above.

[David Edery (david-edery)]

This happens also in RH's distro (validated on Juno & Kilo)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252565

[Mark Goddard (mgoddard)]

This appears to be a duplicate of https://bugs.launchpad.net/nova/+bug/1501735.

I have a fix for that up for review which is different from the one proposed here, but I believe will also fix this issue. Could you try it out with your reproducer and validate that it works? I have provided a fairly in-depth analysis of the problem on the bug report. Please let me know your thoughts.

[David Edery (david-edery)]

Validated that the fix from https://bugs.launchpad.net/nova/+bug/1501735 also solves the stated issue. I'll mark as duplicate.
It was agreed with Mark that the safety mechanisms added in https://review.openstack.org/252565 can stay so that there will be no chance of duplication.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252565
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3031adb857993d8196b4c9febca51ac82cf35fd6
Submitter: Jenkins
Branch: master

commit 3031adb857993d8196b4c9febca51ac82cf35fd6
Author: David Edery <email address hidden>
Date: Wed Dec 2 19:56:54 2015 +0100

    ports & networks gather should validate existance

    _gather_port_ids_and_networks assumes that the input networks variable
    doesn't contain the networks in ifaces. This is a wrong assumption
    ever since the introduction of the "refresh_cache-<instance id>" locks to
    the process in the Liberty cycle (see related bugs) and the
    "Refactor network API 'get_instance_nw_info'" patchset
    (https://review.openstack.org/#/c/146036/).

    The fix validates that the networks stated in ifaces doen't exist in the
    gotten networks list.

    Duplicate networks were observed at the following closed/related bugs.

    Change-Id: I8c2c9e3c89babbe5e48c5129b9854013690b38f6
    Closes-Bug: #1522112
    Related-Bug: #1467581
    Related-Bug: #1501735

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/270686

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270686
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=adc21bb0c62db822c3d656c4721f7b7c3ad41b7a
Submitter: Jenkins
Branch: stable/liberty

commit adc21bb0c62db822c3d656c4721f7b7c3ad41b7a
Author: David Edery <email address hidden>
Date: Wed Dec 2 19:56:54 2015 +0100

    ports & networks gather should validate existance

    _gather_port_ids_and_networks assumes that the input networks variable
    doesn't contain the networks in ifaces. This is a wrong assumption
    ever since the introduction of the "refresh_cache-<instance id>" locks to
    the process in the Liberty cycle (see related bugs) and the
    "Refactor network API 'get_instance_nw_info'" patchset
    (https://review.openstack.org/#/c/146036/).

    The fix validates that the networks stated in ifaces doen't exist in the
    gotten networks list.

    Duplicate networks were observed at the following closed/related bugs.

    Change-Id: I8c2c9e3c89babbe5e48c5129b9854013690b38f6
    Closes-Bug: #1522112
    Related-Bug: #1467581
    Related-Bug: #1501735
    (cherry picked from commit 3031adb857993d8196b4c9febca51ac82cf35fd6)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 12.0.2 release.