ports duplication in the VM XML when using heat and multiple networks

Bug #1522112 reported by David Edery
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
David Edery
Liberty
Fix Released
Medium
David Edery

Bug Description

On latest devstack with the following (part of) local.conf:
REGION_NAME=RegionOne
Q_PLUGIN=ml2
ENABLE_TENANT_VLANS=True
ML2_VLAN_RANGES=RegionOne:2:4090

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service neutron
enable_service h-eng h-api h-api-cfn h-api-cw

After a successful stack.sh, run:
neutron quota-update --network 100 --subnet 100 --port 100

Run the following script to create networks:
#!/bin/bash

NET_NUM=20

while [ $NET_NUM -gt 0 ]; do
    neutron net-create NET_${NET_NUM}
    neutron subnet-create --name 10.21.${NET_NUM}.0/24 --gateway 10.21.${NET_NUM}.1 NET_${NET_NUM} 10.21.${NET_NUM}.0/24
    NET_NUM=$[$NET_NUM-1]
done

Create an instance with 20 ports using the following HEAT template:
heat_template_version: 2013-05-23

description: >
    HOT template that just defines a single compute instance.
    Contains just base features.

resources:
    ovs_port_1:
        type: OS::Neutron::Port
        properties:
            network: NET_1

    ovs_port_2:
        type: OS::Neutron::Port
        properties:
            network: NET_2

    ovs_port_3:
        type: OS::Neutron::Port
        properties:
            network: NET_3

    ovs_port_4:
        type: OS::Neutron::Port
        properties:
            network: NET_4

    ovs_port_5:
        type: OS::Neutron::Port
        properties:
            network: NET_5

    ovs_port_6:
        type: OS::Neutron::Port
        properties:
            network: NET_6

    ovs_port_7:
        type: OS::Neutron::Port
        properties:
            network: NET_7

    ovs_port_8:
        type: OS::Neutron::Port
        properties:
            network: NET_8

    ovs_port_9:
        type: OS::Neutron::Port
        properties:
            network: NET_9

    ovs_port_10:
        type: OS::Neutron::Port
        properties:
            network: NET_10

    ovs_port_11:
        type: OS::Neutron::Port
        properties:
            network: NET_11

    ovs_port_12:
        type: OS::Neutron::Port
        properties:
            network: NET_12

    ovs_port_13:
        type: OS::Neutron::Port
        properties:
            network: NET_13

    ovs_port_14:
        type: OS::Neutron::Port
        properties:
            network: NET_14

    ovs_port_15:
        type: OS::Neutron::Port
        properties:
            network: NET_15

    ovs_port_16:
        type: OS::Neutron::Port
        properties:
            network: NET_16

    ovs_port_17:
        type: OS::Neutron::Port
        properties:
            network: NET_17

    ovs_port_18:
        type: OS::Neutron::Port
        properties:
            network: NET_18

    ovs_port_19:
        type: OS::Neutron::Port
        properties:
            network: NET_19

    ovs_port_20:
        type: OS::Neutron::Port
        properties:
            network: NET_20

    ovs_instance:
        type: OS::Nova::Server
        properties:
            image: cirros-0.3.4-x86_64-uec
            flavor: m1.nano
            networks:
                - port: { get_resource: ovs_port_1 }
                - port: { get_resource: ovs_port_2 }
                - port: { get_resource: ovs_port_3 }
                - port: { get_resource: ovs_port_4 }
                - port: { get_resource: ovs_port_5 }
                - port: { get_resource: ovs_port_6 }
                - port: { get_resource: ovs_port_7 }
                - port: { get_resource: ovs_port_8 }
                - port: { get_resource: ovs_port_9 }
                - port: { get_resource: ovs_port_10 }
                - port: { get_resource: ovs_port_11 }
                - port: { get_resource: ovs_port_12 }
                - port: { get_resource: ovs_port_13 }
                - port: { get_resource: ovs_port_14 }
                - port: { get_resource: ovs_port_15 }
                - port: { get_resource: ovs_port_16 }
                - port: { get_resource: ovs_port_17 }
                - port: { get_resource: ovs_port_18 }
                - port: { get_resource: ovs_port_19 }
                - port: { get_resource: ovs_port_20 }

The stack creation fails and nova conductor (n-cond) shows:
2015-12-02 20:07:26.944 ERROR nova.scheduler.utils [req-9ddafa19-5281-4873-8d08-ecee6e6ad0f0 admin admin] [instance: a8c3f7a2-4298-45a6-81c1-3eea81541ca4] Error from last host: devstack-Box (node devstack-Box): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1902, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2066, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance a8c3f7a2-4298-45a6-81c1-3eea81541ca4 was re-scheduled: Unable to create tap device tap0739f4dd-d0: Device or resource busy\n']

Looking at the generated VM XML file there are duplications of (some of) the ports which causes the above.

Changed in neutron:
assignee: nobody → David Edery (david-edery)
affects: neutron → nova-project
no longer affects: neutron
Revision history for this message
David Edery (david-edery) wrote :

This happens also in RH's distro (validated on Juno & Kilo)

affects: nova-project → nova
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/252565

Changed in nova:
status: New → In Progress
Revision history for this message
Mark Goddard (mgoddard) wrote :

This appears to be a duplicate of https://bugs.launchpad.net/nova/+bug/1501735.

I have a fix for that up for review which is different from the one proposed here, but I believe will also fix this issue. Could you try it out with your reproducer and validate that it works? I have provided a fairly in-depth analysis of the problem on the bug report. Please let me know your thoughts.

Revision history for this message
David Edery (david-edery) wrote :

Validated that the fix from https://bugs.launchpad.net/nova/+bug/1501735 also solves the stated issue. I'll mark as duplicate.
It was agreed with Mark that the safety mechanisms added in https://review.openstack.org/252565 can stay so that there will be no chance of duplication.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/252565
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3031adb857993d8196b4c9febca51ac82cf35fd6
Submitter: Jenkins
Branch: master

commit 3031adb857993d8196b4c9febca51ac82cf35fd6
Author: David Edery <email address hidden>
Date: Wed Dec 2 19:56:54 2015 +0100

    ports & networks gather should validate existance

    _gather_port_ids_and_networks assumes that the input networks variable
    doesn't contain the networks in ifaces. This is a wrong assumption
    ever since the introduction of the "refresh_cache-<instance id>" locks to
    the process in the Liberty cycle (see related bugs) and the
    "Refactor network API 'get_instance_nw_info'" patchset
    (https://review.openstack.org/#/c/146036/).

    The fix validates that the networks stated in ifaces doen't exist in the
    gotten networks list.

    Duplicate networks were observed at the following closed/related bugs.

    Change-Id: I8c2c9e3c89babbe5e48c5129b9854013690b38f6
    Closes-Bug: #1522112
    Related-Bug: #1467581
    Related-Bug: #1501735

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/270686

Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/nova 13.0.0.0b2

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/270686
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=adc21bb0c62db822c3d656c4721f7b7c3ad41b7a
Submitter: Jenkins
Branch: stable/liberty

commit adc21bb0c62db822c3d656c4721f7b7c3ad41b7a
Author: David Edery <email address hidden>
Date: Wed Dec 2 19:56:54 2015 +0100

    ports & networks gather should validate existance

    _gather_port_ids_and_networks assumes that the input networks variable
    doesn't contain the networks in ifaces. This is a wrong assumption
    ever since the introduction of the "refresh_cache-<instance id>" locks to
    the process in the Liberty cycle (see related bugs) and the
    "Refactor network API 'get_instance_nw_info'" patchset
    (https://review.openstack.org/#/c/146036/).

    The fix validates that the networks stated in ifaces doen't exist in the
    gotten networks list.

    Duplicate networks were observed at the following closed/related bugs.

    Change-Id: I8c2c9e3c89babbe5e48c5129b9854013690b38f6
    Closes-Bug: #1522112
    Related-Bug: #1467581
    Related-Bug: #1501735
    (cherry picked from commit 3031adb857993d8196b4c9febca51ac82cf35fd6)

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 12.0.2

This issue was fixed in the openstack/nova 12.0.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.