Ports cannot be mapped to networks

Bug #1405131 reported by Mark Goddard on 2014-12-23
This bug report is a duplicate of:  Bug #1666009: [RFE] Physical network awareness. Edit Remove
52
This bug affects 9 people
Affects Status Importance Assigned to Milestone
Ironic
Confirmed
High
Unassigned
OpenStack Compute (nova)
Low
Unassigned

Bug Description

I have a cluster of bare metal nodes managed by Icehouse OpenStack on CentOS 6.5. I'm using nova, ironic and neutron. The nodes each have a 1G management network interface and a 40G high speed data network interface. These interfaces are registered with their MAC addresses as ironic ports. There is a single flat neutron network for management and a single flat neutron network for the high speed network.

When booting a (baremetal) nova instance, if the instance's networks are specified by their network ID, there is no way (to my knowledge) to guarantee a mapping between interface and network. In 50% of cases, I will end up with a neutron port on the management network, with the MAC address of the node's 40G NIC, and vice versa. The implication of this is that DHCP address assignment fails, and the instance fails to provision with the following error message:

2014-12-23 05:03:20.877 227052 ERROR ironic.conductor.manager [-] Timeout reached when waiting callback for node 49f8b733-525d-44a2-ae19-596b19aa5d1a

My workaround is to manually create neutron ports for each network interface, specifying a MAC address and the appropriate neutron network. These ports are then referenced when booting an instance, instead of specifying a network ID. This workaround is not ideal, in particular because it only works when the nodes are pets (as opposed to cattle).

Based on my browsing of the nova source, I think the point of ill decision occurs at https://github.com/openstack/nova/blob/06c537fbe5bb4ac5a3012642c899df815872267c/nova/network/neutronv2/api.py#L270.

Simple minded, misguided half solution - tag the ironic ports with some metadata, such as the provider:physical_network tag of the neutron network to which they are attached. This can be used in nova to select an appropriate HW port for each of the instance's networks.

Mark Goddard (mgoddard) wrote :

I have implemented the simple solution outlined in the description and it does indeed solve the problem. The implementation isn't perfect and would require some tweaking if it were to be submitted.

Changed in nova:
status: New → Confirmed
importance: Undecided → Low
tags: added: network
Mark Goddard (mgoddard) on 2015-02-04
Changed in nova:
assignee: nobody → Mark Goddard (mgoddard)
Tan Lin (tan-lin-good) wrote :

Hi Mark,

I also work on a similar bug on this https://bugs.launchpad.net/ironic/+bug/1412454,
It would be great if you can propose your fix.

Tan

Download full text (3.3 KiB)

Hi Tan,

Please help me to understand your bug. I think it is as follows:

- Ironic node has two ports.
- Nova instance has a single NIC, so creates only a single Neutron port.
- There is no way to control which ironic port neutron maps its port to.

Is that correct?

I have attached our internal fix for bug 1405131 to the bug for discussion. I am currently at the ironic mid-cycle meetup and am discussing with Sylvain Bauza from the Nova team how to approach getting a fix upstream.

Mark Goddard | Software Engineer, HPC Systems.
<email address hidden>
178-180 Hotwell Road | Bristol | BS8 4RP | UK | www.cray.com

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Tan Lin
Sent: 04 February 2015 13:28
To: Mark Goddard
Subject: [Bug 1405131] Re: Ports cannot be mapped to networks

Hi Mark,

I also work on a similar bug on this https://bugs.launchpad.net/ironic/+bug/1412454,
It would be great if you can propose your fix.

Tan

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1405131

Title:
  Ports cannot be mapped to networks

Status in OpenStack Bare Metal Provisioning Service (Ironic):
  New
Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  I have a cluster of bare metal nodes managed by Icehouse OpenStack on
  CentOS 6.5. I'm using nova, ironic and neutron. The nodes each have a
  1G management network interface and a 40G high speed data network
  interface. These interfaces are registered with their MAC addresses as
  ironic ports. There is a single flat neutron network for management
  and a single flat neutron network for the high speed network.

  When booting a (baremetal) nova instance, if the instance's networks
  are specified by their network ID, there is no way (to my knowledge)
  to guarantee a mapping between interface and network. In 50% of cases,
  I will end up with a neutron port on the management network, with the
  MAC address of the node's 40G NIC, and vice versa. The implication of
  this is that DHCP address assignment fails, and the instance fails to
  provision with the following error message:

  2014-12-23 05:03:20.877 227052 ERROR ironic.conductor.manager [-]
  Timeout reached when waiting callback for node 49f8b733-525d-
  44a2-ae19-596b19aa5d1a

  My workaround is to manually create neutron ports for each network
  interface, specifying a MAC address and the appropriate neutron
  network. These ports are then referenced when booting an instance,
  instead of specifying a network ID. This workaround is not ideal, in
  particular because it only works when the nodes are pets (as opposed
  to cattle).

  Based on my browsing of the nova source, I think the point of ill
  decision occurs at
  https://github.com/openstack/nova/blob/06c537fbe5bb4ac5a3012642c899df815872267c/nova/network/neutronv2/api.py#L270.

  Simple minded, misguided half solution - tag the ironic ports with
  some metadata, such as the provider:physical_network tag of the
  neutron network to which they are attached. This can be used i...

Read more...

Fix proposed to branch: master
Review: https://review.openstack.org/153230

Changed in nova:
status: Confirmed → In Progress
Dmitry Tantsur (divius) on 2015-02-16
Changed in ironic:
status: New → Confirmed
importance: Undecided → High
Mark Goddard (mgoddard) wrote :

The change that has been proposed to nova to fix this issue is incomplete but demonstrates the general idea of the proposed solution. It would be great to get some reviews on this to check that the design is sound before tidying up the rough edges.

devananda (devananda) on 2015-03-27
Changed in ironic:
milestone: none → kilo-rc1
devananda (devananda) wrote :

Bumping to Liberty after further thought, as I'd like more time for us to vet the solution here. It's going to require some coordination between the Nova driver, Ironic's API (what the metadata name is), and documenting how to use it properly (ie, the user must issue additional commands to Neutron to create the networks, then to Ironic to save the right data, before calling "nova boot").

Changed in ironic:
milestone: kilo-rc1 → liberty-1
Changed in ironic:
assignee: nobody → John Stafford (john-stafford)
Changed in nova:
assignee: Mark Goddard (mgoddard) → Devananda van der Veen (devananda)

Reviewed: https://review.openstack.org/169258
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=87cc3928c44c46c14183fcac062a0e48c7271535
Submitter: Jenkins
Branch: master

commit 87cc3928c44c46c14183fcac062a0e48c7271535
Author: Devananda van der Veen <email address hidden>
Date: Tue Mar 31 03:37:13 2015 -0700

    Fix incorrect statement in inline neutronv2 docs

    The function doc header for network/neutron/v2/api.py _create_port()
    states that it accepts an optional set of MAC addresses to use.

    However, it only uses one, which is selected by pop().

    This commit merely changes the doc string to indicate that the function
    operates on only a single MAC address, not on the whole set.

    Change-Id: I0fc845572836b6a6773cc7beaeefd7b0d1cfd5e7
    Related-bug: #1405131

Changed in ironic:
assignee: John Stafford (john-stafford) → nobody
Michael Still (mikal) on 2015-05-06
tags: added: ironic
Changed in nova:
assignee: Devananda van der Veen (devananda) → Mark Goddard (mgoddard)
John L. Villalovos (happycamp) wrote :

Need to wait for this work to get done before can proceed:

https://blueprints.launchpad.net/ironic/+spec/ironic-ml2-integration

Mark Goddard (mgoddard) wrote :

This one has been kicking around for a while now and is a pretty major shortcoming IMHO. Can we try to push this forward in Mitaka? I'm happy to help out, whether using the solution I've proposed (and implemented) or otherwise.

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/153230
Reason: This patch is very old and appears to not be active any more. I am therefore abandoning it to keep the nova review queue sane. Feel free to restore the change when you're actively working on it again.

Changed in nova:
assignee: Mark Goddard (mgoddard) → nobody
status: In Progress → Confirmed
sergiiF (framin) wrote :

FYI.
We had kind of the same issue.
https://bugs.launchpad.net/ironic/+bug/1479128
Solved it differently, patching nova code.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers