[RFE] Physical network awareness

[Mark Goddard]

Ironic ports and portgroups correspond to the physical NICs that are available on baremetal nodes. Neutron ports are logical ports attached to tenant and provider networks. When booting baremetal nodes in a system with multiple physical networks, the mapping between logical and physical ports must account for the physical network connectivity of the system.

This feature aims to make ironic aware of the physical_network attribute of neutron networks and network segments.

https://review.openstack.org/#/c/435781/

[Ruby Loo (rloo)]

The spec https://review.openstack.org/#/c/435781/ has been approved.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/461301

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/464644

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/464644
Committed: https://git.openstack.org/cgit/openstack/ironic-specs/commit/?id=043193e27fbff84c323033753d460f967e4396f2
Submitter: Jenkins
Branch: master

commit 043193e27fbff84c323033753d460f967e4396f2
Author: Mark Goddard <email address hidden>
Date: Mon May 15 14:34:16 2017 +0100

    Update physical network-awareness spec

    While implementing this feature it was determined that storing a
    physical network field for both ports and portgroups would be redundant,
    since a portgroup's physical network should always be the same as all
    of its constituent ports.

    Change-Id: I1b8348df7e87b3b2fb812cd87eb13e35c3d6f2f3
    Related-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/469931

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/469932

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/469933

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/470406

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/461301
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=20bf354a76d138ace2dca045c26cf7188957948e
Submitter: Jenkins
Branch: master

commit 20bf354a76d138ace2dca045c26cf7188957948e
Author: Mark Goddard <email address hidden>
Date: Sun Apr 30 19:22:13 2017 +0100

    Add physical network to port data model

    This change implements the changes to the port data model required to
    support physical network awareness. This includes updating the ports
    table in the database, and extending the Port object.

    Change-Id: Ib22753aa6ae0fedce7fb9ecf63f135fda0185c5b
    Partial-Bug: 1666009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/469931
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=9e3f4121865807294dd3768b095b67eb01ef539e
Submitter: Jenkins
Branch: master

commit 9e3f4121865807294dd3768b095b67eb01ef539e
Author: Mark Goddard <email address hidden>
Date: Wed May 31 17:42:16 2017 +0100

    Move port object creation to conductor

    Previously, the API service created port objects without hitting the
    conductor. This change moves port creation to the conductor service,
    adding a create_port method.

    Currently this just performs the port object creation but in a future
    change this will be used to validate the physical network assignment of
    ports in a portgroup when a port is created, ensuring that all ports in
    the group have the same physical network.

    The conductor RPC API version has been bumped to 1.41.

    Change-Id: I7501bf9fedc668629d5b627475bb54fef5c6bf20
    Partial-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/469932
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=039225610d220ddb3d9d1edf4a74c8cb2a0de4fd
Submitter: Jenkins
Branch: master

commit 039225610d220ddb3d9d1edf4a74c8cb2a0de4fd
Author: Mark Goddard <email address hidden>
Date: Thu Jun 1 15:24:44 2017 +0100

    Validate portgroup physical network consistency

    When creating or updating a port that is a member of a portgroup, we
    need to validate the consistency of the physical networks of the ports
    in the portgroup.

    There are 3 cases we are interested in:

    - Creating a port which is a member of a portgroup.
    - Updating the physical network of a port which is a member of a
      portgroup.
    - Updating the portgroup of a port.

    All ports in a portgroup should have the same value (which may be None)
    for their physical_network field.

    During creation or update of a port in a portgroup we apply the
    following validation criteria:

    - If the portgroup has existing ports with different physical networks,
      we raise PortgroupPhysnetInconsistent. This shouldn't ever happen.
    - If the port has a physical network that is inconsistent with other
      ports in the portgroup, we raise exception.Conflict.

    If a port's physical network is None, this indicates that ironic's VIF
    attachment mapping algorithm should operate in a legacy (physical
    network unaware) mode for this port or portgroup. This allows existing
    ironic nodes to continue to function after an upgrade to a release
    including physical network support.

    Change-Id: I6a6d248155f98109dd36dba5837494f6974846e6
    Partial-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/474921

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/474921
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=50e1aa1d44532add607fa4b6d2d0b0af725e24f3
Submitter: Jenkins
Branch: master

commit 50e1aa1d44532add607fa4b6d2d0b0af725e24f3
Author: Mark Goddard <email address hidden>
Date: Fri Jun 16 10:22:34 2017 +0100

    Remove unnecessary line in docstring

    Accidentally added in https://review.openstack.org/#/c/469932.

    Change-Id: Ic00fc89443da95eb29338a5477f3af19b9d02aaf
    Related-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/477694

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/482080

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470406
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=b9b820954da825f3c80c782fcffcaa5dd4643c22
Submitter: Jenkins
Branch: master

commit b9b820954da825f3c80c782fcffcaa5dd4643c22
Author: Mark Goddard <email address hidden>
Date: Fri May 12 18:00:02 2017 +0100

    Physical network aware VIF attachment

    When attaching virtual interfaces to ironic ports and portgroups, we
    need to take account of the physical network assigned to those ports and
    portgroups. A neutron virtual interface has a set of physical networks
    on which it can be attached which is governed by the segments of its
    network (of which there may be more than one).

    This change makes the ironic VIF attach API physical network-aware using
    the physical network information added to the port object.

    When selecting a port or portgroup to attach a virtual interface to, the
    following ordered criteria are applied:

    * Require ports or portgroups to have a physical network that is
      None or one of the VIF's allowed physical networks.
    * Prefer ports or portgroups with a physical network field which
      is not None.
    * Prefer portgroups to ports.
    * Prefer ports with PXE enabled.

    The change is backwards compatible, as the old behaviour is maintained
    when ports have a physical_network field which is None.

    Change-Id: I3d13bfacfb5578f570791e3c06e769a9a0140a4c
    Partial-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/482080
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=761899776b77b162907ec12713e0b1f8f90e133a
Submitter: Jenkins
Branch: master

commit 761899776b77b162907ec12713e0b1f8f90e133a
Author: Mark Goddard <email address hidden>
Date: Mon Jul 10 10:48:31 2017 +0100

    Fix copy/paste error in VIF attach note

    This nit was picked up in the review of
    I3d13bfacfb5578f570791e3c06e769a9a0140a4c.

    Change-Id: I8fc1b7f4c319f306bdd62c9a99497f3a014dabd7
    Related-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/469933
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=02fff930fbab4caaa1181da9ee0f70523fe8d7ca
Submitter: Jenkins
Branch: master

commit 02fff930fbab4caaa1181da9ee0f70523fe8d7ca
Author: Mark Goddard <email address hidden>
Date: Thu Jun 1 15:36:58 2017 +0100

    Expose ports' physical network attribute in API

    In change Ib22753aa6ae0fedce7fb9ecf63f135fda0185c5b the port data model
    was updated to include a physical_network field, but this was not
    exposed to the user by the REST API. This change exposes the
    physical_network field in the REST API.

    The port CRUD notification object has been updated to include the
    physical_network field.

    The API reference and user guide have been updated to include
    information about the ports' physical network field.

    The API microversion has been bumped to 1.34. During a rolling upgrade
    from Ocata when the API service is pinned, the port physical network
    field is hidden from API responses, and API requests including the field
    are rejected.

    Change-Id: I7023a1d6618608c867c31396fa677d3016ca493e
    Partial-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/485527

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/482982
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=bfd80a5d39ba4ad153600ea044e09333a335517a
Submitter: Jenkins
Branch: master

commit bfd80a5d39ba4ad153600ea044e09333a335517a
Author: Mark Goddard <email address hidden>
Date: Wed Jul 12 14:57:18 2017 +0100

    Improve port update API unit tests

    Currently, the RPCAPI update_port method is mocked, and its return
    value set in the unit tests to the expected value - the modified port.
    This isn't really exercising all of the port update API handler, which
    should be modifying the port object appropriately and passing it to the
    RPCAPI update_port method.

    This change adds a side effect to the RPCAPI update_port mock which
    saves the Port object that it is passed to the DB. This allows us to
    avoid fudging the answer and test the code more thoroughly.

    The TestPost test case already does this for port creation.

    Change-Id: I77860b2a24da659418f93c380db67ff4726257ff
    Related-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mark Goddard (<email address hidden>) on branch: master
Review: https://review.openstack.org/485527
Reason: Abandoning this in favour of a specific solution in [1] after discussion with Ruby.

[1] https://review.openstack.org/#/c/477694

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/485772

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/485773

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470915
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=975040b3e12eedb8cb0bfaa38544685ee1c7209e
Submitter: Jenkins
Branch: master

commit 975040b3e12eedb8cb0bfaa38544685ee1c7209e
Author: Mark Goddard <email address hidden>
Date: Mon Jun 5 11:35:28 2017 +0100

    Add tempest tests for physical networks

    This change adds tests to the ironic tempest plugin to cover the
    API changes made for the physical network awareness feature in
    I7023a1d6618608c867c31396fa677d3016ca493e.

    Change-Id: I8b30764d797f2f8b45c2ae46ce559e74e0281a49
    Partial-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/487550

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/477694
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=7e4030600fe28f5f641e9743462700a28156c40e
Submitter: Jenkins
Branch: master

commit 7e4030600fe28f5f641e9743462700a28156c40e
Author: Ruby Loo <email address hidden>
Date: Mon Jun 26 18:11:48 2017 -0400

    Rolling upgrades support for port.physical_network

    The new field 'physical_network' was added to Port in version
    1.7, and port creation was moved from the API to the conductor
    service in commit 9e3f4121865807294dd3768b095b67eb01ef539e.

    This adds objects.port.Port._convert_to_version(). The method handles
    converting the new physical_network field between different versions of
    the Port.

    Change-Id: I8a5b1e3dd72884a511b43839cdb264354bc46ea0
    Co-Authored-By: Mark Goddard <email address hidden>
    Related-Bug: #1666009
    Related-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/485772
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=43fa6fc294daaf754c426476463e6f41dad6b93b
Submitter: Jenkins
Branch: master

commit 43fa6fc294daaf754c426476463e6f41dad6b93b
Author: Mark Goddard <email address hidden>
Date: Tue Jul 11 13:34:33 2017 +0100

    Refactor get_physnets_by_portgroup_id

    There is a lot of common code between the functions
    ironic.conductor.utils.validate_port_physnets() and
    ironic.drivers.modules.network.common._get_physnets_by_portgroup().

    This change refactors the code, adding a new utility method,
    get_physnets_by_portgroup_id, which returns the physical networks
    associated with a portgroup. There should be at most one such physical
    network, and the presence of multiple will cause
    PortgroupPhysnetInconsistent to be raised.

    Change-Id: I8f01dcc5eaa0c8511ce77622e41db88e27791327
    Related-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/485773
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=fc1e18e3e4a28b1163e7a597ee8b07d465d7a174
Submitter: Jenkins
Branch: master

commit fc1e18e3e4a28b1163e7a597ee8b07d465d7a174
Author: Mark Goddard <email address hidden>
Date: Thu Jul 20 10:41:54 2017 +0100

    Rolling upgrades support for create_port RPCAPI

    The new field 'physical_network' was added to Port in version
    1.7, and port creation was moved from the API to the conductor
    service in commit 9e3f4121865807294dd3768b095b67eb01ef539e.

    This change adds a method can_send_create_port() to the conductor
    RPCAPI that allows the caller to determine whether the conductor is able
    to create ports. During a rolling upgrade this may return False, and the
    API will need to determine whether it is able to create the port locally
    as was done previously.

    The create_port RPC method was added to support validation of the
    physical_network field of ports in portgroups. A port may therefore be
    safely created in the API service if it is not a member of a portgroup.
    If the port being created is a member of a portgroup, then it cannot be
    safely validated by the API service and the request must be rejected.

    Change-Id: I8c417cba085f61c3d2ffe1f7e97c64fa85a014cb
    Related-Bug: #1666009
    Related-Bug: #1526283

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/487550
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=63a852eb8c4e3b885340aec62d19d6d1f15bf995
Submitter: Jenkins
Branch: master

commit 63a852eb8c4e3b885340aec62d19d6d1f15bf995
Author: Mark Goddard <email address hidden>
Date: Wed Jul 26 19:36:00 2017 +0100

    Add negative attribute to negative port tests

    Change I8b30764d797f2f8b45c2ae46ce559e74e0281a49 added tempest tests for
    the port.physical_network field. In that review, it was noted that the
    negative tests should have a decorator applied to mark them as negative.
    This change applies that decorator.

    Change-Id: I69446703894b89596183c50482a4286eaac4ed42
    Related-Bug: #1666009

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/489809

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/489811

[OpenStack Infra (hudson-openstack)]

Change abandoned by John L. Villalovos (<email address hidden>) on branch: master
Review: https://review.openstack.org/489811
Reason: Will work with infra to do a batch upload.

[OpenStack Infra (hudson-openstack)]

Change abandoned by John L. Villalovos (<email address hidden>) on branch: master
Review: https://review.openstack.org/489809
Reason: Will work with infra to do a batch upload.