node provisioning with ansible fails

[Mihai Plasoianu]

Description
===========
Node provisioning fails due to this line:
https://opendev.org/openstack/tripleo-ansible/src/commit/1bc900c749b23974e4156c1d56875b61a0742c6f/tripleo_ansible/playbooks/cli-overcloud-node-provide.yaml#L54

Steps to reproduce
==================
* Install Undercloud (master on CentOS 8)
* Introspect nodes
* Provision nodes with `openstack overcloud node provide --all-manageable`

Expected result
===============
Nodes should move to available state

Actual result
=============
Provisioning fails with error here:
https://opendev.org/openstack/tripleo-ansible/src/commit/1bc900c749b23974e4156c1d56875b61a0742c6f/tripleo_ansible/ansible_plugins/modules/os_baremetal_provide_node.py#L374-L375

All ironic-neutron-agents have the following configuration:
{'bridge_mappings': {}, 'log_agent_heartbeats': False}

Because bridge_mappings is empty, the provisioning fails. I don't know exactly under what circumstances bridge_mappings can be empty and if this is an issue at all. I changed wait_for_bridge_mappings to false and got it working again. What are the implications here?

[Harald Jensås (harald-jensas)]

How was the nodes imported?
What was in instackenv.json ?

[Mihai Plasoianu (m1p)]

All nodes have the same structure in instackenv.json, e.g.:
    {
      "name": "lab-controller-0",
      "pm_type": "idrac",
      "pm_addr": "192.168.100.101",
      "pm_user": "root",
      "pm_password": "secret",
      "capabilities": "node:controller-0,profile:control,boot_mode:uefi,boot_option:local"
    },

They're being imported with:
openstack overcloud node import instackenv.json
openstack overcloud node introspect --all-manageable
openstack overcloud node provide --all-manageable

[Steve Baker (steve-stevebaker)]

Can you paste the output of one baremetal port?
  openstack baremetal port list
  openstack baremetal port show <uuid>

[Harald Jensås (harald-jensas)]

Ok, so no port's where defined in instackenv.json. The interfaces for the baremetal node are discovered during introspection and the baremetal port objects are added in ironic by inspector.

When this happens the 'physical_network' field of the baremetal ports are not populated, resulting in an empty value for the agents 'bridge_mappings'.

You can set the physical_network property of the ports with:

openstack baremetal port set --physical-network $PHYSNET $PORT_UUID

In a tripleo undercloud you want to use PHYSNET=ctlplane in most cases. Only routed topologies such as spine-and-leaf and DCN use other physical_networks.

We probably want to log a better error message with instructions to set the physical_network field.

[Mihai Plasoianu (m1p)]

We deployed train+centos7 again in our lab to fix other bugs. I'll give master+centos8 another go next week, let's see if the issue is still there, then I can paste the additional info.

[Mihai Plasoianu (m1p)]

I just checked Train on CentOS7.

(undercloud) [<email address hidden> ~]$ openstack baremetal port show 347beced-67f3-4650-97ab-32202b387336
[...]
| physical_network | None |
[...]

Apparently, physical_network is not set during introspection, even though the docs suggest it:
https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/environments/baremetal.html#instackenv

[Harald Jensås (harald-jensas)]

@Mihai, I see the doc's are not fully clear. There is currently no way to auto-populate the physical_network when ironic node ports are discovered during introspection.

As a workaround set the attribute with:
  openstack baremetal port set --physical-network $PHYSNET $PORT_UUID

Alternatively add the ports in instackenv.json, to have the field set on node import. i.e

    {
      "name": "lab-controller-0",
      "pm_type": "idrac",
      "pm_addr": "192.168.100.101",
      "pm_user": "root",
      "pm_password": "secret",
      "capabilities": "node:controller-0,profile:control,boot_mode:uefi,boot_option:local",
      "ports": [{"address": "11:11:11:11:11:11"}]
    },

   ^^ This will set the physical_network to 'ctlplane' on import.
   If you need to set a different physical_network it can be specified in instackenv.json as well, with:

    {
      "name": "lab-controller-0",
      "pm_type": "idrac",
      "pm_addr": "192.168.100.101",
      "pm_user": "root",
      "pm_password": "secret",
      "capabilities": "node:controller-0,profile:control,boot_mode:uefi,boot_option:local",
      "ports": [{"address": "11:11:11:11:11:11", "physical_network": "some_net"}]
    },

I have also started work on a fix. Initially ironic Inspector need support to populate the physical_network field. My proposed patches does this based on a CIDR map in inspectors config. In TripleO we can populate the CIDR map in ironic Inspector config internally based on the existing undercloud.conf data. See: https://review.opendev.org/720010 and https://review.opendev.org/718594

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/722088

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/722275

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/722088
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2c2f8c4c79c40a7319dc17d335083d19a724f916
Submitter: Zuul
Branch: master

commit 2c2f8c4c79c40a7319dc17d335083d19a724f916
Author: Harald Jensås <email address hidden>
Date: Wed Apr 22 19:00:55 2020 +0200

    Ironic inspector port_physnet_cidr_map support

    Add support for laying out the configuration for
    the physnet_cidr_map processing hook in ironic
    inspector.

    Also enable the physnet_cidr_map processing hook
    by adding it to IronicInspectorExtraProcessingHooks
    default.

    Depends-On: https://review.opendev.org/722066
    Related-Bug: #1870529
    Change-Id: I5f0f3b254edf25fdcfa328f42d8bf4324e5bbd0d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/722275
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=9cd91775f249bd1e171b3f5c81585a64d01f9e9f
Submitter: Zuul
Branch: master

commit 9cd91775f249bd1e171b3f5c81585a64d01f9e9f
Author: Harald Jensås <email address hidden>
Date: Thu Apr 23 10:52:44 2020 +0200

    Inspector set baremetal port physical_network

    When deploying routed networks the physical_network
    attribute of ironic ports must be set, this has been
    a manual task.

    With this change we add configuration for the
    physnet_cidr_map processing hook in ironic inspector
    so that the physical_network field of ironic ports
    is set automatically when introspecting nodes.

    Depends-On: https://review.opendev.org/722088
    Closes-Bug: #1870529
    Change-Id: I7f31b9f0143507ec3c9c26efd086f0bd95ef2ce7