networks have empty string IP values when disabling networks

[Tim Rozet]

I'm seeing an issue with Rocky deployments where networks seem to be getting empty strings as IPs (when deploying with all networks disabled except ctlplane). I first noticed it when I see this entry in my overcloud /etc/hosts file:
#rocky
# HEAT_HOSTS_START - Do not edit manually within this section!
192.0.2.10 overcloud.ctlplane.localdomain
192.0.2.10 overcloud.localdomain
192.0.2.10 overcloud.storage.localdomain
192.0.2.10 overcloud.internalapi.localdomain
192.0.2.10 overcloud.storagemgmt.localdomain
 overcloud-controller-0.opnfvlf.org overcloud-controller-0
192.0.2.3 overcloud-controller-0.ctlplane.opnfvlf.org overcloud-controller-0.ctlplane

It can be seen that the overcloud-controller-0.opnfvlf.org entry has an empty IP address.

Master seems to work fine and with the same network_data file produces:

# HEAT_HOSTS_START - Do not edit manually within this section!
192.0.2.28 overcloud.ctlplane.localdomain
192.0.2.28 overcloud.localdomain
192.0.2.28 overcloud.internalapi.localdomain
192.0.2.28 overcloud.storage.localdomain
192.0.2.28 overcloud.storagemgmt.localdomain
192.0.2.36 overcloud-controller-0.opnfvlf.org overcloud-controller-0
192.0.2.36 overcloud-controller-0.external.opnfvlf.org overcloud-controller-0.external
192.0.2.36 overcloud-controller-0.internalapi.opnfvlf.org overcloud-controller-0.internalapi
192.0.2.36 overcloud-controller-0.storage.opnfvlf.org overcloud-controller-0.storage
192.0.2.36 overcloud-controller-0.tenant.opnfvlf.org overcloud-controller-0.tenant
192.0.2.36 overcloud-controller-0.storagemgmt.opnfvlf.org overcloud-controller-0.storagemgmt
192.0.2.36 overcloud-controller-0.ctlplane.opnfvlf.org overcloud-controller-0.ctlplane

#ROCKY
# HEAT_HOSTS_START - Do not edit manually within this section!
192.0.2.10 overcloud.ctlplane.localdomain
192.0.2.10 overcloud.localdomain
192.0.2.10 overcloud.storage.localdomain
192.0.2.10 overcloud.internalapi.localdomain
192.0.2.10 overcloud.storagemgmt.localdomain
 overcloud-controller-0.opnfvlf.org overcloud-controller-0
192.0.2.3 overcloud-controller-0.ctlplane.opnfvlf.org overcloud-controller-0.ctlplane

###NET HOST MAP
  | CREATE_COMPLETE | 2018-12-19T19:09:48Z | overcloud-Controller-vdv3y4fuey25-0-knjascvgbmj5 |
| NetHostMap | overcloud-Controller-vdv3y4fuey25-0-knjascvgbmj5-NetHostMap-yaaokcwcbher

| attributes | {u'value': {u'ctlplane': {u'short': u'overcloud-controller-0.ctlplane', u'fqdn': u'overcloud-controller-0.ctlplane.opnfvlf.org'}, u'canonical': {u'short': [u'overcloud-controller-0'], u'fqdn': u'overcloud-controller-0.opnfvlf.org'}}}

###NET IP MAP

  | CREATE_COMPLETE | 2018-12-19T19:09:48Z | overcloud-Controller-vdv3y4fuey25-0-knjascvgbmj5 | NetIpMap

  | attributes | {u'net_ip_map': {u'tenant_uri': u'', u'storage_mgmt_uri': u'', u'ctlplane_uri': u'192.0.2.3', u'ctlplane_subnet': u'192.0.2.3/24', u'tenant_subnet': u'', u'storage': u'', u'internal_api_subnet': u'', u'storage_subnet': u'', u'external_subnet': u'', u'ctlplane': u'192.0.2.3', u'storage_mgmt_subnet': u'', u'external': u'', u'internal_api': u'', u'internal_api_uri': u'', u'storage_mgmt': u'', u'external_uri': u'', u'tenant': u'', u'storage_uri': u''}}

###Looking at hieradata internal_api is empty:
net_ip_map.json: "internal_api": "",
net_ip_map.json: "internal_api_subnet": "",
net_ip_map.json: "internal_api_uri": "",

### network data:
(undercloud) [stack@undercloud ~]$ cat network_data.yaml
- enabled: false
  ip_subnet: 11.0.0.0/24
  name: Tenant
  name_lower: tenant
  vip: false
- allocation_pools:
  - end: 192.168.37.199
    start: 192.168.37.10
  enabled: false
  gateway_ip: 192.168.37.1
  ip_subnet: 192.168.37.0/24
  name: External
  name_lower: external
  vip: true
- enabled: false
  ip_subnet: fd00:fd00:fd00:4000::/64
  name: InternalApi
  name_lower: internal_api
  vip: true
- enabled: false
  ip_subnet: 12.0.0.0/24
  name: Storage
  name_lower: storage
  vip: true
- allocation_pools:
  - end: 172.16.3.250
    start: 172.16.3.4
  enabled: false
  ip_subnet: 172.16.3.0/24
  name: StorageMgmt
  name_lower: storage_mgmt
  vip: true

### MASTER
# HEAT_HOSTS_START - Do not edit manually within this section!
192.0.2.28 overcloud.ctlplane.localdomain
192.0.2.28 overcloud.localdomain
192.0.2.28 overcloud.internalapi.localdomain
192.0.2.28 overcloud.storage.localdomain
192.0.2.28 overcloud.storagemgmt.localdomain
192.0.2.36 overcloud-controller-0.opnfvlf.org overcloud-controller-0
192.0.2.36 overcloud-controller-0.external.opnfvlf.org overcloud-controller-0.external
192.0.2.36 overcloud-controller-0.internalapi.opnfvlf.org overcloud-controller-0.internalapi
192.0.2.36 overcloud-controller-0.storage.opnfvlf.org overcloud-controller-0.storage
192.0.2.36 overcloud-controller-0.tenant.opnfvlf.org overcloud-controller-0.tenant
192.0.2.36 overcloud-controller-0.storagemgmt.opnfvlf.org overcloud-controller-0.storagemgmt
192.0.2.36 overcloud-controller-0.ctlplane.opnfvlf.org overcloud-controller-0.ctlplane

net_ip_map.json: "internal_api": "192.0.2.36",
net_ip_map.json: "internal_api_subnet": "192.0.2.36/24",
net_ip_map.json: "internal_api_uri": "192.0.2.36",

(undercloud) [stack@undercloud ~]$ cat network_data.yaml
- allocation_pools:
  - end: 192.168.37.199
    start: 192.168.37.10
  enabled: false
  gateway_ip: 192.168.37.1
  ip_subnet: 192.168.37.0/24
  name: External
  name_lower: external
  vip: true
- enabled: false
  ip_subnet: fd00:fd00:fd00:4000::/64
  name: InternalApi
  name_lower: internal_api
  vip: true
- enabled: false
  ip_subnet: 12.0.0.0/24
  name: Storage
  name_lower: storage
  vip: true
- enabled: false
  ip_subnet: 11.0.0.0/24
  name: Tenant
  name_lower: tenant
  vip: false
- allocation_pools:
  - end: 172.16.3.250
    start: 172.16.3.4
  enabled: false
  ip_subnet: 172.16.3.0/24
  name: StorageMgmt
  name_lower: storage_mgmt
  vip: true

[Rabi Mishra (rabi)]

I think that's because we don't create resources for disabled networks any more and {{role.name}}HostnameResolveNetwork maps to internal-api[1] that you've disabled in your network_data.yaml.

If you've to disable internal_api network, you've override ServiceNetMap to set {{role.name}}HostnameResolveNetwork to ctlplane (the only one enabled).

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/network/service_net_map.j2.yaml#L92

On master you're seeing the old behaviour as you're probably not using the latest.

[Tim Rozet (trozet)]

rocky version:
(undercloud) [stack@undercloud ~]$ rpm -q openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-9.1.1-0.20181219093830.1fb8761.el7.noarch

Will attach templates/deploy cmd.

[Tim Rozet (trozet)]

master version:
(undercloud) [stack@undercloud ~]$ rpm -q openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-10.1.1-0.20181203023850.b01b1a7.el7.noarch

[Harald Jensås (harald-jensas)]

I think Ceph roles are broken since change: https://review.openstack.org/614457 landed.

None of these roles have the InternalApi network by default:

https://github.com/openstack/tripleo-heat-templates/blob/master/roles/CephAll.yaml
https://github.com/openstack/tripleo-heat-templates/blob/master/roles/CephFile.yaml
https://github.com/openstack/tripleo-heat-templates/blob/master/roles/CephObject.yaml
https://github.com/openstack/tripleo-heat-templates/blob/master/roles/CephStorage.yaml

We don't have per-role ServiceNetMap, so anyone deploying Ceph with this patch will end up with an empty string for the IP value in /etc/hosts.

[Harald Jensås (harald-jensas)]

Sorry, ignore my previous comment, CephStorage role is handled in service_net_map.j2[1] and the HostnameResolveNetwork is actually per-role. Sorry.

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/network/service_net_map.j2.yaml#L87

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/627419

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/627420

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/627963

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/627980

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/628259

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/628265

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627419
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=9590e8450706b1f634f9d53496e53f9b7add601b
Submitter: Zuul
Branch: master

commit 9590e8450706b1f634f9d53496e53f9b7add601b
Author: Harald Jensås <email address hidden>
Date: Mon Dec 24 15:17:08 2018 +0100

    Allow network data with no entries

    When deploying without network isolation we should
    allow an empty network_data.yaml with no networks.

    In combination with tripleo-heat-templates change:
    Ifeb2d2d1acb43c16a5bf29e95965776494d61fef the
    tripleoclient can be update to use /dev/null as
    network data, and networks can be removed from
    roles/Standalone.yaml and roles/Undercloud.yaml.

    Related-Bug: #1809313
    Change-Id: I2e8135bc9389d3bf1a6ef01e273515af5c488a9a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627963
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3df5f8db1ecb2e276d8744ad796a029a0f0a5a6e
Submitter: Zuul
Branch: master

commit 3df5f8db1ecb2e276d8744ad796a029a0f0a5a6e
Author: Harald Jensås <email address hidden>
Date: Wed Jan 2 13:11:46 2019 +0100

    Fall back service_net_map to ctlplane

    Render the service_net_map to fall back to the ctlplane network when
    networks are disabled (or not defined) in network_data.yaml.

    Also use startswith for Ceph roles which should all have 'storage'
    as the HostnameResolveNetwork.

    Closes-Bug: #1809313
    Change-Id: I737d5656b113f7e2238fe7bb555cc2d4cb13877c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627420
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=53027484ae7cb5b34837ff1f0f49243f805eaa32
Submitter: Zuul
Branch: master

commit 53027484ae7cb5b34837ff1f0f49243f805eaa32
Author: Harald Jensås <email address hidden>
Date: Wed Jan 2 13:11:46 2019 +0100

    Skip templating disabled networks

    Ignore disabled networks when rendering templates.

    Add's the ctlplane network to maps to ensure we don't
    end up with no keys/values in map_replace functions.

    Also some Jinja cleanup:
     - Reduce the number of times we iterate over networks
       where we can.
     - Add's indentation to make the code easier to read.

    Related-Bug: #1809313
    Depends-On: I2e8135bc9389d3bf1a6ef01e273515af5c488a9a
    Change-Id: Ifeb2d2d1acb43c16a5bf29e95965776494d61fef

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/628259
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c1116e59c1f60f14fcf10301fe8e43b2b34e8caa
Submitter: Zuul
Branch: master

commit c1116e59c1f60f14fcf10301fe8e43b2b34e8caa
Author: Harald Jensås <email address hidden>
Date: Thu Jan 3 14:59:41 2019 +0100

    Add network data for the undercloud

    The undercloud needs the External network for the external VIP
    address. It uses the _from_pool template creating a fake neutron
    port.

    Other networks are not required, and should be removed to optimize
    the number of heat resources and plan rendering.

    NOTE: Skips validation of network_data_undercloud.yaml against
    network_data.yaml, as is already done to openshift and routed
    networks networks data examples.

    Related Bug: #1809313
    Change-Id: Ib11a134df93e59947168b40bc71fb1da9172d4ac

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/627980
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=51ad17ba1805ef0e5b50d40b2d22610f01f82935
Submitter: Zuul
Branch: master

commit 51ad17ba1805ef0e5b50d40b2d22610f01f82935
Author: Harald Jensås <email address hidden>
Date: Wed Jan 2 15:21:36 2019 +0100

    Add support for networks data in Standalone

    Standalone does not use any of the composable networks by
    default. Deploy Standaloen using /dev/null as network data
    so that these resources are not included when creating the
    plan.

    Undercloud uses only the External network for the external
    VIP. Deploy the undercloud using the Undercloud specific
    network_data_undercloud.yaml, ensures external_from_pool.yaml
    is in the plan.

    Related-Bug: #1809313
    Depends-On: Ib11a134df93e59947168b40bc71fb1da9172d4ac
    Change-Id: I102912851a3b9952daaf7c4d5a34a919f527f805

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/628265
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e8a53f56f21d3da28a2a82138d6c91a919dd353b
Submitter: Zuul
Branch: master

commit e8a53f56f21d3da28a2a82138d6c91a919dd353b
Author: Harald Jensås <email address hidden>
Date: Thu Jan 3 19:54:15 2019 +0100

    Remove networks from Undercloud and Standalone roles

    Change https://review.openstack.org/614457 added these
    networks because of the defaults in ServiceNetMap. With
    changes related to LP Bug #1809313 these are no longer
    required, as the ServiceNetMap fall's back to ctlplane
    when networks are not defined or disabled in networks
    data.

    Related-Bug: #1809313
    Depends-On: I102912851a3b9952daaf7c4d5a34a919f527f805
    Change-Id: Ic4f22692f93db4ce0db0f4fbc83eca6b492b28e7

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/696821

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696821
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2dfb57618991959e951a5c6e92149676ec62b96f
Submitter: Zuul
Branch: stable/queens

commit 2dfb57618991959e951a5c6e92149676ec62b96f
Author: Harald Jensås <email address hidden>
Date: Wed Jan 2 13:11:46 2019 +0100

    Fall back service_net_map to ctlplane

    Render the service_net_map to fall back to the ctlplane network when
    networks are disabled (or not defined) in network_data.yaml.

    Also use startswith for Ceph roles which should all have 'storage'
    as the HostnameResolveNetwork.

    Conflicts:
     network/service_net_map.j2.yaml

    Closes-Bug: #1809313
    Change-Id: I737d5656b113f7e2238fe7bb555cc2d4cb13877c
    (cherry picked from commit 3df5f8db1ecb2e276d8744ad796a029a0f0a5a6e)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/697145

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697145
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c08c29c0a23f18a6b80a946cf6286b13b4be4bc6
Submitter: Zuul
Branch: stable/rocky

commit c08c29c0a23f18a6b80a946cf6286b13b4be4bc6
Author: Harald Jensås <email address hidden>
Date: Wed Jan 2 13:11:46 2019 +0100

    Fall back service_net_map to ctlplane

    Render the service_net_map to fall back to the ctlplane network when
    networks are disabled (or not defined) in network_data.yaml.

    Also use startswith for Ceph roles which should all have 'storage'
    as the HostnameResolveNetwork.

    Conflicts:
     network/service_net_map.j2.yaml

    Closes-Bug: #1809313
    Change-Id: I737d5656b113f7e2238fe7bb555cc2d4cb13877c
    (cherry picked from commit 3df5f8db1ecb2e276d8744ad796a029a0f0a5a6e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.