networks have empty string IP values when disabling networks

Bug #1809313 reported by Tim Rozet on 2018-12-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Undecided
Harald Jensås

Bug Description

I'm seeing an issue with Rocky deployments where networks seem to be getting empty strings as IPs (when deploying with all networks disabled except ctlplane). I first noticed it when I see this entry in my overcloud /etc/hosts file:
#rocky
# HEAT_HOSTS_START - Do not edit manually within this section!
192.0.2.10 overcloud.ctlplane.localdomain
192.0.2.10 overcloud.localdomain
192.0.2.10 overcloud.storage.localdomain
192.0.2.10 overcloud.internalapi.localdomain
192.0.2.10 overcloud.storagemgmt.localdomain
 overcloud-controller-0.opnfvlf.org overcloud-controller-0
192.0.2.3 overcloud-controller-0.ctlplane.opnfvlf.org overcloud-controller-0.ctlplane

It can be seen that the overcloud-controller-0.opnfvlf.org entry has an empty IP address.

Master seems to work fine and with the same network_data file produces:

# HEAT_HOSTS_START - Do not edit manually within this section!
192.0.2.28 overcloud.ctlplane.localdomain
192.0.2.28 overcloud.localdomain
192.0.2.28 overcloud.internalapi.localdomain
192.0.2.28 overcloud.storage.localdomain
192.0.2.28 overcloud.storagemgmt.localdomain
192.0.2.36 overcloud-controller-0.opnfvlf.org overcloud-controller-0
192.0.2.36 overcloud-controller-0.external.opnfvlf.org overcloud-controller-0.external
192.0.2.36 overcloud-controller-0.internalapi.opnfvlf.org overcloud-controller-0.internalapi
192.0.2.36 overcloud-controller-0.storage.opnfvlf.org overcloud-controller-0.storage
192.0.2.36 overcloud-controller-0.tenant.opnfvlf.org overcloud-controller-0.tenant
192.0.2.36 overcloud-controller-0.storagemgmt.opnfvlf.org overcloud-controller-0.storagemgmt
192.0.2.36 overcloud-controller-0.ctlplane.opnfvlf.org overcloud-controller-0.ctlplane

#ROCKY
# HEAT_HOSTS_START - Do not edit manually within this section!
192.0.2.10 overcloud.ctlplane.localdomain
192.0.2.10 overcloud.localdomain
192.0.2.10 overcloud.storage.localdomain
192.0.2.10 overcloud.internalapi.localdomain
192.0.2.10 overcloud.storagemgmt.localdomain
 overcloud-controller-0.opnfvlf.org overcloud-controller-0
192.0.2.3 overcloud-controller-0.ctlplane.opnfvlf.org overcloud-controller-0.ctlplane

###NET HOST MAP
  | CREATE_COMPLETE | 2018-12-19T19:09:48Z | overcloud-Controller-vdv3y4fuey25-0-knjascvgbmj5 |
| NetHostMap | overcloud-Controller-vdv3y4fuey25-0-knjascvgbmj5-NetHostMap-yaaokcwcbher

| attributes | {u'value': {u'ctlplane': {u'short': u'overcloud-controller-0.ctlplane', u'fqdn': u'overcloud-controller-0.ctlplane.opnfvlf.org'}, u'canonical': {u'short': [u'overcloud-controller-0'], u'fqdn': u'overcloud-controller-0.opnfvlf.org'}}}

###NET IP MAP

  | CREATE_COMPLETE | 2018-12-19T19:09:48Z | overcloud-Controller-vdv3y4fuey25-0-knjascvgbmj5 | NetIpMap

  | attributes | {u'net_ip_map': {u'tenant_uri': u'', u'storage_mgmt_uri': u'', u'ctlplane_uri': u'192.0.2.3', u'ctlplane_subnet': u'192.0.2.3/24', u'tenant_subnet': u'', u'storage': u'', u'internal_api_subnet': u'', u'storage_subnet': u'', u'external_subnet': u'', u'ctlplane': u'192.0.2.3', u'storage_mgmt_subnet': u'', u'external': u'', u'internal_api': u'', u'internal_api_uri': u'', u'storage_mgmt': u'', u'external_uri': u'', u'tenant': u'', u'storage_uri': u''}}

###Looking at hieradata internal_api is empty:
net_ip_map.json: "internal_api": "",
net_ip_map.json: "internal_api_subnet": "",
net_ip_map.json: "internal_api_uri": "",

### network data:
(undercloud) [stack@undercloud ~]$ cat network_data.yaml
- enabled: false
  ip_subnet: 11.0.0.0/24
  name: Tenant
  name_lower: tenant
  vip: false
- allocation_pools:
  - end: 192.168.37.199
    start: 192.168.37.10
  enabled: false
  gateway_ip: 192.168.37.1
  ip_subnet: 192.168.37.0/24
  name: External
  name_lower: external
  vip: true
- enabled: false
  ip_subnet: fd00:fd00:fd00:4000::/64
  name: InternalApi
  name_lower: internal_api
  vip: true
- enabled: false
  ip_subnet: 12.0.0.0/24
  name: Storage
  name_lower: storage
  vip: true
- allocation_pools:
  - end: 172.16.3.250
    start: 172.16.3.4
  enabled: false
  ip_subnet: 172.16.3.0/24
  name: StorageMgmt
  name_lower: storage_mgmt
  vip: true

### MASTER
# HEAT_HOSTS_START - Do not edit manually within this section!
192.0.2.28 overcloud.ctlplane.localdomain
192.0.2.28 overcloud.localdomain
192.0.2.28 overcloud.internalapi.localdomain
192.0.2.28 overcloud.storage.localdomain
192.0.2.28 overcloud.storagemgmt.localdomain
192.0.2.36 overcloud-controller-0.opnfvlf.org overcloud-controller-0
192.0.2.36 overcloud-controller-0.external.opnfvlf.org overcloud-controller-0.external
192.0.2.36 overcloud-controller-0.internalapi.opnfvlf.org overcloud-controller-0.internalapi
192.0.2.36 overcloud-controller-0.storage.opnfvlf.org overcloud-controller-0.storage
192.0.2.36 overcloud-controller-0.tenant.opnfvlf.org overcloud-controller-0.tenant
192.0.2.36 overcloud-controller-0.storagemgmt.opnfvlf.org overcloud-controller-0.storagemgmt
192.0.2.36 overcloud-controller-0.ctlplane.opnfvlf.org overcloud-controller-0.ctlplane

net_ip_map.json: "internal_api": "192.0.2.36",
net_ip_map.json: "internal_api_subnet": "192.0.2.36/24",
net_ip_map.json: "internal_api_uri": "192.0.2.36",

(undercloud) [stack@undercloud ~]$ cat network_data.yaml
- allocation_pools:
  - end: 192.168.37.199
    start: 192.168.37.10
  enabled: false
  gateway_ip: 192.168.37.1
  ip_subnet: 192.168.37.0/24
  name: External
  name_lower: external
  vip: true
- enabled: false
  ip_subnet: fd00:fd00:fd00:4000::/64
  name: InternalApi
  name_lower: internal_api
  vip: true
- enabled: false
  ip_subnet: 12.0.0.0/24
  name: Storage
  name_lower: storage
  vip: true
- enabled: false
  ip_subnet: 11.0.0.0/24
  name: Tenant
  name_lower: tenant
  vip: false
- allocation_pools:
  - end: 172.16.3.250
    start: 172.16.3.4
  enabled: false
  ip_subnet: 172.16.3.0/24
  name: StorageMgmt
  name_lower: storage_mgmt
  vip: true

Rabi Mishra (rabi) wrote :

I think that's because we don't create resources for disabled networks any more and {{role.name}}HostnameResolveNetwork maps to internal-api[1] that you've disabled in your network_data.yaml.

If you've to disable internal_api network, you've override ServiceNetMap to set {{role.name}}HostnameResolveNetwork to ctlplane (the only one enabled).

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/network/service_net_map.j2.yaml#L92

On master you're seeing the old behaviour as you're probably not using the latest.

Tim Rozet (trozet) wrote :

rocky version:
(undercloud) [stack@undercloud ~]$ rpm -q openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-9.1.1-0.20181219093830.1fb8761.el7.noarch

Will attach templates/deploy cmd.

Tim Rozet (trozet) wrote :
Tim Rozet (trozet) wrote :

master version:
(undercloud) [stack@undercloud ~]$ rpm -q openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-10.1.1-0.20181203023850.b01b1a7.el7.noarch

Tim Rozet (trozet) wrote :
Harald Jensås (harald-jensas) wrote :

I think Ceph roles are broken since change: https://review.openstack.org/614457 landed.

None of these roles have the InternalApi network by default:

https://github.com/openstack/tripleo-heat-templates/blob/master/roles/CephAll.yaml
https://github.com/openstack/tripleo-heat-templates/blob/master/roles/CephFile.yaml
https://github.com/openstack/tripleo-heat-templates/blob/master/roles/CephObject.yaml
https://github.com/openstack/tripleo-heat-templates/blob/master/roles/CephStorage.yaml

We don't have per-role ServiceNetMap, so anyone deploying Ceph with this patch will end up with an empty string for the IP value in /etc/hosts.

Harald Jensås (harald-jensas) wrote :

Sorry, ignore my previous comment, CephStorage role is handled in service_net_map.j2[1] and the HostnameResolveNetwork is actually per-role. Sorry.

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/network/service_net_map.j2.yaml#L87

Related fix proposed to branch: master
Review: https://review.openstack.org/627419

Changed in tripleo:
assignee: nobody → Harald Jensås (harald-jensas)
status: New → In Progress
tags: added: queens-backport-potential

Related fix proposed to branch: master
Review: https://review.openstack.org/628265

Reviewed: https://review.openstack.org/627419
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=9590e8450706b1f634f9d53496e53f9b7add601b
Submitter: Zuul
Branch: master

commit 9590e8450706b1f634f9d53496e53f9b7add601b
Author: Harald Jensås <email address hidden>
Date: Mon Dec 24 15:17:08 2018 +0100

    Allow network data with no entries

    When deploying without network isolation we should
    allow an empty network_data.yaml with no networks.

    In combination with tripleo-heat-templates change:
    Ifeb2d2d1acb43c16a5bf29e95965776494d61fef the
    tripleoclient can be update to use /dev/null as
    network data, and networks can be removed from
    roles/Standalone.yaml and roles/Undercloud.yaml.

    Related-Bug: #1809313
    Change-Id: I2e8135bc9389d3bf1a6ef01e273515af5c488a9a

Reviewed: https://review.openstack.org/627963
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3df5f8db1ecb2e276d8744ad796a029a0f0a5a6e
Submitter: Zuul
Branch: master

commit 3df5f8db1ecb2e276d8744ad796a029a0f0a5a6e
Author: Harald Jensås <email address hidden>
Date: Wed Jan 2 13:11:46 2019 +0100

    Fall back service_net_map to ctlplane

    Render the service_net_map to fall back to the ctlplane network when
    networks are disabled (or not defined) in network_data.yaml.

    Also use startswith for Ceph roles which should all have 'storage'
    as the HostnameResolveNetwork.

    Closes-Bug: #1809313
    Change-Id: I737d5656b113f7e2238fe7bb555cc2d4cb13877c

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers