multistack DCN deployment fails on compute; hiera memcached_node_ips is nil

[John Fulton]

The compute node deployment step from the DCN doc [1] fails [2] because `hiera memcached_node_ips` returns nil on my compute node.

The compute deployment was done using -e all-nodes-extra-map-data.json, where said file has the desired hiera data and said file was produced using the process from the doc:

cat /var/lib/mistral/control-plane/group_vars/overcloud.json \
  | jq '{"parameter_defaults": {"AllNodesExtraMapData": .}}' \
  > all-nodes-extra-map-data.json

[1] https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/distributed_compute_node.html#export-dcn

[2]
        "2019-08-08 18:52:33,078 ERROR: 41334 -- ['/usr/bin/podman', 'run', '--user', 'root',
'--name', u'container-puppet-nova_libvirt', '--env', u'PUPPET_TAGS=file,file_line,concat,augea
s,cron,nova_config,nova_paste_api_ini,libvirtd_config,nova_config,file,libvirt_tls_password',
'--env', u'NAME=nova_libvirt', '--env', u'HOSTNAME=edge0-distributedcomputehci-0', '--env', 'N
O_ARCHIVE=', '--env', 'STEP=6', '--env', 'NET_HOST=true', '--env', 'DEBUG=False', '--volume',
'/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmpTld0Fk:/etc/config.pp:ro', '--volume'
, '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-tru
st/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.c
rt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust
.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/
config-data:/var/lib/config-data/:rw', '--volume', '/var/lib/container-puppet/puppetlabs/facte
r.conf:/etc/puppetlabs/facter/facter.conf:ro', '--volume', '/var/lib/container-puppet/puppetla
bs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--log-driver', 'k8s-file', '--l
og-opt', u'path=/var/log/containers/stdouts/container-puppet-nova_libvirt.log', '--security-op
t', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-pu
ppet/modules/:ro', '--entrypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', '
host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/containe
r-puppet.sh:/var/lib/container-puppet/container-puppet.sh:ro', u'192.168.24.1:8787/tripleomast
er/centos-binary-nova-compute:current-tripleo'] run failed after + mkdir -p /etc/puppet",
        "+ '[' -n file,file_line,concat,augeas,cron,nova_config,nova_paste_api_ini,libvirtd_co
nfig,nova_config,file,libvirt_tls_password ']'",
        "+ TAGS='--tags file,file_line,concat,augeas,cron,nova_config,nova_paste_api_ini,libvi
rtd_config,nova_config,file,libvirt_tls_password'",
        "+ origin_of_time=/var/lib/config-data/nova_libvirt.origin_of_time",
        "+ touch /var/lib/config-data/nova_libvirt.origin_of_time",
        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/
etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cro
n,nova_config,nova_paste_api_ini,libvirtd_config,nova_config,file,libvirt_tls_password /etc/co
nfig.pp",
        "puppet-user: Error: Function lookup() did not find a value for the name 'memcached_node_ips'",

[John Fulton (jfulton-org)]

Input from undercloud:

[stack@undercloud edge-common]$ cat all-nodes-extra-map-data.json | curl -F 'f:1=<-' ix.io
http://ix.io/1R4L

What I found on compute node:

[root@edge0-distributedcomputehci-0 hieradata]# cat all_nodes.json | curl -F 'f:1=<-' ix.io
http://ix.io/1R4M

[John Fulton (jfulton-org)]

This bug might have been fixed by https://review.opendev.org/#/c/672593

I am testing if that's the case

[John Fulton (jfulton-org)]

I confirm that https://review.opendev.org/#/c/672593 fixes this bug

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/678247

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/678247
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=7137cae523c57638eefb5a1f7b5ea73ecd38c573
Submitter: Zuul
Branch: master

commit 7137cae523c57638eefb5a1f7b5ea73ecd38c573
Author: James Slagle <email address hidden>
Date: Fri Aug 23 10:43:58 2019 -0400

    Restore AllNodesExtraMapData

    This commit is the same as I890bbef764b6bae7cb71cfedaff2ce17829737fb, which was
    previously applied to tripleo-common, but lost in the migration to
    tripleo-ansible.

    Commit a7661065743086961b8ef93056b810e7d2a49eda removed the
    all-nodes-config.j2.yaml Heat template. This template had the support
    for specifying hieradata to merge into the all_nodes hieradata via the
    AllNodesExtraMapData.

    This commit restores the merge logic so that the merged hieradata from
    the parameter value and the calculated hieradata is rendered in
    all_nodes.json.

    Change-Id: I49948f8d74c3846321af4f423a4891a71d0fabdc
    Closes-Bug: #1839546

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-ansible 0.3.0 release.