[overcloud deployment] Could not find data item oslo_messaging_rpc_password in any Hiera data file

Bug #1772152 reported by Jonas Libbrecht on 2018-05-19
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Medium
Unassigned

Bug Description

Description
===========

Overcloud fails to deploy during ceilometer deployment on the nova compute node.

Steps to reproduce
==================

* Followed documentation on: https://docs.openstack.org/tripleo-docs/latest/install/basic_deployment/basic_deployment_cli.html

* Enabled seperate ceph installation via given manual

* Enabled fake_pxe because baremetals do not have ilo/idrac/ipmi : http://blog.leifmadsen.com/blog/2016/11/11/tripleo-using-the-fake_pxe-driver-with-ironic/

* Configuration files:

undercloud.conf: https://pastebin.com/hMLcs9zP
instackenv.json: https://pastebin.com/JtYRF7xz

* Tagged the nodes with profiles: https://pastebin.com/tPYaf78A

* Deployed with command: https://pastebin.com/EZ9Wf5KZ

Expected result
===============

Stack creation COMPLETE

Actual result
=============

Overcloud deployment output:
```
 Stack overcloud CREATE_FAILED

START with options: ['stack', 'failures', 'list', 'overcloud']
command: stack failures list -> heatclient.osc.v1.stack_failures.ListStackFailures (auth=True)
Using auth plugin: password
overcloud.AllNodesDeploySteps.CephStorageDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 5b6cf953-75c7-40ba-843d-d781423282e3
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
None
  deploy_stderr: |
None
overcloud.AllNodesDeploySteps.ComputeDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: b9a0e54a-74e1-447a-a22f-5eb30b108951
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "Warning: Undefined variable 'deploy_config_name'; ",
            " (file & line not available)",
            "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/man ifests/profile/base/ceilometer.pp:78:30 on node overcloud-novacompute-0.localdomain"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/5e03d617-6b1b-4e44-8c9a-fcaab68cfa08_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=4 changed=1 unreachable=0 failed=1

    (truncated, view all with --long)
  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: a7ab9f67-459c-41d3-8304-91ef4dda85ff
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
None
  deploy_stderr: |
None
END return value: 0
Heat Stack create failed.
Heat Stack create failed.
END return value: 1
```

I noticed that this may be important:
```
May 19 08:13:32 localhost puppet-user[12879]: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
May 19 08:13:32 localhost puppet-user[12879]: Undefined variable 'deploy_config_name';
May 19 08:13:32 localhost puppet-user[12879]: (file & line not available)
May 19 08:13:32 localhost puppet-user[12879]: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
May 19 08:13:32 localhost puppet-user[12879]: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/
puppet/modules/tripleo/manifests/profile/base/ceilometer.pp:78:30 on node overcloud-novacompute-0.localdomain
```

Environment
===========

- OpenStack version: Pike
- Overcloud images: https://images.rdoproject.org/master/delorean/current-tripleo-rdo-internal/
- Director OS: CentOS 7
- Director Kernel: Linux director001.openstack.local 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
- External routable IP range: 192.168.102.0/24
- Storage usage: Ceph (followed steps as given in Tripleo documentation), no hyperconverged
- Network: as defined in documentation (NIC 1 = routable external network with DHCP, NIC 2 = Provisioning network, isolated in VLAN)

Kernel of failed node: Linux overcloud-novacompute-0 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Logs & Configs
==============

- /var/log/messages on nova compute: https://pastebin.com/RmtbLKMk
- Extended output of overcloud deployment: https://pastebin.com/FHHcP386
- undercloud.conf: https://pastebin.com/hMLcs9zP
- instackenv.json: https://pastebin.com/JtYRF7xz
- tagging.sh: https://pastebin.com/tPYaf78A
- deployment.sh: https://pastebin.com/EZ9Wf5KZ

Download full text (4.1 KiB)

Tried to disable the Telemetry but still giving the same error but this time during cinder deployment but on the compute and controller.

```
 Stack overcloud CREATE_FAILED

START with options: ['stack', 'failures', 'list', 'overcloud']
command: stack failures list -> heatclient.osc.v1.stack_failures.ListStackFailures (auth=True)
Using auth plugin: password
overcloud.AllNodesDeploySteps.CephStorageDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 6fa2f152-8669-4e30-b7e3-3fd340ed86f7
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
None
  deploy_stderr: |
None
overcloud.AllNodesDeploySteps.ComputeDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 0661624a-731f-4608-8d28-2146f213eb4f
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "Warning: Undefined variable 'deploy_config_name'; ",
            " (file & line not available)",
            "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/neutron.pp:90:30 on node overcloud-novacompute-0.localdomain"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/7644e62b-35fa-4eca-8060-535b4fcb0ac0_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=4 changed=1 unreachable=0 failed=1

    (truncated, view all with --long)
  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 862066c2-c08f-46cd-902b-e2ec70a6ff43
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "Warning: Undefined variable 'deploy_config_name'; ",
            " (file & line not available)",
            "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/cinder.pp:87:30 on node overcloud-controller-0.localdomain"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/4143f88a-ff90-4af3-99bc-c3055428878e_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=4 changed=1 unreachable=0 failed=1

    (truncated, view all with --long)
  deploy_stderr: |

END return value: 0
Heat Stack create failed.
Heat Stack create failed.
END return value: 1

```

Disabled Telemetry followed through documentation: https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/disable_telemetry.html

So there must be something wrong with the correct d...

Read more...

summary: - [ceilometer] Could not find data item oslo_messaging_rpc_password in any
- Hiera data file
+ [overcloud deployment] Could not find data item
+ oslo_messaging_rpc_password in any Hiera data file
Changed in tripleo:
status: New → Triaged
importance: Undecided → Medium
milestone: none → rocky-3
Harald Jensås (harald-jensas) wrote :
Download full text (4.1 KiB)

I get this as well. No Ceph involved in by environment.

I have attached the ansible log file.

My deploy command:

openstack overcloud deploy --templates /home/stack/tripleo-heat-templates \
  -n /home/stack/templates/network_data.yaml \
  -r /home/stack/templates/my_roles_data.yaml \
  -e /home/stack/environments/node_data.yaml \
  -e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/tripleo-heat-templates/environments/low-memory-usage.yaml \
  -e /home/stack/tripleo-heat-templates/environments/network-environment.yaml \
  -e /home/stack/environments/network-environment-overrides.yaml \
  -e /home/stack/environments/docker_registry.yaml

It is for all services.

        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/ceilometer.pp:78:30 on node overcloud-compute-leaf1-0.localdomain",
        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/nova.pp:85:30 on node overcloud-compute-leaf1-0.localdomain",
        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/neutron.pp:94:30 on node overcloud-compute-leaf1-0.localdomain",
        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/ceilometer.pp:78:30 on node overcloud-compute-leaf0-0.localdomain",
        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/nova.pp:85:30 on node overcloud-compute-leaf0-0.localdomain",
        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/neutron.pp:94:30 on node overcloud-compute-leaf0-0.localdomain",
        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/ceilometer.pp:78:30 on node overcloud-controller-0.localdomain",
        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/nova.pp:85:30 on node overcloud-controller-0.localdomain",
        "Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item oslo_messaging_rpc_password in any Hiera...

Read more...

Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers