Deployment with detached services failed: No such file or directory - /etc/rabbitmq/definitions

[Dennis Dmitriev]

Scenario:
1. Create cluster
2. Add 3 nodes with controller role
3. Add 3 nodes with rabbit role
4. Add 1 compute and cinder
5. Verify networks
6. Deploy the cluster

Result: deploy failed while primary controller was deploying with the message in puppet.log (see the logs in diagnostic snapshot):

node-1 2015-10-31T02:25:00.196679 err: Could not set 'file' on ensure: No such file or directory - /etc/rabbitmq/definitions at 27:/etc/puppet/modules/osnailyfacter/modular/astute/dump_rabbitmq_definitions.pp

Similar bug was fixed for 7.0: https://bugs.launchpad.net/fuel/+bug/1492312

ISO version:
 shotgun_sha 25dd78a3118267e3616df0727ce746e7dead2d67
 api 1.0
 fuelmain_sha b9b39ee9d3bbdb1883d924840ab691b95e1ef4d1
 astute_sha 687458e949dedfa6877e3a2e683f8ee0c1cbd43c
 fuel-ostf_sha 25c87380395f2e42ca01ea6357b72d4407ff4201
 build_id 97
 openstack_version 2015.1.0-8.0
 auth_required True
 fuel-library_sha 01272e0a719c940ef3df5b4167990ff811da1b10
 fuel-agent_sha 9da73b497be5f91cb79f91e74d73eb0525be1c71
 production docker
 network-checker_sha 518f7ef2706e564950469435f7cf27df175e1b27
 release_versions:
    2015.1.0-8.0:
       VERSION:
          build_id 97
          fuelmenu_sha 153c91e884a4601d30f8ebe3349170f96e3456e2
          openstack_version 2015.1.0-8.0
          network-checker_sha 518f7ef2706e564950469435f7cf27df175e1b27
          fuel-upgrade_sha 1e894e26d4e1423a9b0d66abd6a79505f4175ff6
          fuel-library_sha 01272e0a719c940ef3df5b4167990ff811da1b10
          fuel-createmirror_sha 5b61266e19eb7cdc745fea7a7d97c90f2515c921
          feature_groups:
              - mirantis
          fuel-nailgun-agent_sha 00b4b11553c250f22c0079fb74c8b782dcb7b740
          fuel-nailgun_sha af02bd8f9aac1a1007f3ce17e5d7842853bd7758
          shotgun_sha 25dd78a3118267e3616df0727ce746e7dead2d67
          fuel-agent_sha 9da73b497be5f91cb79f91e74d73eb0525be1c71
          api 1.0
          fuelmain_sha b9b39ee9d3bbdb1883d924840ab691b95e1ef4d1
          astute_sha 687458e949dedfa6877e3a2e683f8ee0c1cbd43c
          fuel-ostf_sha 25c87380395f2e42ca01ea6357b72d4407ff4201
          release 8.0
          build_number 97
          python-fuelclient_sha 555eeac8810c36b0616f5d12277e9099a53bb338
          production docker
 fuel-upgrade_sha 1e894e26d4e1423a9b0d66abd6a79505f4175ff6
 fuel-createmirror_sha 5b61266e19eb7cdc745fea7a7d97c90f2515c921
 build_number 97
 fuelmenu_sha 153c91e884a4601d30f8ebe3349170f96e3456e2
 fuel-nailgun-agent_sha 00b4b11553c250f22c0079fb74c8b782dcb7b740
 fuel-nailgun_sha af02bd8f9aac1a1007f3ce17e5d7842853bd7758
 python-fuelclient_sha 555eeac8810c36b0616f5d12277e9099a53bb338
 feature_groups:
     - mirantis
 release 8.0

[Michael Polenchuk (mpolenchuk)]

Looks like hiera didn't merge data from override/detach-rabbitmq.yaml
as it yaml contains "rabbit_hash: enabled: false" param that should disable run of astute/rabbitmq_definitions task.

[Michael Polenchuk (mpolenchuk)]

/etc/hiera.yaml was modified by modular/hiera task (removed override/detach-rabbitmq from hierarchy)

[Matthew Mosesohn (raytrac3r)]

This is caused by hiera task being re-run after openstack-network/agents/metadata.pp and before heat.pp tasks. hiera task rewrites /etc/hiera.yaml and removes any plugin configuration. Here's the task list:
 http://paste.openstack.org/show/Fc1eCobjHIufCtcbhpDF/

It should be only run once at the very beginning

[Matthew Mosesohn (raytrac3r)]

If I use task_graph.py against current master, it only runs hiera in the beginning (pre deploy and deploy tasks). I'm still trying to find a recent commit to blame which introduces this.

[Matthew Mosesohn (raytrac3r)]

https://review.openstack.org/#/c/232141/34/deployment/puppet/osnailyfacter/modular/openstack-network/tasks.yaml line 128. If we want a noop task, we should run /bin/true, not hiera.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/240897

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240897
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3a87704f78ccc83c9199aabe83c2e5129455345c
Submitter: Jenkins
Branch: master

commit 3a87704f78ccc83c9199aabe83c2e5129455345c
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 2 18:02:11 2015 +0300

    Change openstack-network anchor task to type skipped

    This task was running hiera task after neutron tasks
    were completed, causing hiera metadata to modify
    mid-deployment. This was breaking deployments with
    plugins, which write to /etc/hiera.yaml. Hiera task
    overwrites this file, causing a discrepancy during
    deployment and breaking rabbitmq dump definitions
    task.

    Change-Id: Ie5a0715e468888cb8359b746f220099c04f01af2
    Closes-Bug: #1512017

[Anastasia Palkina (apalkina)]

Verified on ISO #171

"release_versions": {"2015.1.0-8.0": {"VERSION": {"fuel-upgrade_sha": "1e894e26d4e1423a9b0d66abd6a79505f4175ff6", "fuel-createmirror_sha": "22a7aacd95bbdca69f9e0f08b70facabdec8fb28", "shotgun_sha": "c377d163519f6d10b69a654019d6086ba5f14edc", "api": "1.0", "fuelmain_sha": "266f9b374934c64629a84b5146632cc0de70ac91", "astute_sha": "959b06c5ef8143125efd1727d350c050a922eb12", "fuel-ostf_sha": "9690a2de829d3b063ed1e64b0b10dde39f711dc0", "openstack_version": "2015.1.0-8.0", "build_id": "171", "fuelmenu_sha": "06bbcebed6c8d0b0f9279e2997d2f958c800e98c", "build_number": "171", "fuel-library_sha": "31aa9fb129a8b8739f8ca6ea186caa3c9f46820f", "fuel-nailgun-agent_sha": "3e9d17211d65c80bf97c8d83979979f6c7feb687", "fuel-nailgun_sha": "37cfc3206d9c0b510112ea7a7f461e40f8c08d6a", "fuel-agent_sha": "61f741f4d89a3f1095dd8b494f82695fb69153d6", "production": "docker", "network-checker_sha": "a57e1d69acb5e765eb22cab0251c589cd76f51da", "feature_groups": ["mirantis"], "release": "8.0", "python-fuelclient_sha": "e685d68c1c0d0fa0491a250f07d9c3a8d0f9608c"}}}, "fuel-upgrade_sha": "1e894e26d4e1423a9b0d66abd6a79505f4175ff6", "fuel-createmirror_sha": "22a7aacd95bbdca69f9e0f08b70facabdec8fb28", "shotgun_sha": "c377d163519f6d10b69a654019d6086ba5f14edc", "api": "1.0", "fuelmain_sha": "266f9b374934c64629a84b5146632cc0de70ac91", "astute_sha": "959b06c5ef8143125efd1727d350c050a922eb12", "fuel-ostf_sha": "9690a2de829d3b063ed1e64b0b10dde39f711dc0", "openstack_version": "2015.1.0-8.0", "build_id": "171", "fuelmenu_sha": "06bbcebed6c8d0b0f9279e2997d2f958c800e98c", "build_number": "171", "auth_required": true, "fuel-library_sha": "31aa9fb129a8b8739f8ca6ea186caa3c9f46820f", "fuel-nailgun-agent_sha": "3e9d17211d65c80bf97c8d83979979f6c7feb687", "fuel-nailgun_sha": "37cfc3206d9c0b510112ea7a7f461e40f8c08d6a", "fuel-agent_sha": "61f741f4d89a3f1095dd8b494f82695fb69153d6", "production": "docker", "network-checker_sha": "a57e1d69acb5e765eb22cab0251c589cd76f51da", "feature_groups": ["mirantis"], "release": "8.0", "python-fuelclient_sha": "e685d68c1c0d0fa0491a250f07d9c3a8d0f9608c"