Deployment with Murano and separate db, keystone, rabbit services failed with Could not set 'file' on ensure: No such file or directory - /etc/rabbitmq/rabbitmq-murano.config20150902-19114-gbsj3a.lock at 56:/etc/puppet/modules/murano/manifests/rabbitmq.pp

[Andrey Sledzinskiy]

Deployment with Murano and separate db, keystone, rabbit services failed with Could not set 'file' on ensure: No such file or directory - /etc/rabbitmq/rabbitmq-murano.config20150902-19114-gbsj3a.lock at 56:/etc/puppet/modules/murano/manifests/rabbitmq.pp

Steps:
1. Upload and install separate-database, rabbitmq, keystone plugins on master (https://github.com/stackforge/fuel-plugin-detach-rabbitmq, https://github.com/stackforge/fuel-plugin-detach-database, https://github.com/stackforge/fuel-plugin-detach-keystone)
2. Create next cluster - Ubuntu, HA, neutron Vlan, Murano, enable rabbitmq, database, keystone plugins on settings tab, 3 controller nodes, 3 database+keystone+rabbitmq nodes, 1 compute, 1 cinder node
3. Start deployment

Actual result - deployment failed on controller with:
2015-09-02 10:20:44 ERR Could not set 'file' on ensure: No such file or directory - /etc/rabbitmq/rabbitmq-murano.config20150902-19114-gbsj3a.lock at 56:/etc/puppet/modules/murano/manifests/rabbitmq.pp

{

    "build_id": "263",
    "build_number": "263",
    "release_versions":

{

    "2015.1.0-7.0":

{

    "VERSION":

{

    "build_id": "263",
    "build_number": "263",
    "api": "1.0",
    "fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0",
    "nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468",
    "feature_groups":

            [
                "mirantis"
            ],
            "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
            "openstack_version": "2015.1.0-7.0",
            "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
            "production": "docker",
            "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512",
            "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25",
            "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4",
            "release": "7.0",
            "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "1556601b9b7503285714d7d1e02cc0807b1c68f0",
"nailgun_sha": "b564ae20116297750bf6402b3a017e219bf4b468",
"feature_groups":

    [
        "mirantis"
    ],
    "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
    "openstack_version": "2015.1.0-7.0",
    "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
    "production": "docker",
    "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512",
    "astute_sha": "e63709d16bd4c1949bef820ac336c9393c040d25",
    "fuel-ostf_sha": "582a81ccaa1e439a3aec4b8b8f6994735de840f4",
    "release": "7.0",
    "fuelmain_sha": "4dc6799370da4cddf06c04e4ecb7646102298535"

}

[Alexey Shtokolov (ashtokolov)]

It can happen cause Murano uses rabbit on controllers. And if you detach rabbit from controllers Murano couldn't find it.

[Nikita Koshikov (nkoshikov)]

Hi

In order to deploy murano - you need to install rabbitmq-server. We have separete task for this:
notice('MODULAR: murano-rabbitmq.pp')

$murano_hash = hiera_hash('murano', {})

if $murano_hash['enabled'] {
  include '::rabbitmq::params'

  package { 'rabbitmq-server':
    ensure => $package_ensure,
    name => $package_name,
    provider => $package_provider,
  }

  if $package_source {
    Package['rabbitmq-server'] {
      source => $package_source,
    }
  }
}

[Andrey Maximov (maximov)]

This is use case when we run detached components. Murano fails to deploy only if other components are spread across nodes. So let's mention this in release notes, that Murano works in 7.0 only with standard reference architecture (all-in-one controller)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/220078

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220078
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=aa6ecf7aadbfe2d2b9af85e5f2437f46bc565502
Submitter: Jenkins
Branch: master

commit aa6ecf7aadbfe2d2b9af85e5f2437f46bc565502
Author: Michael Polenchuk <email address hidden>
Date: Thu Sep 3 13:24:47 2015 +0300

    Make sure rabbitmq package is installed

    Ensure that rabbitmq package is installed before murano
    service configuration.

    DocImpact: detached RabbitMQ + Murano resulting in rabbitmq being
    installed on the controller anyway.

    Change-Id: Ia8852a625566ab1742e584d742e9820329328e0c
    Closes-Bug: #1491334

[Vladimir Khlyunev (vkhlyunev)]

Deployment was successful on ISO #286. Waiting for documentation changes from https://review.openstack.org/220078

[Anastasia Kuznetsova (akuznetsova)]

Verified on ISO 286

> Main Rabbit works on separated nodes:

root@node-3:~# rabbitmqctl cluster_status
Cluster status of node 'rabbit@node-3' ...
[{nodes,[{disc,['rabbit@node-3','rabbit@node-4','rabbit@node-5']}]},
 {running_nodes,['rabbit@node-5','rabbit@node-4','rabbit@node-3']},
 {cluster_name,<<"<email address hidden>">>},
 {partitions,[]}]
root@node-3:~# ps -aux | grep rabbit
root 9097 0.0 0.0 61676 1728 ? S Sep07 0:00 su rabbitmq -s /bin/sh -c /usr/sbin/rabbitmq-server >> "/var/log/rabbitmq/startup_log" 2>/dev/null
rabbitmq 9103 0.0 0.0 4440 656 ? Ss Sep07 0:00 sh -c /usr/sbin/rabbitmq-server >> "/var/log/rabbitmq/startup_log" 2>/dev/null
rabbitmq 9105 0.0 0.0 4440 656 ? S Sep07 0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 9111 2.4 2.1 616944 65344 ? Sl Sep07 22:59 /usr/lib/erlang/erts-5.10.4/bin/beam -W w -A 64 -K true -A30 -P 1048576 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.4/sbin/../ebin -noshell -noinput -sname rabbit@node-3 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -rabbit tcp_listeners [{"10.109.19.6",5673}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@node-3-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@node-3"
rabbitmq 9380 0.0 0.0 7456 428 ? Ss Sep07 0:00 inet_gethost 4
rabbitmq 9381 0.0 0.0 9552 676 ? S Sep07 0:00 inet_gethost 4
rabbitmq 13146 0.0 0.3 90304 12296 ? Ss Sep07 0:04 /usr/bin/python /usr/bin/rabbit-fence.py
root 16927 0.0 0.0 10464 940 pts/1 S+ 09:32 0:00 grep --color=auto rabbit
root@node-3:~# exit
logout

Connection to node-3 closed.

Murano RabbitMQ works on controller:

[root@nailgun ~]# ssh node-1
Warning: Permanently added 'node-1' (RSA) to the list of known hosts.
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-63-generic x86_64)

 * Documentation: https://help.ubuntu.com/
root@node-1:~# ps -aux | grep rabbit
root 19178 0.0 0.0 10460 936 pts/22 S+ 09:33 0:00 grep --color=auto rabbit
rabbitmq 25965 0.0 0.0 7492 368 ? S Sep07 0:01 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq 26087 0.0 0.0 4440 460 ? S Sep07 0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 26092 0.1 0.9 603136 29352 ? Sl Sep07 1:40 /usr/lib/erlang/erts-5.10.4/bin/beam -W w -A 64 -P 1048576 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname murano@localhost -boot start_sasl -config /etc/rabbitmq/rabbitmq-murano -kernel...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/223456

[Dmitry Mescheryakov (dmitrymex)]

Marking 8.0 as invalid as fix was merged into master in 7.0 scope.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Evgeny Konstantinov (<email address hidden>) on branch: master
Review: https://review.openstack.org/223456
Reason: Invalid