deployment fails: Rabbitmq_user[guest] node 'rabbit' not running at all

[Frank J. Cameron]

I've had deployment of an HA environment fail three attempts in a row. It appears that 'rabbitmqctl list-users' is running before the service has initialized so the command fails; the last time it happened I sshed to the node as soon as the error was reported and ran the command manually with no trouble. Before the fourth attempt, I hacked a "rescue retry" into /etc/puppet/modules/rabbitmq/lib/puppet/provider/rabbitmq_user/rabbitmqctl.rb; I neglected to add logging so I don't know if it worked because of the hack or because the timing just worked out, but the deployment was successful. I won't be able to do any more test deployments though.

  def self.instances
    rabbitmqctl('list_users').split(/\n/)[1..-2].collect do |line|
      if line =~ /^(\S+)(\s+\S+|)$/
        new(:name => $1)
      else
        raise Puppet::Error, "Cannot parse invalid user line: #{line}"
      end
    end
  rescue
    retry
  end

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1"
  api: "1.0"
  build_number: "11"
  build_id: "2014-09-17_21-40-34"
  astute_sha: "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13"
  fuellib_sha: "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"
  ostf_sha: "64cb59c681658a7a55cc2c09d079072a41beb346"
  nailgun_sha: "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d"
  fuelmain_sha: "8ef433e939425eabd1034c0b70e90bdf888b69fd"

Failure log:

2014-10-03T22:13:53.667080+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker)) LAST SUCCESSFUL OP :
2014-10-03T22:13:53.667080+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker))
2014-10-03T22:13:53.667161+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker)) "start"
2014-10-03T22:13:53.667161+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker)) LSO: {"node-1.cde.ctc.com"=>"start"}
2014-10-03T22:13:53.667161+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker)) STATUS IS: running
2014-10-03T22:14:00.103519+01:00 debug: Executing '/usr/sbin/crm resource manage master_p_rabbitmq-server'
2014-10-03T22:14:00.369498+01:00 notice: (/Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]/enable) enable changed 'true' to 'true'
2014-10-03T22:14:00.378113+01:00 debug: (/Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]) The container Class[Rabbitmq::Service] will propagate my refresh event
2014-10-03T22:14:00.379173+01:00 info: (/Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]) Evaluated in 13.62 seconds
2014-10-03T22:14:00.380314+01:00 info: (Class[Rabbitmq::Service]) Starting to evaluate the resource
2014-10-03T22:14:00.388625+01:00 debug: (Class[Rabbitmq::Service]) The container Stage[main] will propagate my refresh event
2014-10-03T22:14:00.389772+01:00 info: (Class[Rabbitmq::Service]) Evaluated in 0.01 seconds
2014-10-03T22:14:00.390886+01:00 info: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) Starting to evaluate the resource
2014-10-03T22:14:00.393611+01:00 debug: Executing '/usr/sbin/rabbitmqctl list_users'
2014-10-03T22:14:00.617753+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) Could not evaluate: Execution of '/usr/sbin/rabbitmqctl list_users' returned 2: Listing users ...
2014-10-03T22:14:00.618017+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) Error: unable to connect to node 'rabbit@node-1': nodedown
2014-10-03T22:14:00.618017+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest])
2014-10-03T22:14:00.618017+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) DIAGNOSTICS
2014-10-03T22:14:00.618017+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) ===========
2014-10-03T22:14:00.619011+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest])
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) attempted to contact: ['rabbit@node-1']
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest])
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) rabbit@node-1:
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) * connected to epmd (port 4369) on node-1
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) * epmd reports: node 'rabbit' not running at all
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) other nodes on node-1: [murano]
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) * suggestion: start the node
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest])
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) current node details:
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) - node name: 'rabbitmqctl11897@node-1'
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) - home dir: /var/lib/rabbitmq
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) - cookie hash: soeIWU2jk2YNseTyDSlsEA==
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest])
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest])
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util/execution.rb:188:in `execute'
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/provider/command.rb:23:in `execute'
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:237:in `rabbitmqctl'
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:444:in `send'
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/provider.rb:444:in `rabbitmqctl'
2014-10-03T22:14:00.619141+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /etc/puppet/modules/rabbitmq/lib/puppet/provider/rabbitmq_user/rabbitmqctl.rb:30:in `exists?'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/property/ensure.rb:81:in `retrieve'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/type.rb:1035:in `retrieve'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/type.rb:1063:in `retrieve_resource'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:223:in `from_resource'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:17:in `evaluate'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
2014-10-03T22:14:00.619249+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2014-10-03T22:14:00.619304+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
2014-10-03T22:14:00.619304+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
2014-10-03T22:14:00.619304+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
2014-10-03T22:14:00.619304+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
2014-10-03T22:14:00.619304+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
2014-10-03T22:14:00.619304+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
2014-10-03T22:14:00.619304+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-10-03T22:14:00.619304+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
2014-10-03T22:14:00.619584+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-10-03T22:14:00.619584+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
2014-10-03T22:14:00.619584+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2014-10-03T22:14:00.619584+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
2014-10-03T22:14:00.619584+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
2014-10-03T22:14:00.620421+01:00 err: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) /usr/bin/puppet:4
2014-10-03T22:14:00.620569+01:00 info: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) Evaluated in 0.23 seconds
2014-10-03T22:14:00.621531+01:00 info: (Class[Rabbitmq::Server]) Starting to evaluate the resource
2014-10-03T22:14:00.624394+01:00 debug: (Class[Rabbitmq::Server]) The container Stage[main] will propagate my refresh event
2014-10-03T22:14:00.625514+01:00 info: (Class[Rabbitmq::Server]) Evaluated in 0.00 seconds
2014-10-03T22:14:00.631480+01:00 info: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]) Starting to evaluate the resource
2014-10-03T22:14:00.632915+01:00 notice: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]) Dependency Rabbitmq_user[guest] has failures: true
2014-10-03T22:14:00.634836+01:00 warning: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]) Skipping because of failed dependencies
2014-10-03T22:14:00.635985+01:00 info: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]) Evaluated in 0.00 seconds

Success log:

/usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:1082014-10-03T23:05:40.041782+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker)) LAST SUCCESSFUL OP :
2014-10-03T23:05:40.041883+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker))
2014-10-03T23:05:40.041883+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker)) "start"
2014-10-03T23:05:40.041883+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker)) LSO: {"node-1.cde.ctc.com"=>"start"}
2014-10-03T23:05:40.041883+01:00 debug: (Service[p_rabbitmq-server](provider=pacemaker)) STATUS IS: running
2014-10-03T23:05:48.965064+01:00 debug: Executing '/usr/sbin/crm resource manage master_p_rabbitmq-server'
2014-10-03T23:05:49.225535+01:00 notice: (/Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]/ensure) ensure changed 'stopped' to 'running'
2014-10-03T23:05:49.226619+01:00 debug: (/Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]) The container Class[Rabbitmq::Service] will propagate my refresh event
2014-10-03T23:05:49.227711+01:00 info: (/Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]) Unscheduling refresh on Service[p_rabbitmq-server]
2014-10-03T23:05:49.228819+01:00 info: (/Stage[main]/Rabbitmq::Service/Service[p_rabbitmq-server]) Evaluated in 41.22 seconds
2014-10-03T23:05:49.229938+01:00 info: (Class[Rabbitmq::Service]) Starting to evaluate the resource
2014-10-03T23:05:49.238204+01:00 debug: (Class[Rabbitmq::Service]) The container Stage[main] will propagate my refresh event
2014-10-03T23:05:49.239321+01:00 info: (Class[Rabbitmq::Service]) Evaluated in 0.01 seconds
2014-10-03T23:05:49.240443+01:00 info: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) Starting to evaluate the resource
2014-10-03T23:05:49.243512+01:00 debug: Executing '/usr/sbin/rabbitmqctl list_users'
2014-10-03T23:05:49.478386+01:00 debug: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) Nothing to manage: no ensure and the resource doesn't exist
2014-10-03T23:05:49.479483+01:00 info: (/Stage[main]/Rabbitmq::Server/Rabbitmq_user[guest]) Evaluated in 0.24 seconds
2014-10-03T23:05:49.480594+01:00 info: (Class[Rabbitmq::Server]) Starting to evaluate the resource
2014-10-03T23:05:49.489413+01:00 debug: (Class[Rabbitmq::Server]) The container Stage[main] will propagate my refresh event
2014-10-03T23:05:49.490495+01:00 info: (Class[Rabbitmq::Server]) Evaluated in 0.01 seconds
2014-10-03T23:05:49.497557+01:00 info: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]) Starting to evaluate the resource
2014-10-03T23:05:49.509255+01:00 notice: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]/ensure) created
2014-10-03T23:05:49.510331+01:00 debug: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]) The container Class[Neutron::Quota] will propagate my refresh event
2014-10-03T23:05:49.511408+01:00 info: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]) Scheduling refresh of Service[neutron-server]
2014-10-03T23:05:49.511655+01:00 info: (/Stage[main]/Neutron::Quota/Neutron_config[quotas/quota_router]) Evaluated in 0.01 seconds

[Vladimir Kuklin (vkuklin)]

Frank, thank you for posting the bug report. We have not yet experienced such an issue, so we need a more verbose logging output to see whether there were any errors related to pacemaker/corosync, puppet traces and rabbitmq logs. Is it possible for you to upload a diagnostic snapshot of your deployment, so that we could analyze it and try to reproduce your environment?

[Frank J. Cameron (fjc)]

I have the diagnostic snapshot saved and will upload it when I can.

[Frank J. Cameron (fjc)]

The snapshot was huge, so I removed some of the bootstrap logs: https://s3.amazonaws.com/scuttle/fuel-snapshot-2014-10-06_13-53-47.tgz

[Tatyanka (tatyana-leontovich)]

reproduced on CI
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "31"
  build_id: "2014-10-20_00-01-06"
  astute_sha: "6a11a7c481d116e6cfdb422fab1d4bbb29cbea1c"
  fuellib_sha: "598f2b7212f686559630752a3d2e68569541172b"
  ostf_sha: "de177931b53fbe9655502b73d03910b8118e25f1"
  nailgun_sha: "33654affeeaca073397d0b1d0d71924fd0a98511"
  fuelmain_sha: "598122bd61175a96a71a1ab76818d5d67529cdb7"

[Vladimir Kuklin (vkuklin)]

Frank, thank you for reporting. We have finally experienced this issue. Fix will be ready shortly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130281

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/130282

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130281
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7a6ca611b53988902285d9f5d52bc6dd1b933713
Submitter: Jenkins
Branch: master

commit 7a6ca611b53988902285d9f5d52bc6dd1b933713
Author: Vladimir Kuklin <email address hidden>
Date: Wed Oct 22 20:55:27 2014 +0400

    Wait for rabbitmq to respond to commands

    Sometimes rabbitmq service starts but rabbitmq
    server is not ready to accept requests. This
    commit adds wait loop for rabbitmq and then
    allows rabbitmq_* resources to work with
    rabbitmq server.

    Change-Id: Ic01773a4f84bbc1fa1a4139a0a6eb366191a8b9b
    Closes-bug: #1377491

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130282
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8899fb5a5cbaead0e69dee7b8317514bd5f9fbf3
Submitter: Jenkins
Branch: stable/5.1

commit 8899fb5a5cbaead0e69dee7b8317514bd5f9fbf3
Author: Vladimir Kuklin <email address hidden>
Date: Wed Oct 22 20:55:27 2014 +0400

    Wait for rabbitmq to respond to commands

    Sometimes rabbitmq service starts but rabbitmq
    server is not ready to accept requests. This
    commit adds wait loop for rabbitmq and then
    allows rabbitmq_* resources to work with
    rabbitmq server.

    Change-Id: Ic01773a4f84bbc1fa1a4139a0a6eb366191a8b9b
    Closes-bug: #1377491

[Bogdan Dobrelya (bogdando)]

Related bug in puppet-rabbitmq upstream https://tickets.puppetlabs.com/browse/MODULES-1452

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127166
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=507d54e4c9e1eaf05d875d183c581661eb872f3a
Submitter: Jenkins
Branch: master

commit 507d54e4c9e1eaf05d875d183c581661eb872f3a
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Oct 8 18:00:42 2014 +0200

    Adapt rabbitmq module for Fuel

    * Sync to v5.0.0 (supports Ubuntu 14.04, major SSL fixes)
    * Fix deprecated in 5.0.0 params
    * Reimplement waiting/retries in rabbitmq provider
      (https://tickets.puppetlabs.com/browse/MODULES-1452)
    * Configure clustering with an erlang cookie hardcode. Also
      pass the cookie to the OCF script.
    * Make OCF script always update an erlang cookie, if specified
      (default false)
    * Add dependency module: nanliu-staging
    * Add support for rabbimtq class false value for nova module
      (https://bugs.launchpad.net/puppet-nova/+bug/1407077)
    * Do not use nova::rabbitmq to install and configure rabbitmq
      service anymore, install it separately.
      (TODO move it as a granular deploy step)
    * Adjust other existing rabbit class calls for new 5.0.0 rabbit module
    * Remove deprecated usage of rabbitmq::server class and replace it
      to ::rabbitmq (cinder, murano, openstack, zabbix, mcollective,
      nailgun)
    * Remove rabbit.pub.key and do not manage repos for rabbit package
      as we have it in nailgun repos and Fuel uses these repos for offline
      installation of rabbitmq.
      (Related upstream bug https://tickets.puppetlabs.com/browse/MODULES-1631)
    * Move custom rabbit OCF file from nova to openstack and remove
      OS-aware services disable from rabbit's manifests
    * Make pacemaker provider HA wrapper for rabbitmq service and move
      it to pacemaker_wrappers::rabbitmq
    * Fix linting
    * Fix rabbitmq service name for zabbix mon
    * Fix env_config for rabbitmq::nailgun, unify RABBIT_* env args.
    * Use key_content, package_gpg_key, config_*, cluster_*,
      environment_* params to ensure the same rabbit configuration as
      before and make offline installation possible.
    * Add cluster_partition_handling param for openstack::controller
    * Del unused rabbitmq_node_ip_address and rename as node_ip_address
    * Del unneeded rabbit_cluster due to it is being evaluated via
      passed rabbitmq_cluster_nodes as well
    * Adjust debug log_levels for rabbit config as well
    * For '$production !~ /docker/' case, remove custom init scripts
      for rabbit as non relevant anymore.
    * Enable rspec tests for rabbitmq in utils
    * Disable rspecs for erlang module.

    Related blueprint merge-openstack-puppet-modules
    Related-bug: #1377491

    Change-Id: I6e631b975c5835d2d6f1984d5a070a45231fe9ab
    Signed-off-by: Bogdan Dobrelya <email address hidden>