Fuel master node deployment failed with ERROR: rabbitmq failed to start.

[Dmitry Kalashnik]

Steps:
1. Start fuel installation

Actual result - installation failed:
http://paste.openstack.org/show/481956/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "292"
  build_id: "292"
  fuel-nailgun_sha: "a0ca2ca443abfe1da7a345546383b255c0a45ba0"
  python-fuelclient_sha: "1d61ed2dbf2183f6c0f9dcd00a4a9ca23aeafaa5"
  fuel-agent_sha: "06bd5bc2280fa2a305852ab7d5aa30a5eb452431"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "1c3a981dff739f205dc94bdb900ba21970a02992"
  fuel-library_sha: "8f2f65f2ed17309d15575501b3d0d7edc41cfe60"
  fuel-ostf_sha: "9910a4726cbd038c257582b429527e40c4c3cb20"
  fuel-mirror_sha: "31b9df814960ec69b644ca9b689dacec0c7e10a1"
  fuelmenu_sha: "680b720291ff577f4c058cee25f85e563c96312e"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "7702258023b34e67e94594aef0d679a67fd844c5"

[Artem Panchenko (apanchenko-8)]

As you can see docker container for RabbitMQ is running, but service is down:

[root@nailgun ~]# docker ps | grep rabbitmq
a7c397add136 fuel/rabbitmq_8.0 "/usr/sbin/init" About an hour ago Up About an hour fuel-core-8.0-rabbitmq

[root@nailgun ~]# dockerctl shell rabbitmq ps auxfw
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 3160 0.0 0.0 35884 1448 ? Rs+ 09:34 0:00 ps auxfw
root 1 0.0 0.0 40956 712 ? Ss 08:26 0:00 /usr/sbin/init
dbus 43 0.0 0.0 26592 284 ? Ss 08:26 0:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root 83 0.0 0.0 211560 544 ? Ssl 08:26 0:00 /usr/sbin/rsyslogd -n
root 88 0.0 0.0 26156 620 ? Ss 08:26 0:00 /usr/sbin/crond -n
root 101 0.0 0.0 37496 256 ? Ss 08:26 0:00 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid
root 3105 0.0 0.0 25252 244 ? Ss 09:01 0:00 /usr/sbin/anacron -s

[root@nailgun ~]# dockerctl shell rabbitmq systemctl status rabbitmq-server.service
* rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/etc/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2015-12-15 08:27:11 UTC; 5h 41min ago
  Process: 2896 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
  Process: 677 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
 Main PID: 677 (code=killed, signal=TERM)

That's what I've found in RabbitMQ logs:

http://paste.openstack.org/show/481950/

and in Docker logs (container stdout/stderr):

http://paste.openstack.org/show/481960/

According to content of '/var/tmp/setup.log' file, the service wasn't started because of `pkill -u rabbitmq` returned non-zero value (no processes found):

+ rm -f '/var/lib/rpm/__db.*'
+ rpm --rebuilddb
+ mkdir -p /var/log/rabbitmq
+ chown -R rabbitmq:rabbitmq /var/log/rabbitmq
+ systemctl daemon-reload
+ puppet apply --debug --verbose --color false --detailed-exitcodes --logdest /var/log/puppet/rabbitmq.log /etc/puppet/modules/nailgun/examples/rabbitmq-only.pp
Unable to add resolve nil for fact osd_devices_list: unexpected return
Could not retrieve fact='ssh_client_version_full', resolution='<anonymous>': undefined method `lines' for nil:NilClass
Could not retrieve fact='ssh_client_version_full', resolution='<anonymous>': undefined method `lines' for nil:NilClass
Could not retrieve fact='ssh_client_version_major', resolution='<anonymous>': undefined method `gsub' for nil:NilClass
Could not retrieve fact='ssh_client_version_full', resolution='<anonymous>': undefined method `lines' for nil:NilClass
Could not retrieve fact='ssh_client_version_release', resolution='<anonymous>': undefined method `gsub' for nil:NilClass
Could not retrieve fact='ssh_server_version_full', resolution='<anonymous>': undefined method `lines' for nil:NilClass
Could not retrieve fact='ssh_server_version_full', resolution='<anonymous>': undefined method `lines' for nil:NilClass
Could not retrieve fact='ssh_server_version_major', resol...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/257937

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257937
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=a93fccc53f0e3700d8ff1c9535cc0ea2c5620024
Submitter: Jenkins
Branch: master

commit a93fccc53f0e3700d8ff1c9535cc0ea2c5620024
Author: Artem Panchenko <email address hidden>
Date: Tue Dec 15 17:01:26 2015 +0200

    Fix RabbitMQ container startup script

    Don't stop-kill-start RabbitMQ server after manifests run,
    because it's automatically restarted by Puppet if necessary.

    Also there is no need to start services directly from
    'start.sh' in foreground, because docker doesn't watch it
    anymore.

    Change-Id: I6e5f47a9fc92de66119fc918fd1e22eb324e2a7d
    Closes-bug: #1526351

[Sergey Shevorakov (sshevorakov)]

Changing tag from swarm-fail-driver to swarm-blocker since this bug fails 8 test cases (2.5% of all).

[ElenaRossokhina (esolomina)]

verified on ISO#427
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "427"
  build_id: "427"
  fuel-nailgun_sha: "9ebbaa0473effafa5adee40270da96acf9c7d58a"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "fae42170a54b98d8e8c8db99b0fbb312633c693c"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"