Error during start rabbitmq-server

Bug #1308483 reported by Ivan Berezovskiy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Ilya Pekelny

Bug Description

{"build_id": "2014-04-12_22-22-54", "mirantis": "yes", "build_number": "354", "nailgun_sha": "61410bcf3201cd737e68ece8ab15313acc746476", "production": "dev", "ostf_sha": "118c955085ea7829f3a34decd38d63554b74451c", "fuelmain_sha": "1c042c80f6d101800c393b2f626cd9279a8e45b1", "astute_sha": "401bc474b1d8cebb8ba70b3b6154107e08fd725d", "release": "5.0", "fuellib_sha": "101f3645ead182bc47024ff7568b04554de06bba"}

Apply patch for neutron (https://review.openstack.org/#/c/86926/7) on master node manifests

1. Cluster simple
2. CentOS
3. Neutron GRE
4. Controller only

(/Stage[main]/Rabbitmq::Service/Service[rabbitmq-server]/ensure) change from stopped to running failed: Could not start Service[rabbitmq-server]: Execution of '/sbin/service rabbitmq-server start' returned 2:

[root@node-13 ~]# /sbin/service rabbitmq-server start
Starting rabbitmq-server: RabbitMQ is going to make 3 attempts to find master node and start.
3 attempts left to start RabbitMQ Server before consider start failed.
2 attempts left to start RabbitMQ Server before consider start failed.
1 attempts left to start RabbitMQ Server before consider start failed.
FAILED - check /var/log/rabbitmq/startup_{log, _err}
Setting policy "ha-all" for pattern "." to "{\"ha-mode\":\"all\", \"ha-sync-mode\":\"automatic\"}" with priority "0" ...
Error: unable to connect to node 'rabbit@node-13': nodedown

DIAGNOSTICS
===========

nodes in question: ['rabbit@node-13']

hosts, their running nodes and ports:
- unable to connect to epmd on node-13: address (cannot connect to host/port)

current node details:
- node name: 'rabbitmqctl5345@node-13'
- home dir: /var/lib/rabbitmq
- cookie hash: iHzRAgOqSpPAppXAwZ5KGg==

Tags: icehouse
Revision history for this message
Ivan Berezovskiy (iberezovskiy) wrote :
description: updated
Changed in fuel:
importance: Undecided → Critical
milestone: none → 5.0
assignee: nobody → Fuel Library Team (fuel-library)
Revision history for this message
Sergey Vasilenko (xenolog) wrote :

# cat /etc/sysconfig/network-scripts/ifcfg-br-mgmt
DEVICE=br-mgmt
IPADDR=192.168.0.8
NETMASK=255.255.255.0
BOOTPROTO=none
ONBOOT=yes
USERCTL=no

[root@node-13 ~]# ifup br-mgmt ; echo $?
Determining if ip address 192.168.0.8 is already in use for device br-mgmt...
Error, some other host already uses address 192.168.0.8.
1
[root@node-13 ~]#

[root@node-13 ~]# ip -f inet a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
15: br-storage: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 192.168.1.6/24 brd 192.168.1.255 scope global br-storage
16: br-fw-admin: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 10.108.50.9/24 brd 10.108.50.255 scope global br-fw-admin
[root@node-13 ~]# ip a show br-mgmt
14: br-mgmt: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 9a:d2:6b:05:62:48 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::98d2:6bff:fe05:6248/64 scope link
       valid_lft forever preferred_lft forever

[root@node-13 ~]# ip a add 192.168.0.118/24 dev br-mgmt
[root@node-13 ~]# ping 192.168.0.8
PING 192.168.0.8 (192.168.0.8) 56(84) bytes of data.
64 bytes from 192.168.0.8: icmp_seq=1 ttl=64 time=2.06 ms
64 bytes from 192.168.0.8: icmp_seq=2 ttl=64 time=0.378 ms
^C
--- 192.168.0.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1965ms
rtt min/avg/max/mdev = 0.378/1.220/2.062/0.842 ms
[root@node-13 ~]# arp -an
? (192.168.0.8) at 7e:f1:47:3a:bc:4d [ether] on br-mgmt
? (10.108.50.2) at 64:42:6a:59:df:d2 [ether] on br-fw-admin
[root@node-13 ~]#

Changed in fuel:
status: New → Invalid
Changed in fuel:
importance: Critical → High
status: Invalid → Triaged
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Ok, the source of the issue is IP address conflict - probably an another node (external to the deployed OS env) took a 192.168.0.8 IP causing this one to fail. Hence, invalid, indeed...

Changed in fuel:
status: Triaged → Incomplete
Changed in fuel:
status: Incomplete → Invalid
Revision history for this message
Ilya Pekelny (i159) wrote :

I'm trying to deploy 3 compute cluster for RabbitMQ load testing. I've got the same problem. My network looks correct.

root@node-1:~# ifconfig | grep 192.1
          inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0
          inet addr:192.168.0.2 Bcast:0.0.0.0 Mask:255.255.255.0
          inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0

root@node-1:~# ifconfig | grep 10.2
          inet addr:10.20.0.5 Bcast:10.20.0.255 Mask:255.255.255.0

root@node-2:~# ifconfig | grep 192.1
          inet addr:192.168.0.3 Bcast:192.168.0.255 Mask:255.255.255.0
          inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0

root@node-2:~# ifconfig | grep 10.2
          inet addr:10.20.0.3 Bcast:10.20.0.255 Mask:255.255.255.0

root@node-3:~# ifconfig | grep 192.1
          inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0
          inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0

root@node-3:~# ifconfig | grep 10.2
          inet addr:10.20.0.4 Bcast:10.20.0.255 Mask:255.255.255.0

root@node-1:~# sudo service rabbitmq-server restart
 * Restarting message broker rabbitmq-server RabbitMQ is going to make 3 attempts to find master node and start.
3 attempts left to start RabbitMQ Server before consider start failed.

.... infinite waiting

Changed in fuel:
status: Invalid → New
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Simple mode was deprecated, lowering to medium

Changed in fuel:
importance: High → Medium
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

@Ilya, please provide a diagnostic logs snapshot

Changed in fuel:
status: New → Incomplete
assignee: Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)
Changed in fuel:
assignee: Bogdan Dobrelya (bogdando) → Ilya Pekelny (i159)
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

This bug has not been updated for 4 weeks. Marking it as Invalid. If you think it is not Invalid, please provide your comments and update bug status.

Changed in fuel:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.