bug in df-l3-agent in multinode

[Yuli]

After installing the latest version of the code in the
multinode environment on the second host I am getting
the following error in logs:

AMQP server on 127.0.0.1:5672 is unreachable:

[Li Ma (nick-ma-z)]

I think it has something related with rabbitmq setup in your second node. The rabbitmq client connects 127.0.0.1, not the controller node.

Do you set SERVICE_HOST and RABBIT_HOST in devstack local.conf?

[Yuli (stremovsky)]

In the second host I have the following config
(the IP is correct):

SERVICE_HOST=10.100.100.8
RABBIT_HOST=$SERVICE_HOST

[duankebo (duankebo)]

But the log "AMQP server on 127.0.0.1:5672 is unreachable" says the computer node is connecting 127.0.0.1. This means the configuration doesn't take effect. Will you please check you local configuration.

[duankebo (duankebo)]

And can you send to whole log file and configuration files to me ?

[Yuli (stremovsky)]

duankebo I send you the config files.

[Yuli (stremovsky)]

cd /opt/stack/dragonflow
grep RABBIT_HOST * -r

doc/source/multi-node-conf/compute_node_local_controller.conf:RABBIT_HOST=$SERVICE_HOST
doc/source/multi-node-conf/ramcloud_compute_node.conf:RABBIT_HOST=$SERVICE_HOST
doc/source/multi-node-conf/zookeeper_compute_node.conf:RABBIT_HOST=$SERVICE_HOST
doc/source/multi-node-conf/redis_compute_node_local_controller.conf:RABBIT_HOST=$SERVICE_HOST
doc/source/multi-node-conf/rethinkDB_compute_node.conf:RABBIT_HOST=$SERVICE_HOST
doc/source/readme.rst: RABBIT_HOST <- Management IP address of the controller node

RABBIT_HOST variable is not used, except for config file.

[Yuli (stremovsky)]

stack@dfstack:/opt/stack/neutron$ grep -i rabbit -A1 /opt/stack/logs/stack.sh.log
2016-05-10 08:38:24.894 | +./stack.sh:main:651 RABBIT_USERID=stackrabbit
2016-05-10 08:38:24.900 | +./stack.sh:main:652 is_service_enabled rabbit
2016-05-10 08:38:24.936 | +functions-common:is_service_enabled:2077 return 1
--
2016-05-10 08:40:35.629 | +lib/rpc_backend:install_rpc_backend:47 is_service_enabled rabbit
2016-05-10 08:40:35.669 | +functions-common:is_service_enabled:2077 return 1
--
2016-05-10 08:44:06.243 | +lib/rpc_backend:iniset_rpc_backend:117 is_service_enabled rabbit
2016-05-10 08:44:06.280 | +functions-common:is_service_enabled:2077 return 1
--
2016-05-10 08:44:06.301 | +lib/rpc_backend:iniset_rpc_backend:118 iniset /etc/nova/nova.conf DEFAULT rpc_backend rabbit
2016-05-10 08:44:06.346 | +lib/rpc_backend:iniset_rpc_backend:119 iniset /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts 10.100.100.8
2016-05-10 08:44:06.394 | +lib/rpc_backend:iniset_rpc_backend:120 iniset /etc/nova/nova.conf oslo_messaging_rabbit rabbit_password devstack
2016-05-10 08:44:06.439 | +lib/rpc_backend:iniset_rpc_backend:121 iniset /etc/nova/nova.conf oslo_messaging_rabbit rabbit_userid stackrabbit
2016-05-10 08:44:06.486 | +lib/rpc_backend:iniset_rpc_backend:122 '[' -n '' ']'
--
2016-05-10 08:50:17.083 | +lib/rpc_backend:restart_rpc_backend:58 is_service_enabled rabbit
2016-05-10 08:50:17.120 | +functions-common:is_service_enabled:2077 return 1

[Yuli (stremovsky)]

stack@dfstack:/opt/stack/neutron$ grep rabbit -i /etc/neutron/neutron.conf
# The messaging driver to use, defaults to rabbit. Other drivers include amqp
#rpc_backend = rabbit
[oslo_messaging_rabbit]
# Deprecated group/name - [DEFAULT]/rabbit_durable_queues
# Determines how the next RabbitMQ node is chosen in case the one we are
# one RabbitMQ node is provided in config. (string value)
# The RabbitMQ broker address where a single node is used. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_host
#rabbit_host = localhost
# The RabbitMQ broker port where a single node is used. (port value)
# Deprecated group/name - [DEFAULT]/rabbit_port
#rabbit_port = 5672
# RabbitMQ HA cluster host:port pairs. (list value)
# Deprecated group/name - [DEFAULT]/rabbit_hosts
#rabbit_hosts = $rabbit_host:$rabbit_port
# Connect over SSL for RabbitMQ. (boolean value)
# Deprecated group/name - [DEFAULT]/rabbit_use_ssl
#rabbit_use_ssl = false
# The RabbitMQ userid. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_userid
#rabbit_userid = guest
# The RabbitMQ password. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_password
#rabbit_password = guest
# The RabbitMQ login method. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_login_method
#rabbit_login_method = AMQPLAIN
# The RabbitMQ virtual host. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_virtual_host
#rabbit_virtual_host = /
# How frequently to retry connecting with RabbitMQ. (integer value)
#rabbit_retry_interval = 1
# How long to backoff for between retries when connecting to RabbitMQ. (integer
# Deprecated group/name - [DEFAULT]/rabbit_retry_backoff
#rabbit_retry_backoff = 2
# Maximum interval of RabbitMQ connection retries. Default is 30 seconds.
#rabbit_interval_max = 30
# Maximum number of RabbitMQ connection retries. Default is 0 (infinite retry
# Deprecated group/name - [DEFAULT]/rabbit_max_retries
#rabbit_max_retries = 0
# Try to use HA queues in RabbitMQ (x-ha-policy: all). If you change this
# option, you must wipe the RabbitMQ database. In RabbitMQ 3.0, queue mirroring
# generated names) are mirrored across all nodes, run: "rabbitmqctl set_policy
# Deprecated group/name - [DEFAULT]/rabbit_ha_queues
#rabbit_ha_queues = false
#rabbit_transient_queues_ttl = 1800
#rabbit_qos_prefetch_count = 0
# Number of seconds after which the Rabbit broker is considered down if
# Deprecated group/name - [DEFAULT]/fake_rabbit
#fake_rabbit = false
# Max number of not acknowledged message which RabbitMQ can send to
# Max number of not acknowledged message which RabbitMQ can send to rpc
# Max number of not acknowledged message which RabbitMQ can send to rpc reply

[Yuli (stremovsky)]

Looks like in neutron, rabbit is not activly used:

stack@dfstack:/opt/stack/neutron$ grep RABBIT * -r
tools/configure_for_func_testing.sh: RABBIT_USERID=${RABBIT_USERID:-stackrabbit}
tools/configure_for_func_testing.sh: RABBIT_HOST=${RABBIT_HOST:-$SERVICE_HOST}
tools/configure_for_func_testing.sh: RABBIT_PASSWORD=${RABBIT_HOST:-secretrabbit}

[hujie (mike-hu)]

Hi, Yuli, would you please send out the stack.log file?

[hujie (mike-hu)]

In my opinion, compute node don’t need to enable df-l3-agent and if you did that, current Dragonflow script do not support to convert the rabbit specific configuration into /etc/neutron/neutron.conf,
  So you could not see the [oslo_messaging_rabbit] section in neutron.conf on compute node, and the default config is connect to 127.0.0.1:5672, indeed you just need to delete the enable df-l3-agent config from compute node and enable it on
Control node which is also the network node, the rabbit client will connect to server by the default config 127.0.0.1:5672 and it's done

[Yuli (stremovsky)]

When working with FIP do I need to run the df-l3-agent on all computer nodes ?

[hujie (mike-hu)]

Hi Yuli, if this bug is done, we could close it:)