Deploying cs:trusty/rabbitmq-server-50 to an LXC on a VM as part of Openstack installation breaks

[Imran Khakoo]

Juju status is forever stuck in this status:

$ juju status rabbitmq-server

environment: maas
machines:
  "1":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: VMNode2MAAS.maas
    instance-id: /MAAS/api/1.0/nodes/node-bca10166-8c6b-11e6-b752-0cc47a2275d4/
    series: trusty
    containers:
      1/lxc/1:
        agent-state: started
        agent-version: 1.25.6.1
        dns-name: 10.1.1.112
        instance-id: juju-machine-1-lxc-1
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual availability-zone=default
services:
  rabbitmq-server:
    charm: cs:trusty/rabbitmq-server-50
    exposed: false
    life: dying
    service-status:
      current: maintenance
      message: 'Waiting for rabbitmq app to start: /var/run/rabbitmq/pid'
      since: 09 Oct 2016 14:53:10+02:00
    relations:
      amqp:
      - neutron-api
      cluster:
      - rabbitmq-server
    units:
      rabbitmq-server/0:
        workload-status:
          current: maintenance
          message: 'Waiting for rabbitmq app to start: /var/run/rabbitmq/pid'
          since: 09 Oct 2016 14:53:10+02:00
        agent-status:
          current: executing
          message: running update-status hook
          since: 09 Oct 2016 14:53:09+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        life: dying
        machine: 1/lxc/1
        open-ports:
        - 5672/tcp
        public-address: 10.1.1.112
networks:
  subnet-1:
    provider-id: subnet-1
    cidr: 172.26.0.0/22
  subnet-2:
    provider-id: subnet-2
    cidr: 10.1.1.0/24

Debugging on the machine itself:

Doing a service restart:
ubuntu@juju-machine-1-lxc-1:~$ sudo service rabbitmq-server restart
 * Restarting message broker rabbitmq-server [fail]

Checking /var/log/rabbitmq/startup_log:
ERROR: epmd error for host "blue-drop": address (cannot connect to host/port)

For the above, I do have a machine called blue-drop but its fully qualified DNS name is blue-drop.maas. Could this be what's causing a problem with rabbitMQ?

How do I resolve the above situation?

Full juju-status:

environment: maas
machines:
  "0":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: VMNode1MAAS.maas
    instance-id: /MAAS/api/1.0/nodes/node-83c24f76-8c6b-11e6-b70c-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual availability-zone=default
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: VMNode2MAAS.maas
    instance-id: /MAAS/api/1.0/nodes/node-bca10166-8c6b-11e6-b752-0cc47a2275d4/
    series: trusty
    containers:
      1/lxc/0:
        agent-state: started
        agent-version: 1.25.6.1
        dns-name: 10.1.1.144
        instance-id: juju-machine-1-lxc-0
        series: trusty
        hardware: arch=amd64
      1/lxc/1:
        agent-state: started
        agent-version: 1.25.6.1
        dns-name: 10.1.1.112
        instance-id: juju-machine-1-lxc-1
        series: trusty
        hardware: arch=amd64
      1/lxc/2:
        agent-state: started
        agent-version: 1.25.6.1
        dns-name: 10.1.1.150
        instance-id: juju-machine-1-lxc-2
        series: trusty
        hardware: arch=amd64
      1/lxc/3:
        agent-state: started
        agent-version: 1.25.6.1
        dns-name: 10.1.1.132
        instance-id: juju-machine-1-lxc-3
        series: trusty
        hardware: arch=amd64
      1/lxc/4:
        agent-state: started
        agent-version: 1.25.6.1
        dns-name: 10.1.1.129
        instance-id: juju-machine-1-lxc-4
        series: trusty
        hardware: arch=amd64
      1/lxc/5:
        agent-state: started
        agent-version: 1.25.6.1
        dns-name: 10.1.1.109
        instance-id: juju-machine-1-lxc-5
        series: trusty
        hardware: arch=amd64
      1/lxc/6:
        agent-state: started
        agent-version: 1.25.6.1
        dns-name: 10.1.1.106
        instance-id: juju-machine-1-lxc-6
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual availability-zone=default
  "2":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: blue-drop.maas
    instance-id: /MAAS/api/1.0/nodes/node-b76c3dca-5a50-11e6-8378-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=24 mem=131072M tags=node-b76c3dca-5a50-11e6-8378-0cc47a2275d4
      availability-zone=default
  "3":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: chilly-comparison.maas
    instance-id: /MAAS/api/1.0/nodes/node-3db96c1c-5ade-11e6-946e-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=32 mem=65536M tags=node-3db96c1c-5ade-11e6-946e-0cc47a2275d4
      availability-zone=default
  "4":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: old-vm-hostMAAS.maas
    instance-id: /MAAS/api/1.0/nodes/node-f1da8b92-8c6e-11e6-b1df-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=32 mem=65536M availability-zone=default
  "5":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: VMNode3MAAS.maas
    instance-id: /MAAS/api/1.0/nodes/node-dda28254-8c6b-11e6-9074-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual availability-zone=default
  "6":
    agent-state: started
    agent-version: 1.25.6.1
    dns-name: VMNode4MAAS.maas
    instance-id: /MAAS/api/1.0/nodes/node-f87b8338-8c6f-11e6-862a-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual availability-zone=default
services:
  cinder-api:
    charm: cs:trusty/cinder-255
    exposed: false
    service-status:
      current: blocked
      message: 'Missing relations: messaging'
      since: 10 Oct 2016 09:52:45+02:00
    relations:
      cinder-volume-service:
      - nova-cloud-controller
      cluster:
      - cinder-api
      identity-service:
      - keystone
      image-service:
      - glance
      shared-db:
      - mysql
    units:
      cinder-api/0:
        workload-status:
          current: blocked
          message: 'Missing relations: messaging'
          since: 10 Oct 2016 09:52:45+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:52:48+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: 1/lxc/6
        open-ports:
        - 8776/tcp
        public-address: 10.1.1.106
  cinder-volume:
    charm: cs:trusty/cinder-255
    exposed: false
    service-status:
      current: blocked
      message: 'Missing relations: messaging'
      since: 10 Oct 2016 09:56:10+02:00
    relations:
      cluster:
      - cinder-volume
      image-service:
      - glance
      shared-db:
      - mysql
    units:
      cinder-volume/0:
        workload-status:
          current: blocked
          message: 'Missing relations: messaging'
          since: 10 Oct 2016 09:56:10+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:56:12+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: "5"
        open-ports:
        - 8776/tcp
        public-address: VMNode3MAAS.maas
  glance:
    charm: cs:trusty/glance-251
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 10 Oct 2016 09:54:16+02:00
    relations:
      cluster:
      - glance
      identity-service:
      - keystone
      image-service:
      - cinder-api
      - cinder-volume
      - nova-cloud-controller
      - nova-compute
      shared-db:
      - mysql
    units:
      glance/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 10 Oct 2016 09:54:16+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:54:18+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: 1/lxc/3
        open-ports:
        - 9292/tcp
        public-address: 10.1.1.132
  keystone:
    charm: cs:trusty/keystone-256
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 10 Oct 2016 09:56:09+02:00
    relations:
      cluster:
      - keystone
      identity-service:
      - cinder-api
      - glance
      - neutron-api
      - nova-cloud-controller
      - openstack-dashboard
      shared-db:
      - mysql
    units:
      keystone/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 10 Oct 2016 09:56:09+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:56:11+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: 1/lxc/2
        open-ports:
        - 5000/tcp
        public-address: 10.1.1.150
  mysql:
    charm: cs:trusty/mysql-55
    exposed: false
    service-status:
      current: unknown
      since: 09 Oct 2016 14:17:10+02:00
    relations:
      cluster:
      - mysql
      shared-db:
      - cinder-api
      - cinder-volume
      - glance
      - keystone
      - neutron-api
      - nova-cloud-controller
      - nova-compute
    units:
      mysql/1:
        workload-status:
          current: unknown
          since: 09 Oct 2016 14:17:10+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:52:17+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: 1/lxc/0
        open-ports:
        - 3306/tcp
        public-address: 10.1.1.144
  neutron-api:
    charm: cs:trusty/neutron-api-244
    exposed: false
    service-status:
      current: blocked
      message: 'Missing relations: messaging'
      since: 10 Oct 2016 09:56:10+02:00
    relations:
      amqp:
      - rabbitmq-server
      cluster:
      - neutron-api
      identity-service:
      - keystone
      neutron-api:
      - nova-cloud-controller
      neutron-plugin-api:
      - neutron-gateway
      - neutron-openvswitch
      shared-db:
      - mysql
    units:
      neutron-api/0:
        workload-status:
          current: blocked
          message: 'Missing relations: messaging'
          since: 10 Oct 2016 09:56:10+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:56:13+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: 1/lxc/4
        open-ports:
        - 9696/tcp
        public-address: 10.1.1.129
  neutron-gateway:
    charm: cs:trusty/neutron-gateway-230
    exposed: false
    service-status:
      current: blocked
      message: 'Missing relations: messaging'
      since: 10 Oct 2016 09:52:27+02:00
    relations:
      cluster:
      - neutron-gateway
      neutron-plugin-api:
      - neutron-api
    units:
      neutron-gateway/0:
        workload-status:
          current: blocked
          message: 'Missing relations: messaging'
          since: 10 Oct 2016 09:52:27+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:52:30+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: "2"
        public-address: blue-drop.maas
  neutron-openvswitch:
    charm: cs:trusty/neutron-openvswitch-236
    exposed: false
    service-status: {}
    relations:
      neutron-plugin:
      - nova-compute
      neutron-plugin-api:
      - neutron-api
    subordinate-to:
    - nova-compute
  nova-cloud-controller:
    charm: cs:trusty/nova-cloud-controller-290
    exposed: false
    service-status:
      current: blocked
      message: 'Missing relations: messaging'
      since: 10 Oct 2016 09:56:25+02:00
    relations:
      cinder-volume-service:
      - cinder-api
      cloud-compute:
      - nova-compute
      cluster:
      - nova-cloud-controller
      identity-service:
      - keystone
      image-service:
      - glance
      neutron-api:
      - neutron-api
      shared-db:
      - mysql
    units:
      nova-cloud-controller/0:
        workload-status:
          current: blocked
          message: 'Missing relations: messaging'
          since: 10 Oct 2016 09:56:25+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:56:28+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: 1/lxc/5
        open-ports:
        - 8774/tcp
        public-address: 10.1.1.109
  nova-compute:
    charm: cs:trusty/nova-compute-257
    exposed: false
    service-status:
      current: blocked
      message: 'Missing relations: messaging'
      since: 10 Oct 2016 09:53:26+02:00
    relations:
      cloud-compute:
      - nova-cloud-controller
      compute-peer:
      - nova-compute
      image-service:
      - glance
      neutron-plugin:
      - neutron-openvswitch
      shared-db:
      - mysql
    units:
      nova-compute/0:
        workload-status:
          current: blocked
          message: 'Missing relations: messaging'
          since: 10 Oct 2016 09:56:26+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:56:29+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: "3"
        public-address: chilly-comparison.maas
        subordinates:
          neutron-openvswitch/1:
            workload-status:
              current: blocked
              message: 'Missing relations: messaging'
              since: 09 Oct 2016 17:21:42+02:00
            agent-status:
              current: idle
              since: 10 Oct 2016 09:54:46+02:00
              version: 1.25.6.1
            agent-state: started
            agent-version: 1.25.6.1
            upgrading-from: cs:trusty/neutron-openvswitch-236
            public-address: chilly-comparison.maas
      nova-compute/1:
        workload-status:
          current: blocked
          message: 'Missing relations: messaging'
          since: 10 Oct 2016 09:53:26+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:53:28+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: "4"
        public-address: old-vm-hostMAAS.maas
        subordinates:
          neutron-openvswitch/0:
            workload-status:
              current: blocked
              message: 'Missing relations: messaging'
              since: 09 Oct 2016 17:21:42+02:00
            agent-status:
              current: idle
              since: 10 Oct 2016 09:54:06+02:00
              version: 1.25.6.1
            agent-state: started
            agent-version: 1.25.6.1
            upgrading-from: cs:trusty/neutron-openvswitch-236
            public-address: old-vm-hostMAAS.maas
  openstack-dashboard:
    charm: cs:trusty/openstack-dashboard-241
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 10 Oct 2016 09:53:40+02:00
    relations:
      cluster:
      - openstack-dashboard
      identity-service:
      - keystone
    units:
      openstack-dashboard/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 10 Oct 2016 09:53:40+02:00
        agent-status:
          current: idle
          since: 10 Oct 2016 09:53:42+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        machine: "6"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: VMNode4MAAS.maas
  rabbitmq-server:
    charm: cs:trusty/rabbitmq-server-50
    exposed: false
    life: dying
    service-status:
      current: maintenance
      message: 'Waiting for rabbitmq app to start: /var/run/rabbitmq/pid'
      since: 09 Oct 2016 14:53:10+02:00
    relations:
      amqp:
      - neutron-api
      cluster:
      - rabbitmq-server
    units:
      rabbitmq-server/0:
        workload-status:
          current: maintenance
          message: 'Waiting for rabbitmq app to start: /var/run/rabbitmq/pid'
          since: 09 Oct 2016 14:53:10+02:00
        agent-status:
          current: executing
          message: running update-status hook
          since: 09 Oct 2016 14:53:09+02:00
          version: 1.25.6.1
        agent-state: started
        agent-version: 1.25.6.1
        life: dying
        machine: 1/lxc/1
        open-ports:
        - 5672/tcp
        public-address: 10.1.1.112
networks:
  subnet-1:
    provider-id: subnet-1
    cidr: 172.26.0.0/22
  subnet-2:
    provider-id: subnet-2
    cidr: 10.1.1.0/24

[Imran Khakoo (imrankhakoo)]

EDIT:
$ sudo nano /etc/rabbitmq/rabbitmq-env.conf

has

RABBITMQ_NODENAME=rabbit@blue-drop

which I modified to

<email address hidden>

and tried a service rabbitmq-server restart. Still yielded a fail.

Unresolved

[Imran Khakoo (imrankhakoo)]

Another question is, the LXC on which rabbitMQ is installed, is not even on blue-drop.maas. I SSH'ed into blue-drop.maas and couldn't find any trace of rabbitMQ. Any clue why it thought blue-drop was the RABBITMQ_NODENAME?

[James Page (james-page)]

RabbitMQ is very sensitive to mis-configured DNS (it entirely relies on hostnames being resolvable), so the charm will resolve the current IP on the LXC container to a hostname, and then configure the internal NODENAME to that hostname.

Is is possible you have a subnet clash in DNS? it looks like 10.1.1.112 is resolving to blue-drop, which is obviously not the hostname of the container.

[Imran Khakoo (imrankhakoo)]

Yes that was it. I removed rabbitmq and redeployed it to another node and it worked fine. Thanks.

[Dimiter Naydenov (dimitern)]

I've found the same issue with cs:xenial/rabbitmq-server-54 on MAAS 2.0: when rabbitmq-server is deployed to a LXD container with multiple NICs, I can see in the unit log "getting local hostname" and trying to set rabbit@eth2 (which fails). Trying to reverse-resolve the private address 10.100.0.110, returns 'eth2.juju-xyz-machine-1-lxd-2.maas', which is typical with MAAS 2.0. So I think using "local hostname" is wrong, it should use the FQDN or at least that part of it which ends with $(hostname) - in my case "eth2.juju-xyz-machine-1-lxd-2.".

Note that, even though --bind 'amqp=internal-api' was used during deployment, the charm still tries to use the unit-get private-address and access-network (not set) and does not call network-get amqp --primary-address