Juju Openstack, manual placement: Nova and Neutron can't communicate with instance.

[Imran Khakoo]

Nova Logs:(/var/log/nova/nova-compute.log is filled with these)

2016-09-12 13:00:30.521 8832 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.
2016-09-12 13:01:00.546 8832 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on localhost:5672
2016-09-12 13:01:00.547 8832 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...

Neutron Logs: (All the logs in /var/log/neutron/*.log are filled with these)
2016-09-12 13:36:14.807 9872 ERROR neutron.openstack.common.rpc.common [req-f0c8e9cb-49df-4dcf-b606-b64e052f46d4 None] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds.

Juju Status:
environment: maas
machines:
  "0":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode6.maas
    instance-id: /MAAS/api/1.0/nodes/node-94940378-7373-11e6-a997-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=2 mem=4096M tags=node-94940378-7373-11e6-a997-0cc47a2275d4
      availability-zone=default
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.25.6
    dns-name: throwaway.maas
    instance-id: /MAAS/api/1.0/nodes/node-6296524e-70e5-11e6-a8fd-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1024M tags=virtual,node-6296524e-70e5-11e6-a8fd-0cc47a2275d4
      availability-zone=default
  "3":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode1.maas
    instance-id: /MAAS/api/1.0/nodes/node-258495a0-6087-11e6-b265-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual,node-258495a0-6087-11e6-b265-0cc47a2275d4
      availability-zone=default
  "4":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode2.maas
    instance-id: /MAAS/api/1.0/nodes/node-268ef3dc-6087-11e6-8b87-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual,node-268ef3dc-6087-11e6-8b87-0cc47a2275d4
      availability-zone=default
  "5":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode3.maas
    instance-id: /MAAS/api/1.0/nodes/node-7ee52eb2-608b-11e6-8aa7-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual,node-7ee52eb2-608b-11e6-8aa7-0cc47a2275d4
      availability-zone=default
  "6":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode4.maas
    instance-id: /MAAS/api/1.0/nodes/node-7eec4684-608b-11e6-bb4e-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual,node-7eec4684-608b-11e6-bb4e-0cc47a2275d4
      availability-zone=default
  "7":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode5.maas
    instance-id: /MAAS/api/1.0/nodes/node-7e9aa7fc-608b-11e6-9a59-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=4 mem=4096M tags=virtual,node-7e9aa7fc-608b-11e6-9a59-0cc47a2275d4
      availability-zone=default
  "8":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode7.maas
    instance-id: /MAAS/api/1.0/nodes/node-1ac2cf9c-7374-11e6-99fc-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=2 mem=4096M tags=node-1ac2cf9c-7374-11e6-99fc-0cc47a2275d4
      availability-zone=default
  "9":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode8.maas
    instance-id: /MAAS/api/1.0/nodes/node-cb383560-7374-11e6-a7a3-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=2 mem=4096M tags=node-cb383560-7374-11e6-a7a3-0cc47a2275d4
      availability-zone=default
  "10":
    agent-state: started
    agent-version: 1.25.6
    dns-name: blue-drop.maas
    instance-id: /MAAS/api/1.0/nodes/node-b76c3dca-5a50-11e6-8378-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=24 mem=131072M tags=node-b76c3dca-5a50-11e6-8378-0cc47a2275d4
      availability-zone=default
  "11":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode12.maas
    instance-id: /MAAS/api/1.0/nodes/node-3c59d7ee-78e8-11e6-97d1-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=2048M availability-zone=default
  "12":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode10.maas
    instance-id: /MAAS/api/1.0/nodes/node-3c593000-78e8-11e6-b2b0-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=2048M availability-zone=default
  "13":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode11.maas
    instance-id: /MAAS/api/1.0/nodes/node-3c65d51c-78e8-11e6-afe5-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=2048M availability-zone=default
  "14":
    agent-state: started
    agent-version: 1.25.6
    dns-name: VMNode9.maas
    instance-id: /MAAS/api/1.0/nodes/node-3cde3eda-78e8-11e6-9f85-0cc47a2275d4/
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=2048M availability-zone=default
services:
  ceph:
    charm: cs:trusty/ceph-265
    exposed: false
    service-status:
      current: active
      message: Unit is ready and clustered
      since: 12 Sep 2016 16:47:15+02:00
    relations:
      mon:
      - ceph
    units:
      ceph/0:
        workload-status:
          current: active
          message: Unit is ready and clustered
          since: 12 Sep 2016 16:47:15+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:47:17+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "11"
        public-address: VMNode12.maas
      ceph/1:
        workload-status:
          current: active
          message: Unit is ready and clustered
          since: 12 Sep 2016 16:48:32+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:48:34+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "12"
        public-address: VMNode10.maas
      ceph/2:
        workload-status:
          current: active
          message: Unit is ready and clustered
          since: 12 Sep 2016 16:47:21+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:47:24+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "13"
        public-address: VMNode11.maas
  ceph-radosgw:
    charm: cs:trusty/ceph-radosgw-242
    exposed: false
    service-status:
      current: blocked
      message: 'Missing relations: mon'
      since: 12 Sep 2016 16:49:14+02:00
    relations:
      cluster:
      - ceph-radosgw
    units:
      ceph-radosgw/0:
        workload-status:
          current: blocked
          message: 'Missing relations: mon'
          since: 12 Sep 2016 16:49:14+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:49:16+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "14"
        public-address: VMNode9.maas
  cinder:
    charm: cs:trusty/cinder-255
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 12 Sep 2016 16:50:47+02:00
    relations:
      amqp:
      - rabbitmq-server
      cinder-volume-service:
      - nova-cloud-controller
      cluster:
      - cinder
      identity-service:
      - keystone
      shared-db:
      - mysql
    units:
      cinder/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 12 Sep 2016 16:50:47+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:50:50+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "7"
        open-ports:
        - 8776/tcp
        public-address: VMNode5.maas
  glance:
    charm: cs:trusty/glance-251
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 12 Sep 2016 16:46:53+02:00
    relations:
      cluster:
      - glance
      identity-service:
      - keystone
      image-service:
      - nova-cloud-controller
      - nova-compute
      shared-db:
      - mysql
    units:
      glance/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 12 Sep 2016 16:46:53+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:46:55+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "9"
        open-ports:
        - 9292/tcp
        public-address: VMNode8.maas
  juju-gui:
    charm: cs:trusty/juju-gui-135
    exposed: true
    service-status:
      current: unknown
      since: 12 Sep 2016 14:34:23+02:00
    units:
      juju-gui/0:
        workload-status:
          current: unknown
          since: 12 Sep 2016 14:34:23+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:49:34+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "0"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: VMNode6.maas
  keystone:
    charm: cs:trusty/keystone-256
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 12 Sep 2016 16:46:22+02:00
    relations:
      cluster:
      - keystone
      identity-service:
      - cinder
      - glance
      - nova-cloud-controller
      - openstack-dashboard
      shared-db:
      - mysql
    units:
      keystone/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 12 Sep 2016 16:46:22+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:46:24+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "5"
        open-ports:
        - 5000/tcp
        public-address: VMNode3.maas
  mysql:
    charm: cs:trusty/mysql-55
    exposed: false
    service-status:
      current: unknown
      since: 12 Sep 2016 14:51:01+02:00
    relations:
      cluster:
      - mysql
      shared-db:
      - cinder
      - glance
      - keystone
      - nova-cloud-controller
      - nova-compute
    units:
      mysql/1:
        workload-status:
          current: unknown
          since: 12 Sep 2016 14:51:01+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:46:26+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "3"
        open-ports:
        - 3306/tcp
        public-address: VMNode1.maas
  neutron-gateway:
    charm: cs:trusty/neutron-gateway-230
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 12 Sep 2016 16:50:21+02:00
    relations:
      amqp:
      - rabbitmq-server
      cluster:
      - neutron-gateway
      quantum-network-service:
      - nova-cloud-controller
    units:
      neutron-gateway/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 12 Sep 2016 16:50:21+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:50:23+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "6"
        public-address: VMNode4.maas
  nova-cloud-controller:
    charm: cs:trusty/nova-cloud-controller-290
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 12 Sep 2016 16:47:32+02:00
    relations:
      amqp:
      - rabbitmq-server
      cinder-volume-service:
      - cinder
      cloud-compute:
      - nova-compute
      cluster:
      - nova-cloud-controller
      identity-service:
      - keystone
      image-service:
      - glance
      quantum-network-service:
      - neutron-gateway
      shared-db:
      - mysql
    units:
      nova-cloud-controller/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 12 Sep 2016 16:47:32+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:47:34+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "8"
        open-ports:
        - 3333/tcp
        - 8773/tcp
        - 8774/tcp
        public-address: VMNode7.maas
  nova-compute:
    charm: cs:trusty/nova-compute-257
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 12 Sep 2016 16:48:18+02:00
    relations:
      amqp:
      - rabbitmq-server
      cloud-compute:
      - nova-cloud-controller
      compute-peer:
      - nova-compute
      image-service:
      - glance
      shared-db:
      - mysql
    units:
      nova-compute/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 12 Sep 2016 16:48:18+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:48:20+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "10"
        public-address: blue-drop.maas
  openstack-dashboard:
    charm: cs:trusty/openstack-dashboard-241
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 12 Sep 2016 16:50:49+02:00
    relations:
      cluster:
      - openstack-dashboard
      identity-service:
      - keystone
    units:
      openstack-dashboard/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 12 Sep 2016 16:50:49+02:00
        agent-status:
          current: executing
          message: running update-status hook
          since: 12 Sep 2016 16:50:44+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "4"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: VMNode2.maas
  rabbitmq-server:
    charm: cs:trusty/rabbitmq-server-49
    exposed: false
    service-status:
      current: active
      message: Unit is ready
      since: 12 Sep 2016 16:46:16+02:00
    relations:
      amqp:
      - cinder
      - neutron-gateway
      - nova-cloud-controller
      - nova-compute
      cluster:
      - rabbitmq-server
    units:
      rabbitmq-server/0:
        workload-status:
          current: active
          message: Unit is ready
          since: 12 Sep 2016 16:46:16+02:00
        agent-status:
          current: idle
          since: 12 Sep 2016 16:46:18+02:00
          version: 1.25.6
        agent-state: started
        agent-version: 1.25.6
        machine: "1"
        open-ports:
        - 5672/tcp
        public-address: throwaway.maas
networks:
  subnet-1:
    provider-id: subnet-1
    cidr: 172.26.0.0/22
  subnet-2:
    provider-id: subnet-2
    cidr: 10.1.1.0/24
  subnet-3:
    provider-id: subnet-3
    cidr: 192.168.122.0/24

[James Page (james-page)]

With regards to the logging of connectivity issues from nova and neutron to a RMQ on localhost - this is actually normal until the amqp relations are complete; packaging install and automatically start all services, and the default RMQ configuration URL points to localhost - so I don't think that is the root cause of your problem here.

I do note that you have not deployed the neutron-api or neutron-openvswitch charms; these are required to support any use of Neutron and the ML2 OVS reference implementation for networking.

Is this a bundle you created yourself? or do we have a bundle published somewhere that has missing components....

Marking 'Incomplete' and 'Low' for now; if deploying the missing services resolves your problem, then that's great but please do let us know if you used some sort of official bundle that creates this deployment topology.

[Imran Khakoo (imrankhakoo)]

Hi, thanks for the response. The deployment eventually succeeded. I was deploying to 14.04 and no appropriate 14.04 Openstack bundle existed, so I was deploying manually. Thanks again.

[Launchpad Janitor (janitor)]

[Expired for OpenStack rabbitmq-server charm because there has been no activity for 60 days.]