nova-vmware failed to launch VM

[Ashley Lai]

Launching a VM gives the following error for neutron-nsx on nova-vmware.

jenkins@s9m16j:~/workspace/pipeline_deploy$ nova list
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
| 9e6a26b9-38da-4932-84fb-ef28f15264ee | guestOS-test-vmware-trusty | ERROR | - | NOSTATE | |
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
jenkins@s9m16j:~/workspace/pipeline_deploy$ nova console-log guestOS-test-vmware-trusty
ERROR: Unable to get console log, functionality not implemented (HTTP 501) (Request-ID: req-86b5cc76-df33-4f4b-a99f-c587f0eeaf05)
jenkins@s9m16j:~/workspace/pipeline_deploy$ nova show guestOS-test-vmware-trusty
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | hayward-39 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | domain-c397(openstack) |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2015-10-20T17:25:21Z |
| fault | {"message": "NotImplementedError", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 311, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/compute/manager.py\", line 4078, in get_console_output |
| | output = self.driver.get_console_output(context, instance) |
| | File \"/usr/lib/python2.7/dist-packages/nova/virt/driver.py\", line 366, in get_console_output |
| | raise NotImplementedError() |
| | ", "created": "2015-10-20T20:50:08Z"} |
| flavor | m1.small (2) |
| hostId | 4c709633a29be76c696ea3c5a10bc8f0b2ea465802e8233ecc5e166f |
| id | 9e6a26b9-38da-4932-84fb-ef28f15264ee |
| image | guestOS-test-vmware-trusty (3178fb0a-7683-4198-98f5-5908b96fae83) |
| key_name | guestOS_keypair |
| metadata | {} |
| name | guestOS-test-vmware-trusty |
| os-extended-volumes:volumes_attached | [] |
| security_groups | guestOS_secgroup |
| status | ERROR |
| tenant_id | 79d7f3e54508493c920faaf8c2787f14 |
| updated | 2015-10-20T17:25:32Z |
| user_id | e6abc08d32fe4387a90c78c2a0b60377 |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
jenkins@s9m16j:~/workspace/pipeline_deploy$

juju status:
environment: ci-oil-slave9
machines:
  "0":
    agent-state: started
    agent-version: 1.24.6
    dns-name: sunsetvm06.oil
    instance-id: /MAAS/api/1.0/nodes/node-5e986dca-5e18-11e5-98b2-00163e362f6f/
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=4096M tags=vm-staging,virtual,blacklist-storage,blacklist-compute,hw-alai-staging
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.24.6
    dns-name: hayward-33.oil
    instance-id: /MAAS/api/1.0/nodes/node-d0d25dcc-4b5a-11e5-a2c2-00163e362f6f/
    series: trusty
    containers:
      1/lxc/0:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.43.69
        instance-id: juju-machine-1-lxc-0
        series: trusty
        hardware: arch=amd64
      1/lxc/1:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.47.123
        instance-id: juju-machine-1-lxc-1
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=8 mem=16384M tags=oil-slave-1,hw-glance-sm15k,hardware-sm15k,hw-alai-staging
  "2":
    agent-state: started
    agent-version: 1.24.6
    dns-name: hayward-35.oil
    instance-id: /MAAS/api/1.0/nodes/node-bf0420d0-4b5a-11e5-988c-00163e362f6f/
    series: trusty
    containers:
      2/lxc/0:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.41.141
        instance-id: juju-machine-2-lxc-0
        series: trusty
        hardware: arch=amd64
      2/lxc/1:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.41.156
        instance-id: juju-machine-2-lxc-1
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=8 mem=16384M tags=oil-slave-3,hw-glance-sm15k,hardware-sm15k,hw-alai-staging
  "3":
    agent-state: started
    agent-version: 1.24.6
    dns-name: hayward-39.oil
    instance-id: /MAAS/api/1.0/nodes/node-cd31ddb4-4b5a-11e5-91ed-00163e362f6f/
    series: trusty
    containers:
      3/lxc/0:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.32.235
        instance-id: juju-machine-3-lxc-0
        series: trusty
        hardware: arch=amd64
      3/lxc/1:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.45.188
        instance-id: juju-machine-3-lxc-1
        series: trusty
        hardware: arch=amd64
      3/lxc/2:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.41.159
        instance-id: juju-machine-3-lxc-2
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=8 mem=16384M tags=oil-slave-1,hw-glance-sm15k,hardware-sm15k,hw-alai-staging
  "4":
    agent-state: started
    agent-version: 1.24.6
    dns-name: sunsetvm01.oil
    instance-id: /MAAS/api/1.0/nodes/node-52e3eb4e-5e18-11e5-83b1-00163e362f6f/
    series: trusty
    containers:
      4/lxc/0:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.40.205
        instance-id: juju-machine-4-lxc-0
        series: trusty
        hardware: arch=amd64
      4/lxc/1:
        agent-state: started
        agent-version: 1.24.6
        dns-name: 10.245.32.236
        instance-id: juju-machine-4-lxc-1
        series: trusty
        hardware: arch=amd64
    hardware: arch=amd64 cpu-cores=4 mem=8192M tags=vm-staging,virtual,whitelist-compute,hw-alai-staging
services:
  ceilometer:
    charm: local:trusty/ceilometer-44
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:12:13Z
    relations:
      amqp:
      - rabbitmq-server
      ceilometer-service:
      - ceilometer-agent
      cluster:
      - ceilometer
      identity-service:
      - keystone
      shared-db:
      - mongodb
    units:
      ceilometer/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:12:13Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:15:39Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 3/lxc/1
        open-ports:
        - 8777/tcp
        public-address: 10.245.45.188
  ceilometer-agent:
    charm: local:trusty/ceilometer-agent-20
    exposed: false
    service-status: {}
    relations:
      ceilometer-service:
      - ceilometer
  ceph:
    charm: local:trusty/ceph-105
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:11:48Z
    relations:
      client:
      - cinder
      - glance
      mon:
      - ceph
    units:
      ceph/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:11:48Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:14:54Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: "3"
        public-address: hayward-39.oil
  cinder:
    charm: local:trusty/cinder-136
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:03:15Z
    relations:
      amqp:
      - rabbitmq-server
      ceph:
      - ceph
      cinder-volume-service:
      - nova-cloud-controller
      cluster:
      - cinder
      identity-service:
      - keystone
      image-service:
      - glance
      shared-db:
      - mysql
    units:
      cinder/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:03:15Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:16:08Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: "4"
        public-address: sunsetvm01.oil
  glance:
    charm: local:trusty/glance-150
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:06:31Z
    relations:
      ceph:
      - ceph
      cluster:
      - glance
      identity-service:
      - keystone
      image-service:
      - cinder
      - nova-cloud-controller
      - nova-vmware
      shared-db:
      - mysql
    units:
      glance/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:06:31Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:15:53Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 4/lxc/1
        open-ports:
        - 9292/tcp
        public-address: 10.245.32.236
  heat:
    charm: local:trusty/heat-12
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:09:47Z
    relations:
      amqp:
      - rabbitmq-server
      identity-service:
      - keystone
      shared-db:
      - mysql
    units:
      heat/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:09:47Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:16:07Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 1/lxc/0
        open-ports:
        - 8000/tcp
        - 8004/tcp
        public-address: 10.245.43.69
  keystone:
    charm: local:trusty/keystone-0
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:09:26Z
    relations:
      cluster:
      - keystone
      identity-service:
      - ceilometer
      - cinder
      - glance
      - heat
      - neutron-api
      - nova-cloud-controller
      - openstack-dashboard
      shared-db:
      - mysql
    units:
      keystone/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:09:26Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:16:34Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 2/lxc/0
        public-address: 10.245.41.141
  mongodb:
    charm: local:trusty/mongodb-30
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:05:34Z
    relations:
      database:
      - ceilometer
      replica-set:
      - mongodb
    units:
      mongodb/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:05:34Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:12:54Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 3/lxc/0
        open-ports:
        - 27017/tcp
        - 27019/tcp
        - 27021/tcp
        - 28017/tcp
        public-address: 10.245.32.235
  mysql:
    charm: local:trusty/mysql-326
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:03:08Z
    relations:
      cluster:
      - mysql
      shared-db:
      - cinder
      - glance
      - heat
      - keystone
      - neutron-api
      - neutron-gateway
      - nova-cloud-controller
    units:
      mysql/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:03:08Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:15:22Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 4/lxc/0
        public-address: 10.245.40.205
  neutron-api:
    charm: local:trusty/neutron-api-0
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:09:33Z
    relations:
      amqp:
      - rabbitmq-server
      cluster:
      - neutron-api
      identity-service:
      - keystone
      neutron-api:
      - nova-cloud-controller
      neutron-plugin-api:
      - neutron-openvswitch
      shared-db:
      - mysql
    units:
      neutron-api/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:09:33Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:16:24Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 1/lxc/1
        open-ports:
        - 9696/tcp
        public-address: 10.245.47.123
  neutron-gateway:
    charm: local:trusty/neutron-gateway-64
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:02:36Z
    relations:
      amqp:
      - rabbitmq-server
      cluster:
      - neutron-gateway
      juju-info:
      - nvp-transport-node
      quantum-network-service:
      - nova-cloud-controller
      shared-db:
      - mysql
    units:
      neutron-gateway/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:02:36Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:17:04Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: "1"
        public-address: hayward-33.oil
        subordinates:
          nvp-transport-node/0:
            workload-status:
              current: unknown
              since: 20 Oct 2015 17:14:33Z
            agent-status:
              current: idle
              since: 20 Oct 2015 17:14:50Z
              version: 1.24.6
            agent-state: started
            agent-version: 1.24.6
            upgrading-from: local:trusty/nsx-transport-node-2
            public-address: hayward-33.oil
  neutron-openvswitch:
    charm: local:trusty/neutron-openvswitch-0
    exposed: false
    service-status: {}
    relations:
      amqp:
      - rabbitmq-server
      neutron-plugin-api:
      - neutron-api
  nova-cloud-controller:
    charm: local:trusty/nova-cloud-controller-501
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:05:46Z
    relations:
      amqp:
      - rabbitmq-server
      cinder-volume-service:
      - cinder
      cloud-compute:
      - nova-vmware
      cluster:
      - nova-cloud-controller
      identity-service:
      - keystone
      image-service:
      - glance
      neutron-api:
      - neutron-api
      quantum-network-service:
      - neutron-gateway
      shared-db:
      - mysql
    units:
      nova-cloud-controller/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:05:46Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:17:00Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: "2"
        open-ports:
        - 3333/tcp
        - 8773/tcp
        - 8774/tcp
        - 9696/tcp
        public-address: hayward-35.oil
  nova-vmware:
    charm: local:trusty/nova-compute-vmware-132
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:11:16Z
    relations:
      amqp:
      - rabbitmq-server
      cloud-compute:
      - nova-cloud-controller
      image-service:
      - glance
    units:
      nova-vmware/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:11:16Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:17:06Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: "3"
        public-address: hayward-39.oil
  nvp-transport-node:
    charm: local:trusty/nsx-transport-node-2
    exposed: false
    service-status: {}
    relations:
      juju-info:
      - neutron-gateway
    subordinate-to:
    - neutron-gateway
  openstack-dashboard:
    charm: local:trusty/openstack-dashboard-32
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:08:36Z
    relations:
      cluster:
      - openstack-dashboard
      identity-service:
      - keystone
    units:
      openstack-dashboard/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:08:36Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:15:54Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 2/lxc/1
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: 10.245.41.156
  rabbitmq-server:
    charm: local:trusty/rabbitmq-server-150
    exposed: false
    service-status:
      current: unknown
      since: 20 Oct 2015 17:09:49Z
    relations:
      amqp:
      - ceilometer
      - cinder
      - heat
      - neutron-api
      - neutron-gateway
      - neutron-openvswitch
      - nova-cloud-controller
      - nova-vmware
      cluster:
      - rabbitmq-server
    units:
      rabbitmq-server/0:
        workload-status:
          current: unknown
          since: 20 Oct 2015 17:09:49Z
        agent-status:
          current: idle
          since: 20 Oct 2015 17:14:00Z
          version: 1.24.6
        agent-state: started
        agent-version: 1.24.6
        machine: 3/lxc/2
        open-ports:
        - 5672/tcp
        public-address: 10.245.41.159
networks:
  oil-mgmt:
    provider-id: oil-mgmt
    cidr: 10.245.0.0/18

[Larry Michel (lmic)]

It doesn't look like setting integration-bridge to anything in the nova.conf file works. So, if using a different name br-int then integration-bridge must be specified to avoid this error which seemed the original issue:

2015-11-03 21:27:43.707 4871 ERROR nova.compute.manager [-] [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] Instance failed to spawn
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] Traceback (most recent call last):
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2267, in _build_resources
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] yield resources
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2137, in _build_and_run_instance
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] block_device_info=block_device_info)
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 442, in spawn
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] admin_password, network_info, block_device_info)
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 446, in spawn
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] network_info)
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 239, in build_virtual_machine
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] network_info)
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 182, in get_vif_info
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] is_neutron, vif))
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 168, in get_vif_dict
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] ref = get_network_ref(session, cluster, vif, is_neutron)
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 157, in get_network_ref
2015-11-03 21:27:43.707 4871 TRACE nova.compute.manager [instance: cf93dde9-cc1f-44e3-a2c3-b2fedfbdbc43] network_ref = get_neutron_network(session, network_...

[Larry Michel (lmic)]

attached are juju and nova log files as well as nova conf files.