OpenStack Compute (nova)

Nova - Novncproxy - Console opens to an incorrect VM

Bug #1593672 reported by Arpad Hajdu
28
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Unassigned

Bug Description

Description
===========
After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance.
We tried to shut off all the instances and start them again, but the issue is still exists.

Steps to reproduce
==================
At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...

But as we remember the following happened:
- Deployed 33 Win2k3 instances with "Launch instance and Count 33"
- Deployed 33 Win2k8 instances with "Launch instance and count 33"
- Deployed 33 Win2k12 instances with heat template -> We noticed the problem
- Stack deleted with all the instances
- Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there

root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Description
===========
After deploying several instances of Virtual machines (some of them using heat) some instances console are pointing to a different instance.
We tried to shut off all the instances and start them again, but the issue is still exists.

Steps to reproduce
==================
At the moment I don't have any information since we are using the instances I cannot delete all of them and re-create them...

But as we remember the following happened:
- Deployed 33 Win2k3 instances with "Launch instance and Count 33"
- Deployed 33 Win2k8 instances with "Launch instance and count 33"
- Deployed 33 Win2k12 instances with heat template -> We noticed the problem
- Stack deleted with all the instances
- Deployed 33 Win2k12 instances with "Launch instance and count 33" -> The problem is still there

Steps:
- Open OPenstack dashboard
- Click the instance name
- Change to console tab
- Open console
or
- nova get-vnc-console Windows_2k12R2_perf-26

Expected result
===============
We have a console to an another instance

Environment
===========
Control node:
root@openstack1:/etc/nova# dpkg -l | grep nova
ii nova-api 2:13.0.0-0ubuntu2 all OpenStack Compute - API frontend
ii nova-cells 2:13.0.0-0ubuntu2 all Openstack Compute - cells
ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7

Compute node1:
root@openstackcompute:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7

Compute node2:
root@openstackcompute2:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7

Compute node3:
root@openstackcompute3:~# dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7

Hypervisors:
KVM + Libvirt on all compute nodes

Storage:
CEPH, iSCSI and FC using Netapp

CEPH version - the same on all storage nodes
ii ceph 10.2.0-0ubuntu0.16.04.1 amd64 distributed storage and file system
ii ceph-common 10.2.0-0ubuntu0.16.04.1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-deploy 1.5.32-0ubuntu1 all Deployment and configuration of Ceph.
ii ceph-mds 10.2.0-0ubuntu0.16.04.1 amd64 metadata server for the ceph distributed file system
ii libcephfs1 10.2.0-0ubuntu0.16.04.1 amd64 Ceph distributed file system client library
ii python-cephfs 10.2.0-0ubuntu0.16.04.1 amd64 Python libraries for the Ceph libcephfs library

Networking
Neutron networking with Openvswitch

----
Example process flow:

Instance information:
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | openstackcompute3 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | openstackcompute3 |
| OS-EXT-SRV-ATTR:instance_name | instance-000000f4 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-06-16T12:10:06.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | net-10.10.128=10.10.128.19 |
| config_drive | |
| created | 2016-06-16T12:09:09Z |
| flavor | CCL-Win2k12 (b383ddeb-f52b-4554-94b6-49981bada108) |
| hostId | c340bdd941f659cce547150bf407432536d9a856192a100ec3a2ebec |
| id | fe3f9980-76c5-4845-baaf-589d129ddfbb |
| image | Windows 2012 R2 X64 (86612cca-ce9f-4920-997d-6e81d9ea0068) |
| key_name | CCLTests |
| name | Windows_2k12R2_perf-2 |
| os-extended-volumes:volumes_attached | [{u'id': u'd8a9ce45-dac0-42de-a496-db133f1d2a7f'}, {u'id': u'0050819f-3fe7-4ec8-b4d0-2a939ccc533b'}, {u'id': u'343d0841-501d-4a38-a5a5-31f86a7d5053'}] |
| progress | 0 |
| project_id | 6d0fa2c2b45a4f6c8aac185b4592d44f |
| properties | |
| security_groups | [{u'name': u'CCL'}] |
| status | ACTIVE |
| updated | 2016-06-17T08:35:21Z |
| user_id | 75f412ba1601457a8415519bbdd3a932 |
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+

Get the console link:
root@openstack1:/etc/nova# nova get-vnc-console Windows_2k12R2_perf-2 novnc
+-------+-----------------------------------------------------------------------------------+
| Type | Url |
+-------+-----------------------------------------------------------------------------------+
| novnc | http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3 |
+-------+-----------------------------------------------------------------------------------+

Consoleauth log:
2016-06-17 11:23:37.555 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Token: a33bbd35-1e39-4887-9eed-c86013e598d3 failed to save into memcached.
2016-06-17 11:23:37.557 20844 WARNING nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Instance: fe3f9980-76c5-4845-baaf-589d129ddfbb failed to save into memcached
2016-06-17 11:23:37.558 20844 INFO nova.consoleauth.manager [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Received Token: a33bbd35-1e39-4887-9eed-c86013e598d3, {'instance_uuid': u'fe3f9980-76c5-4845-baaf-589d129ddfbb', 'access_url': u'http://192.168.5.70:6080/vnc_auto.html?token=a33bbd35-1e39-4887-9eed-c86013e598d3', 'token': u'a33bbd35-1e39-4887-9eed-c86013e598d3', 'last_activity_at': 1466159017.555781, 'internal_access_path': None, 'console_type': u'novnc', 'host': u'192.168.5.56', 'port': u'5914'}

If I check the IP of the host it is 192.168.5.56 which belongs to openstackcompute2 and the instance is running on openstackcompute3

2016-06-17 11:23:35.834 20775 DEBUG keystoneauth.session [-] RESP: [200] Content-Length: 7168 X-Subject-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 17 Jun 2016 10:23:35 GMT x-openstack-request-id: req-d525aca5-8b85-47e2-9bc6-0fbddfa7fa24 Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"token": {"methods": ["password"], "roles": [{"id": "2588e57d4cb44ad3a95f0f8aaaffcf94", "name": "admin"}], "expires_at": "2016-06-17T11:23:35.295341Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "name": "admin"}, "catalog": "<removed>", "user": {"domain": {"id": "default", "name": "Default"}, "id": "75f412ba1601457a8415519bbdd3a932", "name": "admin"}, "audit_ids": ["uUYFFZ_PTte2PvD09pFusg"], "issued_at": "2016-06-17T10:23:35.000000Z"}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:35.841 20775 INFO nova.osapi_compute.wsgi.server [req-35805c67-a652-4714-bec4-ac39d1ba82e4 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f HTTP/1.1" status: 404 len: 264 time: 0.5082760
2016-06-17 11:23:35.854 20775 DEBUG nova.api.openstack.wsgi [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7f11ccf1c290>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:35.856 20775 INFO nova.osapi_compute.wsgi.server [req-f98b2a02-a979-497e-aea7-8d8e48837cce 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/ HTTP/1.1" status: 200 len: 665 time: 0.0106051
2016-06-17 11:23:36.283 20775 DEBUG nova.api.openstack.wsgi [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.index of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:36.286 20775 DEBUG nova.compute.api [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Searching by: {'deleted': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', u'name': u'Windows_2k12R2_perf-2'} get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:2055
2016-06-17 11:23:36.402 20775 INFO nova.osapi_compute.wsgi.server [req-9c97f92b-1e82-4270-8f34-5ba033dc5898 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers?name=Windows_2k12R2_perf-2 HTTP/1.1" status: 200 len: 4528 time: 0.1263618
2016-06-17 11:23:36.414 20775 DEBUG nova.api.openstack.wsgi [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x7f11ccfbe410>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699
2016-06-17 11:23:36.416 20775 DEBUG nova.compute.api [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
2016-06-17 11:23:36.543 20775 DEBUG nova.policy [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Policy check for os_compute_api:os-hide-server-addresses failed with credentials {'domain': None, 'project_name': u'admin', 'project_domain': None, 'timestamp': '2016-06-17T10:23:36.409062', 'remote_address': '192.168.5.70', 'quota_class': None, 'resource_uuid': None, 'is_admin': True, 'user': u'75f412ba1601457a8415519bbdd3a932', 'service_catalog': [{u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v2/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volumev2', u'name': u'cinderv2'}, {u'endpoints': [{u'adminURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'region': u'CCL', u'internalURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f', u'publicURL': u'http://openstackblock.ccl-dev.net:8776/v1/6d0fa2c2b45a4f6c8aac185b4592d44f'}], u'type': u'volume', u'name': u'cinder'}], 'tenant': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'read_only': False, 'project_id': u'6d0fa2c2b45a4f6c8aac185b4592d44f', 'user_id': u'75f412ba1601457a8415519bbdd3a932', 'show_deleted': False, 'roles': [u'admin'], 'user_identity': '75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -', 'read_deleted': 'no', 'request_id': 'req-2bf0aad5-ccd0-4727-9a2b-b276656c5632', 'instance_lock_checked': False, 'user_domain': None, 'user_name': u'admin'} enforce /usr/lib/python2.7/dist-packages/nova/policy.py:107
2016-06-17 11:23:36.557 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/ports.json?device_id=fe3f9980-76c5-4845-baaf-589d129ddfbb -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-06-17 11:23:37.269 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 887 X-Openstack-Request-Id: req-445709ad-1f43-470e-81c7-d3dbe679cd74
RESP BODY: {"ports": [{"status": "ACTIVE", "binding:host_id": "openstackcompute3", "description": "", "allowed_address_pairs": [], "extra_dhcp_opts": [], "updated_at": "2016-06-17T08:35:43", "device_owner": "compute:nova", "port_security_enabled": true, "binding:profile": {}, "fixed_ips": [{"subnet_id": "ab18e9de-f27b-4c57-a54e-4e4631a79ec9", "ip_address": "10.10.128.19"}], "id": "55a4f4ee-274b-4d2b-82c0-8c3f20050fd0", "security_groups": ["3b859d0c-3641-48f1-bbf6-3c52fb321bf0"], "device_id": "fe3f9980-76c5-4845-baaf-589d129ddfbb", "name": "", "admin_state_up": true, "network_id": "be93253f-ffa3-40e8-a8a9-93976da0cf5c", "dns_name": null, "binding:vif_details": {"port_filter": true, "ovs_hybrid_plug": false}, "binding:vnic_type": "normal", "binding:vif_type": "ovs", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "mac_address": "fa:16:3e:64:17:71", "created_at": "2016-06-16T12:09:53"}]}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:37.272 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] REQ: curl -g -i -X GET http://openstacknetwork.ccl-dev.net:9696/v2.0/security-groups.json?id=3b859d0c-3641-48f1-bbf6-3c52fb321bf0 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}ec9f74f867943effc91e2ba7fca8c660bb2c6396" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248
2016-06-17 11:23:37.342 20775 DEBUG keystoneauth.session [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] RESP: [200] Date: Fri, 17 Jun 2016 10:23:38 GMT Connection: keep-alive Content-Type: application/json; charset=UTF-8 Content-Length: 1900 X-Openstack-Request-Id: req-e4a09d2e-fc4a-4d43-8542-a00a56b73a88
RESP BODY: {"security_groups": [{"tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "description": "", "id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "security_group_rules": [{"direction": "ingress", "protocol": "tcp", "description": null, "port_range_max": 22, "id": "1345c82e-34e3-443c-b39d-c38a06365529", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 22, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "49d12fbd-d580-4214-8f81-3cdae3638e3a", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}, {"direction": "egress", "protocol": null, "description": null, "port_range_max": null, "id": "671b4a2f-c6a9-4450-b450-f56e6eaa356b", "remote_group_id": null, "remote_ip_prefix": null, "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv6"}, {"direction": "ingress", "protocol": "tcp", "description": "", "port_range_max": 5599, "id": "7d3d2a07-2675-446a-8182-8592dde6f64f", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": 5500, "ethertype": "IPv4"}, {"direction": "ingress", "protocol": "icmp", "description": null, "port_range_max": null, "id": "bc1b57f4-c75f-4851-99c0-7b9b918897f5", "remote_group_id": null, "remote_ip_prefix": "0.0.0.0/0", "security_group_id": "3b859d0c-3641-48f1-bbf6-3c52fb321bf0", "tenant_id": "6d0fa2c2b45a4f6c8aac185b4592d44f", "port_range_min": null, "ethertype": "IPv4"}], "name": "CCL"}]}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-06-17 11:23:37.347 20775 DEBUG nova.objects.instance [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Lazy-loading 'services' on Instance uuid fe3f9980-76c5-4845-baaf-589d129ddfbb obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:895
2016-06-17 11:23:37.430 20775 INFO nova.osapi_compute.wsgi.server [req-2bf0aad5-ccd0-4727-9a2b-b276656c5632 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "GET /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb HTTP/1.1" status: 200 len: 2908 time: 1.0243011
2016-06-17 11:23:37.440 20775 DEBUG nova.api.openstack.wsgi [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] Action: 'create', calling method: <function version_select at 0x7f11cc48b938>, body: {"remote_console": {"type": "novnc", "protocol": "vnc"}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:696
2016-06-17 11:23:37.443 20775 DEBUG nova.compute.api [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] [instance: fe3f9980-76c5-4845-baaf-589d129ddfbb] Fetching instance by UUID get /usr/lib/python2.7/dist-packages/nova/compute/api.py:2008
2016-06-17 11:23:37.564 20775 INFO nova.osapi_compute.wsgi.server [req-8fb95c6e-e6f7-4105-8e13-6cff644eb81c 75f412ba1601457a8415519bbdd3a932 6d0fa2c2b45a4f6c8aac185b4592d44f - - -] 192.168.5.70 "POST /v2.1/6d0fa2c2b45a4f6c8aac185b4592d44f/servers/fe3f9980-76c5-4845-baaf-589d129ddfbb/remote-consoles HTTP/1.1" status: 200 len: 417 time: 0.1293130

If you need any other information I can provide.

Revision history for this message
Arpad Hajdu (hajduarpad-gmail) wrote :
Revision history for this message
Arpad Hajdu (hajduarpad-gmail) wrote :

Added sosreport from control node.
The file edited and SSL parts removed.

Markus Zoeller (markus_z) (mzoeller)
tags: added: console
Revision history for this message
Augustina Ragwitz (auggy) wrote :

Thanks for filing a bug! I think the next step here is for us to attempt to reproduce the basic behavior to make sure the simplest use case works. First, attempt to reproduce launching multiple instances and then attempt to connect to one with the console on a single node devstack. Second, attempt the same on a multinode devstack (at least 2 nodes) since the issue is reported on a 3 node setup. Once that has been done we'll follow up with you.

Revision history for this message
MarkMielke (mark-mielke) wrote :

From the evidence provided, I don't think I can say whether your issue is a match to an issue I had or not, but I'm just going to share, and see if rings any bells for you...

In a prototype instance that we were deploying by hand (first mistake, perhaps!), I failed to set my_ip correctly on one of the compute nodes. As a result, many of the consoles for virtual machines on that compute node, were reporting consoles from the machine that my_ip specified, In my case, it was my_ip that was wrong, although I expect that if you hard code the IP address in other more specific configurations, such as vncserver_proxyclient_address or others, that the symptoms would be the same.

Sean Dague (sdague)
Changed in nova:
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
Thiago Martins (martinx) wrote :

This is happening since Juno! I'm seeing this problem on Ubuntu 16.04 with Ocata, from Cloud Archive.

My console is SPICE, not VNC.

Revision history for this message
Sean Dague (sdague) wrote :

Automatically discovered version mitaka in description. If this is incorrect, please update the description to include 'nova version: ...'

tags: added: openstack-version.mitaka
Francesco Pantano (fmount)
Changed in nova:
status: Confirmed → Fix Released
Revision history for this message
Francesco Pantano (fmount) wrote :

Sorry my comment above is incorrect: I've experienced this bug in Mitaka release...anyone is working on this bug?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.