VNC URL of instance unavailable in CLI

[Ruoyu Zhang]

I use heat template to build an autoscaling group with 'OS::Heat::AutoScalingGroup' and 'OS::Nova::Server' and it works fine. I can see instance running both by CLI and dashboard. However, I can only get to the console by dashboard directly. While using command 'nova get-vnc-console instance_ID novnc', I got an error:'ERROR (NotFound): The resource could not be found. (HTTP 404) (Request-ID: req-6f260624-56ad-45fd-aa21-f86fb2c541d1)' instead of its URL.

[Maru Newby (maru)]

I'm experiencing the same issue without heat being involved.

2015-09-29 23:30:20.498 DEBUG nova.api.openstack.wsgi [req-284db9db-c5a5-45b6-b458-2f3b4639deb2 demo demo] Action: 'action', calling method: <function version_select at 0x60e05f0>, body: {"os-getVNCConsole": {"type": "novnc"}} from (pid=15025) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:789
2015-09-29 23:30:20.498 DEBUG nova.api.openstack.wsgi [req-284db9db-c5a5-45b6-b458-2f3b4639deb2 demo demo] Returning 404 to user: The resource could not be found. from (pid=15025) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1175
2015-09-29 23:30:20.499 INFO nova.osapi_compute.wsgi.server [req-284db9db-c5a5-45b6-b458-2f3b4639deb2 demo demo] 192.168.232.34 "POST /v2.1/d83d5dfb3dda427aa4bea11c90996a84/servers/3aab35bc-8616-4106-9682-a52a688f24f6/action HTTP/1.1" status: 404 len: 368 time: 0.0038269

[Markus Zoeller (markus_z) (mzoeller)]

Could you please provide the following information:
1. Exact version of Nova/OpenStack you are running. If this is from a distro please provide `dpkg -l | grep nova` or `rpm -ql | grep nova`. If this is from git, please provide `git log -1`.
2. Relevant log files. Most probably from the services "nova-compute (n-cpu)", "nova-consoleauth (n-cauth)" and "nova-conductor (n-cond)".

Without these informations we are not able to do a proper bug triage.

After you provided this information, please set this bug status back to "new".

[Ruoyu Zhang (ufo172849z)]

Here is the information:
1. from 'git log -1' running in /opt/stack/nova (I install openstack using devstack, all from the latest version from git)

commit 7b0d0fe0c11340fe13dffebe0e049f951011402b
Merge: f587a78 c56bbc6
Author: Jenkins <email address hidden>
Date: Mon Sep 28 17:49:10 2015 +0000

    Merge "libvirt: Split out resize_image logic from create_image"

2. In 'n-cpu.log' from /opt/stack/logs

2015-09-30 04:41:38.157 DEBUG oslo_service.periodic_task [req-ce40856e-e042-41b3-8c5f-7b9030db99a5 None None] Running periodic task ComputeManager.update_available_resource from (pid=16597) run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-09-30 04:41:38.195 INFO nova.compute.resource_tracker [req-ce40856e-e042-41b3-8c5f-7b9030db99a5 None None] Auditing locally available compute resources for node ubuntu
2015-09-30 04:41:38.283 DEBUG oslo_concurrency.processutils [req-ce40856e-e042-41b3-8c5f-7b9030db99a5 None None] Running cmd (subprocess): env LC_ALL=C LANG=C qemu-img info /opt/stack/data/nova/instances/9ebc4358-a052-40ca-97f2-75f8686fc73f/disk from (pid=16597) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:230
2015-09-30 04:41:38.314 DEBUG oslo_concurrency.processutils [req-ce40856e-e042-41b3-8c5f-7b9030db99a5 None None] CMD "env LC_ALL=C LANG=C qemu-img info /opt/stack/data/nova/instances/9ebc4358-a052-40ca-97f2-75f8686fc73f/disk" returned: 0 in 0.030s from (pid=16597) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:260
2015-09-30 04:41:38.315 DEBUG oslo_concurrency.processutils [req-ce40856e-e042-41b3-8c5f-7b9030db99a5 None None] Running cmd (subprocess): env LC_ALL=C LANG=C qemu-img info /opt/stack/data/nova/instances/9ebc4358-a052-40ca-97f2-75f8686fc73f/disk from (pid=16597) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:230
2015-09-30 04:41:38.348 DEBUG oslo_concurrency.processutils [req-ce40856e-e042-41b3-8c5f-7b9030db99a5 None None] CMD "env LC_ALL=C LANG=C qemu-img info /opt/stack/data/nova/instances/9ebc4358-a052-40ca-97f2-75f8686fc73f/disk" returned: 0 in 0.032s from (pid=16597) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:260
2015-09-30 04:41:38.538 DEBUG nova.compute.resource_tracker [req-ce40856e-e042-41b3-8c5f-7b9030db99a5 None None] Hypervisor: free VCPUs: 7 from (pid=16597) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:580
2015-09-30 04:41:38.539 DEBUG nova.compute.resource_tracker [req-ce40856e-e042-41b3-8c5f-7b9030db99a5 None None] Hypervisor: assignable PCI devices: [{"dev_id": "pci_0000_00_00_0", "product_id": "1237", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_1237", "address": "0000:00:00.0"}, {"dev_id": "pci_0000_00_01_0", "product_id": "7000", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_7000", "address": "0000:00:01.0"}, {"dev_id": "pci_0000_00_01_1", "product_id": "7111", "dev_type": "type-PCI", "numa_node": null, "vendor_id": "8086", "label": "label_8086_7111", "address": "0000:00:01.1"}, {"dev_id": "pci_0000_00_02...

[Markus Zoeller (markus_z) (mzoeller)]

@Ruoyu Zhang:

I don't see anything related to the REST API call for the action "get_vnc_console" [1]. This is the entrypoint which gets called by Horizon and the CLI. This call gets routed to the compute manager which will create a debug log "Getting vnc console" [2].

Please make a grep for the request you mentioned "req-6f260624-56ad-45fd-aa21-f86fb2c541d1" and for the uuid of the instance in the log files ("/opt/stack/logs").

References:
[1] http://developer.openstack.org/api-ref-compute-v2.1.html#getoutputvnc
[2] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4379

[Ruoyu Zhang (ufo172849z)]

I grep info of the request ID and the instance ID but I only get one message of calling "get_vnc_console" by request ID and no information about instance ID at the same time. Here's the information I got:

n-api.log.2015-09-28-181812:2015-09-28 19:27:05.279 DEBUG nova.api.openstack.wsgi [req-6f260624-56ad-45fd-aa21-f86fb2c541d1 admin admin] Action: 'action', calling method: <function version_select at 0x7fed0033cb90>, body: {"os-getVNCConsole": {"type": "novnc"}} from (pid=14311) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:789
n-api.log.2015-09-28-181812:2015-09-28 19:27:05.280 DEBUG nova.api.openstack.wsgi [req-6f260624-56ad-45fd-aa21-f86fb2c541d1 admin admin] Returning 404 to user: The resource could not be found. from (pid=14311) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1175
n-api.log.2015-09-28-181812:2015-09-28 19:27:05.292 INFO nova.osapi_compute.wsgi.server [req-6f260624-56ad-45fd-aa21-f86fb2c541d1 admin admin] 10.0.3.15 "POST /v2.1/7cb8f37d1b524a1dbd83679e188f53d5/servers/c32dd8dc-cabb-4e9e-974f-3b3b1580f84d/action HTTP/1.1" status: 404 len: 368 time: 0.0223050

[Markus Zoeller (markus_z) (mzoeller)]

This got discussed in the nova meeting today [1]. The change in the REST API in version 2.6 [2] seems to be not covered in the python-novaclient.

[1] http://eavesdrop.openstack.org/meetings/nova/2015/nova.2015-10-01-14.01.log.html#l-237
[2] https://github.com/openstack/nova/blob/master/nova/api/openstack/rest_api_version_history.rst#26

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/230024

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/230149

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230024
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=d045019f0f2d28f26730b3617f7d7b993506b6e8
Submitter: Jenkins
Branch: master

commit d045019f0f2d28f26730b3617f7d7b993506b6e8
Author: Matt Riedemann <email address hidden>
Date: Thu Oct 1 09:19:14 2015 -0700

    Set DEFAULT_OS_COMPUTE_API_VERSION to 2.5

    The client currently implements support for microversions 2.2, 2.4 and
    2.11.

    According the nova API microversion history:

    http://docs.openstack.org/developer/nova/api_microversion_history.html

    2.3 just returns extra attributes in the response so it's not a problem
    for the client. We should expose those at some point but it's not a
    breaking change.

    2.5 is a server-side only change since the client allows filtering
    servers by IPv6 address but the server wasn't honoring that until 2.5.
    There are no client side changes for that microversion.

    2.6 is the first backwards incompatible microversion since it replaces
    the old specific console APIs (RDP/serial/SPICE/VNC) with a new generic
    remote-consoles API. The client must be updated to handle this since
    requesting any microversion >=2.6 for consoles will fail right now. This
    is exposed on the CLI right now because the CLI defaults to the 2.latest
    microversion.

    So even though the client actually supports the 2.11 microversion, we
    shouldn't default to higher than 2.5 because we have gaps in handling
    2.6->2.10.

    This change fixes the default for the CLI by setting
    DEFAULT_OS_COMPUTE_API_VERSION=2.5.

    Partial-Bug: #1500688

    Change-Id: I52074f9a3e7faa6a7a51c3fa9766100acf25dee2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/230591

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230149
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=cce7cd32775bc9d77a5ad57971be5d7c36e3aee5
Submitter: Jenkins
Branch: master

commit cce7cd32775bc9d77a5ad57971be5d7c36e3aee5
Author: Mark Doffman <email address hidden>
Date: Thu Oct 1 14:46:13 2015 -0500

    Support the server side remote-console API changes

    In microversion 2.6 the remote-console API was changed.
    See http://docs.openstack.org/developer/nova/api_microversion_history.html
    for details of those changes.

    This adds support for the new remote-console API.

    Change-Id: I67098d6ce8c71f3ac0073628fd3b0aa23cd81918
    Closes-Bug: #1500688

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/230591
Reason: This caused other issues:

https://review.openstack.org/#/c/237850/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/230591
Reason: Squash commit is here:

https://review.openstack.org/#/c/239277/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/239277
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=dc4970728bae71f9745506190cc87b2edb36b0b8
Submitter: Jenkins
Branch: stable/liberty

commit dc4970728bae71f9745506190cc87b2edb36b0b8
Author: Matt Riedemann <email address hidden>
Date: Thu Oct 1 09:19:14 2015 -0700

    Set DEFAULT_OS_COMPUTE_API_VERSION to 2.5

    The client currently implements support for microversions 2.2, 2.4 and
    2.11.

    According the nova API microversion history:

    http://docs.openstack.org/developer/nova/api_microversion_history.html

    2.3 just returns extra attributes in the response so it's not a problem
    for the client. We should expose those at some point but it's not a
    breaking change.

    2.5 is a server-side only change since the client allows filtering
    servers by IPv6 address but the server wasn't honoring that until 2.5.
    There are no client side changes for that microversion.

    2.6 is the first backwards incompatible microversion since it replaces
    the old specific console APIs (RDP/serial/SPICE/VNC) with a new generic
    remote-consoles API. The client must be updated to handle this since
    requesting any microversion >=2.6 for consoles will fail right now. This
    is exposed on the CLI right now because the CLI defaults to the 2.latest
    microversion.

    So even though the client actually supports the 2.11 microversion, we
    shouldn't default to higher than 2.5 because we have gaps in handling
    2.6->2.10.

    This change fixes the default for the CLI by setting
    DEFAULT_OS_COMPUTE_API_VERSION=2.5.

    Partial-Bug: #1500688

    Change-Id: I52074f9a3e7faa6a7a51c3fa9766100acf25dee2
    (cherry picked from commit d045019f0f2d28f26730b3617f7d7b993506b6e8)

    ------------------------------------------------------------------------
    The first change introduced a regression which is fixed in the following
    squashed change to make the complete fix.
    ------------------------------------------------------------------------

    Correct usage of API_MAX_VERSION and DEFAULT_OS_COMPUTE_API_VERSION

    Commit d045019f0f2d28f26730b3617f7d7b993506b6e8 use the variables
    API_MAX_VERSION and DEFAULT_OS_COMPUTE_API_VERSION with wrong way.

    API_MAX_VERSION means the max version of client supported. This
    variable only be changed when client support the new version. And
    it's must bumped sequentially.

    DEFAULT_OS_COMPUTE_API_VERSION is default value of option
    '--os-compute-api-version', this value decided the default behaviour
    of nova client. The nova client CLI behaviour should be that client
    negotiate with server side to find out most recent version betweeyn
    client and server. So the value should be '2.latest'. And it won't
    be changed except we decided to change the default behaviour of nova
    client CLI.

    Change-Id: Iba9bfa136245bd2899c427ac0c231a30c00bd7f3
    Closes-Bug: #1508244
    (cherry picked from commit 28bf8ddb06a618bec280c9fbc21b4be6af2c1787)