OpenStack Compute (nova)

[Rocky] fail to get VNC console

Bug #1792674 reported by end-user
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned

Bug Description

I'm runnig Rocky verion on Centos7.5-1804.

Everything works fine, except VNC console. But when I use VNCviewer( the 3rd software)

to view Instance, type 172.16.237.10:5900, it's working.

############# Begin of nova.conf #

[DEFAULT]

enabled_apis = osapi_compute,metadata

transport_url = rabbit://openstack:ra41P2aebs9@controller-01

my_ip = 172.16.237.10

firewall_driver = nova.virt.firewall.NoopFirewallDriver

[api_database]

connection = mysql+pymysql://nova:nova@controller-01/nova_api

[database]

connection = mysql+pymysql://nova:nova@controller-01/nova

[placement_database]

connection = mysql+pymysql://nova:nova@controller-01/placement

[api]

auth_strategy = keystone

[keystone_authtoken]
auth_url = http://controller-01:5000/v3
www_authenticate_uri = http://controller-01:5000/v3
memcached_servers = controller-01:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = nova

[vnc]
enabled = true
server_listen = 0.0.0.0
server_proxyclient_address = 172.16.237.10
novncproxy_base_url = http://172.16.237.10:6080/vnc_auto.html

[glance]
api_servers = http://controller-01:9292

[oslo_concurrency]
lock_path = /var/lib/nova/tmp

[placement]
region_name = RegionOne
project_domain_name = default
project_name = service
auth_type = password
user_domain_name = default
auth_url = http://controller-01:5000/v3
username = placement
password = placement

[scheduler]

discover_hosts_in_cells_interval = 60

[libvirt]
virt_type = qemu

[neutron]
auth_url = http://controller-01:5000/v3
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = neutron
password = neutron
service_metadata_proxy = true

### end of nova.conf ####

How to reproduce

`nova --debug get-vnc-console <Insance_UUID> novnc

this comes for nova-api.log

2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi [req-ed492726-752f-4348-9bab-20aa28a5be8d 267048d79f3b4f5e8a67010540b2c673 68fdc324c1984ca1b152ccdde4dfbb86 - default default] Unexpected exception in API method: MessagingTimeout: Timed out waiting for a reply to message ID 89caa318d07b4cc4bba21a18ff790338
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/remote_consoles.py", line 168, in create
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi output = handler(context, instance, console_type)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 196, in wrapped
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 186, in inner
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3723, in get_vnc_console
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi access_url=connect_info['access_url'])
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/consoleauth/rpcapi.py", line 93, in authorize_console
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi return cctxt.call(ctxt, 'authorize_console', **msg_args)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi retry=self.retry)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi retry=retry)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi call_monitor_timeout, retry=retry)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi call_monitor_timeout)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi message = self.waiters.get(msg_id, timeout=timeout)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi 'to message ID %s' % msg_id)
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi MessagingTimeout: Timed out waiting for a reply to message ID 89caa318d07b4cc4bba21a18ff790338
2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi
2018-09-15 14:22:05.228 6338 INFO nova.api.openstack.wsgi [req-ed492726-752f-4348-9bab-20aa28a5be8d 267048d79f3b4f5e8a67010540b2c673 68fdc324c1984ca1b152ccdde4dfbb86 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'>
2018-09-15 14:22:05.229 6338 INFO nova.osapi_compute.wsgi.server [req-ed492726-752f-4348-9bab-20aa28a5be8d 267048d79f3b4f5e8a67010540b2c673 68fdc324c1984ca1b152ccdde4dfbb86 - default default] 172.16.237.10 "POST /v2.1/servers/1065c746-24f3-4050-8c00-d6e4e6a00984/remote-consoles HTTP/1.1" status: 500 len: 651 time: 60.1198242

See original description
Tags: nova vnc
Revision history for this message
end-user (hunter009) wrote :

This is part of debug log

# nova --debug get-vnc-console <INSTANCE_NAME> novnc

..... ignore part of them
DEBUG (session:448) REQ: curl -g -i -X POST http://172.16.237.10:8774/v2.1/servers/45631db8-654a-4e79-af3c-b891f2130dec/remote-consoles -H "Accept: application/json" -H "Content-Type: application/json" -H "OpenStack-API-Version: compute 2.65" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}45ed5ea4edfbca4df143d10bba4808b2be85383c" -H "X-OpenStack-Nova-API-Version: 2.65" -d '{"remote_console": {"type": "novnc", "protocol": "vnc"}}'

DEBUG (session:511) RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'oslo_messaging.exceptions.MessagingTimeout'>", "code": 500}}
DEBUG (session:844) POST call to compute for http://172.16.64.12:8774/v2.1/servers/45631db8-654a-4e79-af3c-b891f2130dec/remote-consoles used request id req-14cf5912-2d65-441e-9697-339df0bce340
DEBUG (shell:832) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-14cf5912-2d65-441e-9697-339df0bce340)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 830, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 752, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 2562, in do_get_vnc_console
    data = server.get_vnc_console(args.console_type)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 107, in get_vnc_console
    return self.manager.get_vnc_console(self, console_type)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 876, in get_vnc_console
    return self.get_console_url(server, console_type)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 393, in substitution
    return methods[-1].func(obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 964, in get_console_url
    resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 334, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-14cf5912-2d65-441e-9697-339df0bce340)

description: updated
description: updated
Revision history for this message
end-user (hunter009) wrote :

This must be a bug of rocky, I have tried to install for serveral times. Everytime it coms out the same result.

Revision history for this message
Aleks (iffster) wrote :

Same here.
Also copy of this bug https://bugs.launchpad.net/nova/+bug/1791940

Aleks (iffster)
Changed in nova:
status: New → Confirmed
Revision history for this message
end-user (hunter009) wrote :

Seems that I'm not the only one who has caught this bug.

Revision history for this message
Sam Morrison (sorrison) wrote :

A messaging timeout usually indicates that the service that is meant to be consuming the messages is not working.

Do other operations like "nova console-log" work?
Are there queues with messages that aren't being consumed (check rabbit admin interface)?
Also is the nova-consoleauth service running ok?

Revision history for this message
end-user (hunter009) wrote : Re: [Bug 1792674] Re: [Rocky] fail to get VNC console
Download full text (8.0 KiB)

nova console-log works, which pops up logs of instances starting.

While do you mean openstack-nova-novncproxy.service ? yes, it's running
and listening on port 6080

There is no nova-consoleauth service running on Centos7.

I check every queue, there isn't any message.

Thanks.

From: Sam Morrison <email address hidden>
To: <email address hidden>,
Date: 2018/09/19 09:20
Subject: [Bug 1792674] Re: [Rocky] fail to get VNC console
Sent by: <email address hidden>

A messaging timeout usually indicates that the service that is meant to be
consuming the messages is not working.

Do other operations like "nova console-log" work?
Are there queues with messages that aren't being consumed (check rabbit
admin interface)?
Also is the nova-consoleauth service running ok?

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1792674

Title:
  [Rocky] fail to get VNC console

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  I'm runnig Rocky verion on Centos7.5-1804.

  Everything works fine, except VNC console. But when I use VNCviewer(
  the 3rd software)

  to view Instance, type 172.16.237.10:5900, it's working.

  ############# Begin of nova.conf #

  [DEFAULT]

  enabled_apis = osapi_compute,metadata

  transport_url = rabbit://openstack:ra41P2aebs9@controller-01

  my_ip = 172.16.237.10

  firewall_driver = nova.virt.firewall.NoopFirewallDriver

  [api_database]

  connection = mysql+pymysql://nova:nova@controller-01/nova_api

  [database]

  connection = mysql+pymysql://nova:nova@controller-01/nova

  [placement_database]

  connection = mysql+pymysql://nova:nova@controller-01/placement

  [api]

  auth_strategy = keystone

  [keystone_authtoken]
  auth_url = http://controller-01:5000/v3
  www_authenticate_uri = http://controller-01:5000/v3
  memcached_servers = controller-01:11211
  auth_type = password
  project_domain_name = default
  user_domain_name = default
  project_name = service
  username = nova
  password = nova

  [vnc]
  enabled = true
  server_listen = 0.0.0.0
  server_proxyclient_address = 172.16.237.10
  novncproxy_base_url = http://172.16.237.10:6080/vnc_auto.html

  [glance]
  api_servers = http://controller-01:9292

  [oslo_concurrency]
  lock_path = /var/lib/nova/tmp

  [placement]
  region_name = RegionOne
  project_domain_name = default
  project_name = service
  auth_type = password
  user_domain_name = default
  auth_url = http://controller-01:5000/v3
  username = placement
  password = placement

  [scheduler]

  discover_hosts_in_cells_interval = 60

  [libvirt]
  virt_type = qemu

  [neutron]
  auth_url = http://controller-01:5000/v3
  auth_type = password
  project_domain_name = default
  user_domain_name = default
  region_name = RegionOne
  project_name = service
  username = neutron
  password = neutron
  service_metadata_proxy = true

  ### end of nova.conf ####

  How to reproduce

  `nova --debug get-vnc-console <Insance_UUID> novnc

  this comes for nova-api.log

  2018-09-15 14:22:05.226 6338 ERROR nova.api.openstack.wsgi
[req-ed492726-752f-4348-9bab-20aa28a5be8d 267048d79f3b4f5e8a6...

Read more...

Revision history for this message
Sam Morrison (sorrison) wrote :

Sorry I don't run centos but you will need the consoleauth service running to use vnc-console.

Maybe this is a packaging issue with centos or maybe you need to install a new package in centos that include nova-consoleauth

Revision history for this message
end-user (hunter009) wrote :

@SAM,

Thanks a lot, it saves me tons of time. It works now.

The root cause is that nova-consoleauth does not start up.

Sorry for any inconvenient.

We can close this ticket now.

Revision history for this message
end-user (hunter009) wrote :

By the way, installation docs of Centos on compute node doesn't mention to start openstack-nova-consoleauth.service

I'll file a bug for it.

Changed in nova:
status: Confirmed → Invalid
Revision history for this message
Sam Morrison (sorrison) wrote :

No worries, just note that the consoleauth service runs centrally as opposed to running one per compute node.

Revision history for this message
Aleks (iffster) wrote :

Yeah, thank you very much, it's working now.
I can confirm that service enable and start for "openstack-nova-consoleauth" is gone from all installation documents in "rocky" release (from ubuntu/sles/centos-rhel)

https://docs.openstack.org/nova/rocky/install/controller-install-rdo.html
https://docs.openstack.org/nova/rocky/install/controller-install-obs.html
https://docs.openstack.org/nova/rocky/install/controller-install-ubuntu.html

Revision history for this message
Matt Riedemann (mriedem) wrote :

Installing nova-consoleauth was removed from the docs in Rocky as part of this change:

https://review.openstack.org/#/c/565367/

Need to talk to melwitt about this.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Should have been fixed as part of this:

https://review.openstack.org/#/q/4f01f4ff88de571218a36ba7c4e998296a7b52a4

Just need to re-add those docs it looks like, along with a deprecation warning in the install guide.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.