[nova] Sometimes VMs are not accessible via console from horizon

Bug #1468522 reported by Eugene Nikanorov
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
MOS Nova
6.0.x
Invalid
High
Eugene Nikanorov

Bug Description

Sometimes some VMs are not accessible by console via horizon's.

When attempt is unsuccessful, horizon shows connection error 1006, nova-novncconsole service is logging:
2015-06-18T22:36:43.791855+00:00 info: handler exception: Timed out waiting for a reply to message ID 1ad9f21701bf46e9a6e26c3ab26d9e39.
The problem goes away some times without any services restart.

The following messages could also be seen in the logs, however it's not confirmed that they're related to the issue:

2015-06-17T11:50:16.312581+00:00 info: handler exception: Remote error: MemcachedKeyNoneError Key is None
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 180, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 126, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 121, in check_token\n token_str = self.mc.get(token.encode(\'UTF-8\'))\n', u' File "/usr/lib/python2.6/site-packages/memcache.py", line 898, in get\n return self._get(\'get\', key)\n', u' File "/usr/lib/python2.6/site-packages/memcache.py", line 847, in _get\n self.check_key(key)\n', u' File "/usr/lib/python2.6/site-packages/memcache.py", line 1047, in check_key\n raise Client.MemcachedKeyNoneError("Key is None")\n', u'MemcachedKeyNoneError: Key is None\n'].

MOS version: 6.0

description: updated
summary: - Sometimes VMs are not accessible via console from horizon
+ [nova] Sometimes VMs are not accessible via console from horizon
tags: added: customer customer-found
Changed in mos:
importance: Undecided → High
Changed in mos:
status: New → Confirmed
assignee: nobody → MOS Nova (mos-nova)
milestone: none → 7.0
milestone: 7.0 → 6.1-updates
Revision history for this message
Alexander Gubanov (ogubanov) wrote :

I can't reproduce this issue (tried on more than 10 different ISO 6.1 and 7.0).
Could you please provide more details or ping me once you face this issue again?
Move to Incomplete.

Changed in mos:
status: Confirmed → Incomplete
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

I guess the root cause is the problem with oslo.messaging that we've fixed after 6.1 where nova compute just got stuck and thus didn't server console requests.
Usually nova-compute restart solves the issue.

I'm not sure this is reproducible in 7.0.

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Setting to Invalid for 6.1 according to comment #1. Confirmed for 6.0-updates.

Changed in mos:
status: Incomplete → Invalid
Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Eugene - please provide exact steps to reproduce this.

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

No feedback in 1 month, closing.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.