ironic serial console port validate always failed

Bug #1809418 reported by int32bit
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
int32bit
Rocky
Fix Committed
High
Matt Riedemann

Bug Description

I deployed latest ironic and used IPMI_socat console interface alongs with nova-serialproxy service, but always failed to open console in dashboard, and the nova-serialproxy error as follows:

2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy Traceback (most recent call last):
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 930, in top_new_client
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 860, in do_handshake
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy self.RequestHandlerClass(retsock, address, self)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 324, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 114, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.__init__(self, req, addr, server)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy self.handle()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 581, in handle
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.handle(self)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy self.handle_one_request()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib64/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy method()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 543, in do_GET
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy if not self.handle_websocket():
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 531, in handle_websocket
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy self.new_websocket_client()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 236, in new_websocket_client
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy connect_info = self._get_connect_info(ctxt, token)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 192, in _get_connect_info
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy connect_info = self._get_connect_info_database(ctxt, token)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 166, in _get_connect_info_database
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy raise exception.InvalidToken(token='***')
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy InvalidToken: The token '***' is invalid or has expired
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy
2018-12-20 19:11:40.867 14996 DEBUG nova.console.websocketproxy [-] 197.3.133.254: new handler Process vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:875

So I created a breakpoint in this code and found the port type must be str in nova-serialproxy side as we converted it to str type forcefully before rpc validate_console to compute service[1]. But on the compute side, the drive may be return int type like ironic serial console[2]. So the validate_console_port may always return False as type inconsistence[3].

[1] https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L123-L125
[2] https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L1819
[3] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5369

Tags: console ironic
int32bit (int32bit)
Changed in nova:
assignee: nobody → int32bit (int32bit)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/626853

Changed in nova:
status: New → In Progress
int32bit (int32bit)
tags: added: ironic
tags: added: console
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → High
Changed in nova:
assignee: int32bit (int32bit) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → int32bit (int32bit)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/626853
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=47937b6113dfe34202926536a2c43d3f85676026
Submitter: Zuul
Branch: master

commit 47937b6113dfe34202926536a2c43d3f85676026
Author: int32bit <email address hidden>
Date: Fri Dec 21 15:35:36 2018 +0800

    Convert port to str when validate console port

    The port type must be str type in nova-serialproxy side as we convert to str
    forcefully before rpc validate_console to compute service[1]. But on the
    compute side, the driver may be return int type like ironic serial console[2].
    So the validate_console_port may always return False as type inconsistence[3].

    [1] https://github.com/openstack/nova/blob/18.0.0/nova/console/websocketproxy.py#L144
    [2] https://github.com/openstack/nova/blob/18.0.0/nova/virt/ironic/driver.py#L1830
    [3] https://github.com/openstack/nova/blob/18.0.0/nova/compute/manager.py#L5417

    Closes-Bug: 1809418

    Change-Id: If5f3dc725c5836e67d09deeefb3bce2249f71a69

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/632720

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/632720
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ba269da87a7ac0130f87fca78083a09231533c2f
Submitter: Zuul
Branch: stable/rocky

commit ba269da87a7ac0130f87fca78083a09231533c2f
Author: int32bit <email address hidden>
Date: Fri Dec 21 15:35:36 2018 +0800

    Convert port to str when validate console port

    The port type must be str type in nova-serialproxy side as we convert to str
    forcefully before rpc validate_console to compute service[1]. But on the
    compute side, the driver may be return int type like ironic serial console[2].
    So the validate_console_port may always return False as type inconsistence[3].

    [1] https://github.com/openstack/nova/blob/18.0.0/nova/console/websocketproxy.py#L144
    [2] https://github.com/openstack/nova/blob/18.0.0/nova/virt/ironic/driver.py#L1830
    [3] https://github.com/openstack/nova/blob/18.0.0/nova/compute/manager.py#L5417

    Closes-Bug: 1809418

    Change-Id: If5f3dc725c5836e67d09deeefb3bce2249f71a69
    (cherry picked from commit 47937b6113dfe34202926536a2c43d3f85676026)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.2.0

This issue was fixed in the openstack/nova 18.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.