ironic serial console port validate always failed

Bug #1809418 reported by int32bit on 2018-12-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
int32bit
Rocky
High
Matt Riedemann

Bug Description

I deployed latest ironic and used IPMI_socat console interface alongs with nova-serialproxy service, but always failed to open console in dashboard, and the nova-serialproxy error as follows:

2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy Traceback (most recent call last):
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 930, in top_new_client
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 860, in do_handshake
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy self.RequestHandlerClass(retsock, address, self)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 324, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 114, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.__init__(self, req, addr, server)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy self.handle()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 581, in handle
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.handle(self)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy self.handle_one_request()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib64/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy method()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 543, in do_GET
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy if not self.handle_websocket():
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 531, in handle_websocket
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy self.new_websocket_client()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 236, in new_websocket_client
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy connect_info = self._get_connect_info(ctxt, token)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 192, in _get_connect_info
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy connect_info = self._get_connect_info_database(ctxt, token)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 166, in _get_connect_info_database
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy raise exception.InvalidToken(token='***')
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy InvalidToken: The token '***' is invalid or has expired
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy
2018-12-20 19:11:40.867 14996 DEBUG nova.console.websocketproxy [-] 197.3.133.254: new handler Process vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:875

So I created a breakpoint in this code and found the port type must be str in nova-serialproxy side as we converted it to str type forcefully before rpc validate_console to compute service[1]. But on the compute side, the drive may be return int type like ironic serial console[2]. So the validate_console_port may always return False as type inconsistence[3].

[1] https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L123-L125
[2] https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L1819
[3] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5369

int32bit (int32bit) on 2018-12-21
Changed in nova:
assignee: nobody → int32bit (int32bit)

Fix proposed to branch: master
Review: https://review.openstack.org/626853

Changed in nova:
status: New → In Progress
int32bit (int32bit) on 2018-12-27
tags: added: ironic
tags: added: console
Matt Riedemann (mriedem) on 2019-01-22
Changed in nova:
importance: Undecided → High
Changed in nova:
assignee: int32bit (int32bit) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem) on 2019-01-22
Changed in nova:
assignee: Matt Riedemann (mriedem) → int32bit (int32bit)

Reviewed: https://review.openstack.org/626853
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=47937b6113dfe34202926536a2c43d3f85676026
Submitter: Zuul
Branch: master

commit 47937b6113dfe34202926536a2c43d3f85676026
Author: int32bit <email address hidden>
Date: Fri Dec 21 15:35:36 2018 +0800

    Convert port to str when validate console port

    The port type must be str type in nova-serialproxy side as we convert to str
    forcefully before rpc validate_console to compute service[1]. But on the
    compute side, the driver may be return int type like ironic serial console[2].
    So the validate_console_port may always return False as type inconsistence[3].

    [1] https://github.com/openstack/nova/blob/18.0.0/nova/console/websocketproxy.py#L144
    [2] https://github.com/openstack/nova/blob/18.0.0/nova/virt/ironic/driver.py#L1830
    [3] https://github.com/openstack/nova/blob/18.0.0/nova/compute/manager.py#L5417

    Closes-Bug: 1809418

    Change-Id: If5f3dc725c5836e67d09deeefb3bce2249f71a69

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/632720
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ba269da87a7ac0130f87fca78083a09231533c2f
Submitter: Zuul
Branch: stable/rocky

commit ba269da87a7ac0130f87fca78083a09231533c2f
Author: int32bit <email address hidden>
Date: Fri Dec 21 15:35:36 2018 +0800

    Convert port to str when validate console port

    The port type must be str type in nova-serialproxy side as we convert to str
    forcefully before rpc validate_console to compute service[1]. But on the
    compute side, the driver may be return int type like ironic serial console[2].
    So the validate_console_port may always return False as type inconsistence[3].

    [1] https://github.com/openstack/nova/blob/18.0.0/nova/console/websocketproxy.py#L144
    [2] https://github.com/openstack/nova/blob/18.0.0/nova/virt/ironic/driver.py#L1830
    [3] https://github.com/openstack/nova/blob/18.0.0/nova/compute/manager.py#L5417

    Closes-Bug: 1809418

    Change-Id: If5f3dc725c5836e67d09deeefb3bce2249f71a69
    (cherry picked from commit 47937b6113dfe34202926536a2c43d3f85676026)

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

This issue was fixed in the openstack/nova 18.2.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers