[RFE] Nova serial console support for Ironic

[Dao Cong Tien]

Today, Ironic provides a IPMI+Shellinabox console as a serial console via web, but this kind of console is not compatible with Nova serial console. Hence, the admin cannot see the console on Horizon dashboard.

It's nice if we can make Ironic support the Nova serial console. If so, the admin can see and interact with the bare metals console through Horizon. And the Nova CLI 'nova get-serial-console <instance>' will work by returning a valid URL (websocket URL).

[Jim Rollenhagen (jim-rollenhagen)]

Agree, this would be great. Could you please add some information on how we can make this work?

[Dao Cong Tien (tiendc)]

Jim Rollenhagen:

My idea is to add a new proxy, says ironic-ipmiproxy, into Ironic scope. The proxy stands between nova-serialproxy and the bare metals. I plan to use python lib pyghmi to help the proxy talk to the bare metals. For serving nova-serialproxy, the proxy will start a TCP socket server, with an admin port and each port for each active bare metal.

I also plan to add support for the proxy into the existing console driver (IPMIShellinaboxConsole), so all the vendors that use the console will not need to update their drivers to get the feature work.

[Akira Yoshiyama (yosshy)]

Do you like below?
https://github.com/yosshy/wiki/wiki/image/ironic_console_on_horizon-4.png

I have a series of Ironic/nova commits for it. I'll send them for review soon.

[Akira Yoshiyama (yosshy)]

and my spec: https://review.openstack.org/#/c/293827/1

[Akira Yoshiyama (yosshy)]

more screenshots:
https://github.com/yosshy/wiki/wiki/image/ironic_console_on_horizon-6.png
https://github.com/yosshy/wiki/wiki/image/ironic_console_on_horizon-7.png

[Akira Yoshiyama (yosshy)]

I've put my spec and patchset for review:
https://review.openstack.org/#/q/topic:bp/nova-compatible-serial-console

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/294866

[Jim Rollenhagen (jim-rollenhagen)]

Thanks for the extra info Akira, we'll continue working out the details in the spec :)

Also, in talking with Spamaps in IRC just now, it would be great if it was possible to optionally have consoles always enabled such that we could log them and implement "nova console-log" as well. That's probably quite a bit of work worth saving for a separate discussion though, this is a good start.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/296869

[Dao Cong Tien (tiendc)]

Akira: I've had a look at your spec, it's great. But I think my spec still has some advantages in comparison to yours. Please allow me to use some ideas of yours in my spec. My commit is below:

https://review.openstack.org/296869

[Akira Yoshiyama (yosshy)]

@tiendc: thanks for your information. It looks nice, especially pointing that each node isn't tied to a specific ironic-conductor. As you know, my spec is based on the current web console implementation with Shell-in-a-box, so I'm afraid that the current one has the same problem. However, I don't know it is actually a problem.

@jim-rollenhagen: thanks for your comment and sorry about my late reply. I've designed a couple of console-log implementations, but no best one. IMO, it's better to separate a console-log spec from this one.

[Akira Yoshiyama (yosshy)]

FYI: my current spec and patch set: https://review.openstack.org/#/q/topic:bug/1553083

[Zhenguo Niu (niu-zglinux)]

For Shellinabox console, I think we should add a new http proxy to nova, and use nova get-shellinabox-console to get a http/https url instead of the ws/wss url returned by nova get-serial-console. I would like to propose a separate spec for that, and I have already tested it in my local environment, thanks!

[Dao Cong Tien (tiendc)]

Zhenguo Niu: Thanks for your info. I would love to see your spec. But if I am right, your spec will be for Nova instead of Ironic. Anyway, I'm looking forward to seeing it.

[Zhenguo Niu (niu-zglinux)]

Dao Cong Tien: Yes, all work will be done in nova side instead of ironic, as shellinabox is capable of integration with nova and horizon, will draft a spec soon,

[Jim Rollenhagen (jim-rollenhagen)]

@yosshy: yeah, agree console-log can be a separate piece of work. :)

[Zhenguo Niu (niu-zglinux)]

Here is the spec in nova to add shellinabox console support https://review.openstack.org/#/c/300582/

[Akira Yoshiyama (yosshy)]

@jim-rollenhagen: Thanks for your reply. I checked "Add console log support" spec*1. I agree it. We need a daemon that supports serial console and console-log for ironic-conductor. And...I've implemented it*2. To use it, I have to modify my spec and my current console driver.

*1: https://review.openstack.org/#/c/164586/
*2: https://gist.github.com/yosshy/68f9a0b9b534dec27aad599744915b50

[Akira Yoshiyama (yosshy)]

I made new drivers for Ironic and Nova with the new console service.

Instance info: https://github.com/yosshy/wiki/wiki/image/ironic_console_on_horizon-21.png
console log: https://github.com/yosshy/wiki/wiki/image/ironic_console_on_horizon-22.png
serial console: https://github.com/yosshy/wiki/wiki/image/ironic_console_on_horizon-23.png

[Akira Yoshiyama (yosshy)]

Related reviews:
https://review.openstack.org/#/q/topic:bug/1526368

[Akira Yoshiyama (yosshy)]

Some patch sets are abandoned. See below:
https://review.openstack.org/#/q/topic:bug/1526368+status:open

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/302291

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/303894

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/304933

[Akira Yoshiyama (yosshy)]

My new spec is below:

Add nova-compatible-serial-console.rst to not-implemented
https://review.openstack.org/#/c/306755/

It is based on ironic-console-server.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/307181

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/293827
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/293872
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/293871
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/293875
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/293874
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/293873
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dao Cong Tien (<email address hidden>) on branch: master
Review: https://review.openstack.org/296869
Reason: I and Akira Yoshiyama agree to build up and maintain one solution for this feature at https://review.openstack.org/#/c/306754/.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dao Cong Tien (<email address hidden>) on branch: master
Review: https://review.openstack.org/304933
Reason: Spec abandoned.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dao Cong Tien (<email address hidden>) on branch: master
Review: https://review.openstack.org/303894
Reason: Spec abandoned.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/302291
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/301714
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/301715
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira YOSHIYAMA (<email address hidden>) on branch: master
Review: https://review.openstack.org/307181
Reason: To be unified as a new spec based on the console session of the last summit.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/319505

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/328168

[OpenStack Infra (hudson-openstack)]

Change abandoned by Akira Yoshiyama (<email address hidden>) on branch: master
Review: https://review.openstack.org/293874
Reason: This will be developed after driver composition reform is done.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/319505
Committed: https://git.openstack.org/cgit/openstack/ironic-specs/commit/?id=75290c72f7c3138bfdab151728718f771dfb9685
Submitter: Jenkins
Branch: master

commit 75290c72f7c3138bfdab151728718f771dfb9685
Author: Yuiko Takada <email address hidden>
Date: Fri May 20 16:11:32 2016 +0900

    Nova compatible serial console support

    This spec proposes adding support for Nova compatible serial console into
    Ironic. The solution is to add a driver using socat.

    Change-Id: I0a6f5295e0f1cc465247a9b121962710e5b1012a
    Related-Bug: #1553083

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/335378

[Ruby Loo (rloo)]

The spec has been approved; setting tag to rfe-approved.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328168
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=22a80f77c578b4c10298ee16daf351fd5bfc25d4
Submitter: Jenkins
Branch: master

commit 22a80f77c578b4c10298ee16daf351fd5bfc25d4
Author: Dao Cong Tien <email address hidden>
Date: Fri Jun 10 15:25:46 2016 +0700

    Nova-compatible serial console: socat console_utils

    This adds console_utils functions for 'socat' console.
    Implements:
    - get_socat_console_url(): returns url for socat console
    - start_socat_console(): uses (socat + console_cmd execution)
    - stop_socat_console(): stops socat/console_cmd subprocess

    Change-Id: I79ddd83d12cc8111e05b5107359d6db8a8881d61
    Spec: https://review.openstack.org/#/c/319505/
    Related-Bug: #1553083

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293873
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=857372a2269cdd0f8a1ae5b9e9f6e0ee193f01be
Submitter: Jenkins
Branch: master

commit 857372a2269cdd0f8a1ae5b9e9f6e0ee193f01be
Author: Yuiko Takada Mori <email address hidden>
Date: Thu Jul 14 11:17:24 2016 +0900

    IPMITool: add IPMISocatConsole and IPMIConsole class

    IPMISocatConsole is a new console interface class using ipmitool and
    socat. It has the same APIs of IPMIShellinaboxConsole class and
    provides TCP4/TCP6 connection service to connect serial-on-LAN of
    nodes.
    IPMIConsole is a new console interface class using ipmitool and used
    as parent class of IPMIShellinaboxConsole and IPMISocatConsol class.

    This patch set implements new console driver interfaces
    IPMISocatConsole.

    To use PXE + IPMItool + socat, specify pxe_ipmitool_socat.
    To use IPA + IPMItool + socat, specify agent_ipmitool_socat.

    Spec: https://review.openstack.org/#/c/319505/
    Partial-Bug: #1553083
    Change-Id: I35a7dcb7e89baf16d096501fd44dbc12adc8c61e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/335378
Committed: https://git.openstack.org/cgit/openstack/ironic-specs/commit/?id=0457a87dc6705baf11872a69b7c8da2a0cd25bb4
Submitter: Jenkins
Branch: master

commit 0457a87dc6705baf11872a69b7c8da2a0cd25bb4
Author: Yuiko Takada Mori <email address hidden>
Date: Wed Jun 29 17:39:59 2016 +0900

    Fix Nova-compatible Serial Console's Other deployer impact

    This is a follow-up patch of 75290c72f7c3138bfdab151728718f771dfb9685
    adds explanation about config option [console]terminal
    and remove detail config file part.

    Change-Id: I295ddd245e6bf4e7d132d3772f5b0465dec80227
    Related-Bug: #1553083

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293872
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=bdb488e6a8fd34e7e38458280adacebf8e300312
Submitter: Jenkins
Branch: master

commit bdb488e6a8fd34e7e38458280adacebf8e300312
Author: Yuiko Takada Mori <email address hidden>
Date: Fri Jul 29 00:07:04 2016 +0900

    Add node serial console documentation

    This commit adds the serial console documentation.

    Partial-Bug: #1553083
    Change-Id: I3f5cef40a1fc9276efa85b643cabbc2b81f53169

[Dmitry Tantsur (divius)]

Implemented on Ironic side, please feel free to add Nova to this bug.

[Akihiro Motoki (amotoki)]

The nova patch has been merged.

commit c9a64996ecc317b2c05d688e0f5d31c37122ca01
Author: Dao Cong Tien <email address hidden>
Date: Fri Jun 10 15:02:43 2016 +0700

    Ironic Driver: override get_serial_console()

    This implements Nova-compatible serial console for Ironic in
    Nova scope.

    [Nova] blueprint ironic-serial-console-support
    [Ironic] spec
        https://review.openstack.org/319505
    [Ironic] related patches
        https://review.openstack.org/#/c/293873/
        https://review.openstack.org/#/c/328168/
    [Ironic] Related-Bug: #1553083

    Change-Id: I38e803021d71fc0760a8ae99b3e97dd0aecb5088