[RFE] Add CLI serial over lan support to Ironic

[Christopher Brown]

Currently there is no way to access serial over lan (SOL) AKA Console Redirection within OpenStack Ironic.

Almost all servers support SOL and it is a very useful tool for debugging and troubleshooting, especially with remote hardware. Often a web console is impractical or simply not possible.

The command line version using IPMItool is as follows:

ipmitool -I lanplus -U $USERNAME -P $PASSWORD -H $HOSTNAME sol activate

So we can configure OpenStack to know about all the above arguments then hopefully it isn't too difficult to enable support so you can just:

openstack baremetal get-console UUID

[Dmitry Tantsur (divius)]

I assume, judging by our IRC conversation, that it means CLI support, not just shellinabox, right? If so, could you please update the description to make it clear?

[Christopher Brown (snecklifter)]

Hi Dmitry,

Thanks, yes, CLI support so nothing to do with shellinabox.

The rcons tool in xCAT provides exactly the functionality I am looking for - it is basically an ipmitool wrapper which implements SOL console but without having to input parameters - very useful for debugging baremetal booting.

http://xcat-docs.readthedocs.org/en/2.11/guides/admin-guides/references/man1/rcons.1.html

Thanks

[Matthew Rooney (joemonk)]

I would like to start working on this but I have a few questions:

- Is the client expected to have access to the IPMI network or should we proxy this through the Ironic controller?
- Is the client expected to have ipmitool?

The existing command, 'ironic node-get-console UUID', returns information about an existing console.

- Would it be preferable to have a new command that drops the user into a shell instead?

[Sam Betts (sambetts)]

A user running the command:

    openstack baremetal get-console UUID

 should not have access to the IPMI network, and shoudn't be proxied into it either because that would potentially break the multi-tenancy separation. The IPMI network is likely to contain nodes that belong to more than one tenant and providing access to the IPMI endpoints of all those nodes opens a potential attack vector.

I think perhaps that the Ironic conductor needs to be the one to run the ipmitool command, and connect to the SOL console, then there needs to be an endpoint set up through the Ironic API service or through a separate public facing service, that allows the user to connect to the SOL console running on the conductor, via telnet or something like that?

[Jim Rollenhagen (jim-rollenhagen)]

Yeah, I agree that our client shouldn't be responsible for making a connection to the BMC, regardless of if proxied or not. ironicclient should really only talk to the API, and not conductors or BMCs.

I tend to think users should fetch the info and start ipmitool themselves; we've talked about having some tools scripts in the past, maybe this is something that fits there.

[Christopher Brown (snecklifter)]

@jim-rollenhagen Not sure I follow. Doesn't the client already talk to the BMC to issue power commands such as

openstack baremetal power --on

rcons in xCAT uses conserver so it would be great just to mirror the way that interacts.

My point here is that because ironic already knows how to connect to the bmc, getting it to bring up the SOL console shouldn't (hopefully!) be much of a step further and console access is incredibly useful.

I can organise a video of rcons in use if that would help?

[Matthew Rooney (joemonk)]

I have composed a short video detailing the idea behind the solution I am working on.

The proposed method is as follows:

- Ironic controller set up with separate sshd running under a user with restricted permissions
- Ironic command to issue request for console
e.g. node-get-console UUID
- Client creates keypair and sends to the controller
- Controller adds the user's newly created public key to the authorized_keys file and locks the connection to the generated ipmitool command for requested node
- Controller completes connection between user and baremetal ipmi
- The user is presented with the serial console
- If the user exits the serial console, the ssh connection is dropped also
- After this connection has ended, the controller removes the public key from authorized_keys

Video: https://youtu.be/AYx9OyTZmO8

[Christopher Brown (snecklifter)]

Hi Matthew,

Yes, that's pretty much exactly the functionality I was after. The use of an sshd is something I hadn't considered.

I expected that because the ironic controller already has details for accessing ipmi on the baremetal nodes, all that would be required is for the controller to convert the request to view a node's console.

I suppose the advantage of your solution is that it can be run from any node? The xCAT implementation only allows connections locally from the head/mgmt node I believe.

[Jim Rollenhagen (jim-rollenhagen)]

@Chris, no, the power commands use the ironic API, and the conductor is responsible for the IPMI calls. What I meant is that the client shouldn't be making a direct (or proxied) connection to the BMC. I'm fine some websocket or other streaming endpoint that can move data between the client and the conductor that is connected to the console.

tl;dr I agree with Sam here.

At any rate, I think it's clear this needs a spec :)

[Matthew Rooney (joemonk)]

I have a working solution now at least for my purposes. I have made some minor changes to the API but mostly elsewhere. I am continuing to iron out a few things with my code as well as add some more tests.

I also agree that a spec is required.

[Sam Betts (sambetts)]

Watching the video I think this design is in the right direction, and its much more inline with how the current shellinabox console implementation works.

I think that the ssh connection by users should be unauthenticated, to be alligned with the shellinabox console or that the get-console command should POST the users public ssh key through to ironic to be added to the authorized keys list. The other thing is that there should only be 1 sshd instance per node, and only one SOL connection to the node itself, with multiplexing to multiple users done in the ironic service by using a tool like screen to let multiple users connect to the same session.

The implementation for this should also not be just specific to IPMI, in the same way as the shellinabox implementation is written, it should be able to be used by any driver by specifying what command to use to connect to the console, e.g. the ssh driver, and ipmi tool driver both implement shellinabox consoles by passing in different commands into start_shellinabox_console.

So basically I see it like this:

start_console:

    Start screen session running the required command to connect to the SOL
    Start the sshd daemon, and configured to run a "screen -x" command for any new ssh connections into the screen session started above.

get_console:

    Returns the URL required to SSH into the sshd daemon started above

stop_console:

    Ends screen and sshd processes

The user experience should be something along these lines:

    ironic node-get-console <node_uuid> --method cli

    returns something like ssh://<ip_addr>:<port>

    then they can ssh ssh://<ip_addr>:<port>

or

    ironic node-get-console <node_uuid> --method gui

    returns shellinabox http url

    then they can open a browser and connect to the console

[Christopher Brown (snecklifter)]

Hi Sam,

That's a great idea regarding cli vs gui.

If there is anything I can do to help move this along please let me know.

I have access to plenty of test hardware.

[Matthew Rooney (joemonk)]

Pushed WIP for review
ironic: https://review.openstack.org/311196
python-ironicclient: https://review.openstack.org/311205

[Ruby Loo (rloo)]

We had a design session at the barcelona summit, here's the etherpad: https://etherpad.openstack.org/p/ironic-ocata-summit-vnc-console.

I don't know if any of it is related to this, and/or if there is any overlap, but Sam mentioned that there might be.

[Emilien Macchi (emilienm)]

This bug was last updated over 180 days ago, as tripleo is a fast moving project and we'd like to get the tracker down to currently actionable bugs, this is getting marked as Invalid. If the issue still exists, please feel free to reopen it.

[Dmitry Tantsur (divius)]

I think this was fixed with the introduction of the socat console.