There's an race condition in setting console mode for shellinabox

[Zhenguo Niu]

Currently, the locks of node are released before really starting/stopping shellinabox process, then there can be multi threads running the codes at the same time, the pid file may be unlinked by thread A, and thread B gets success returned value but pid file doesn't exit, it will be stuck in popen.communicate().

[Zhenguo Niu (niu-zglinux)]

#!/bin/bash

for i in `seq 1 10000`; do
   echo $i
   ironic node-set-console-mode bm-1 true
   ironic node-set-console-mode bm-1 false
done

when testing with be above script, the conductor will be stuck after a few loop.

[Ruby Loo (rloo)]

hi Zhenguo, are you still working on this?

[Vladyslav Drok (vdrok)]

Jay, do you know the reason of this? Looking at the code, I'm not really sure what's the problem here. Otherwise, I'd move it back to confirmed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/603812

[Dmitry Galkin (galkindmitrii)]

Hi,

We had exactly this problem with shellinabox. Node gets locked on console enable call and is not automatically unlocked. Console or any operations do not work after. Easily reproducible with bash script from above.

With the patch submitted I was not able to lock the node by triggering console enable/disable in a loop.

I assume that might be happening with socat as well, but we run shellinabox and I don't have another environment to test.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/603812
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=0256e1e69f4dd38cf0a80532337fa30c89502cb4
Submitter: Zuul
Branch: master

commit 0256e1e69f4dd38cf0a80532337fa30c89502cb4
Author: Dmitry Galkin <email address hidden>
Date: Wed Sep 19 14:02:48 2018 +0000

    Fix node exclusive lock not released on console start/restart.

    This patch forces kill of console process with SIGKILL if it did not
    terminate on SIGTERM within the CONF.console.kill_timeout and reads the
    shellinabox subprocess stdout/stderr after CONF.console.subprocess_timeout
    or if subprocess exited with non zero code.

    Change-Id: I55a112d877d94f31d27487846ff59fe27f602f8b
    Closes-Bug: 1587313
    Story: 1587313
    Task: 9654

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 12.0.0 release.