nailgun-agent hangs when it can not list disks

[Łukasz Oleś]

When one of disks is broken, for example(from dmesg):

[12861.715699] Buffer I/O error on device fd0, logical block 0
[12873.965252] end_request: I/O error, dev fd0, sector 0

nailgun-agent will hang. Node will go to offline state.

[Sergii Golovatiuk (sgolovatiuk)]

Due to definitions https://wiki.openstack.org/wiki/Fuel/How_to_contribute this bug as such cases may appear and there is no workaround for this.

[Mike Scherbakov (mihgen)]

This is not a Medium, certainly higher.
If node goes offline, we can't really deploy anything on it. I hope agent, when starts, verifies if there is a copy already running. Otherwise we may end up with hundreds of nailgun-agents in Linux before node goes into stuck mode.

[Łukasz Oleś (loles)]

Mike, there will be only one agent. It uses locks and new will not start until the old one finishes

[Mike Scherbakov (mihgen)]

Vladimir, folks,
why can't we simply surround the code with
require 'timeout'
status = Timeout::timeout(5) {
  # Something that should be interrupted if it takes more than 5 seconds...
}
?

Failure of one disk should not affect the whole node from being discovered in Nailgun.

[Vladimir Sharshov (vsharshov)]

+1 for Mike solution.

But we have little side affect: some disks will be disappeared from web UI. I think this is small price.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/180100

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/180100
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=689fdeddb8b08cf07300dd554603f65d495559f4
Submitter: Jenkins
Branch: master

commit 689fdeddb8b08cf07300dd554603f65d495559f4
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue May 5 13:35:08 2015 +0300

    Prevent agent hangs if ohai does not return disks

    Instead of freeze we got all data without disks now.
    Current timeout - 30 sec.

    Co-Authored-By: Mike Scherbakov (mihgen) <email address hidden>
    Change-Id: I65d1b570cd01e12b521403c6d6e990043eb2c2ab
    Closes-Bug: #1396086