Wrong node-discover accessibility check logic

[Aleksey Zvyagintsev]

Currently, is no way to check mcollvective status before reporting to fuel-master 'Node discovered and on-line'

It's raise errors in flow :

1) Bootstrap node started
2)Nailgun-agent report that node discovered [1]
3)Fuel-qa\user add nodes to cluster and press 'Deploy'
4)Mcollective still not ran [0] due some reason (long start-up\etc)
Like result- error will be raised:
[Method verify_networks. Network verification not avaliable because nodes ["1", "2", "3", "4", "5"] not avaliable via mcollective. ]

I propose to fix issue in simplest way:
Add lock\check for mcollective status in nailgun-agent.cron before run nailgun-agent.

Other way: improve agent to check mcollective itself
(this soluthion more complex and maybe be not necessary - mcollective can be deprecated in fuel-future.)

[0]
https://github.com/openstack/fuel-agent/blob/master/contrib/fuel_bootstrap/files/trusty/usr/bin/fix-configs-on-startup#L65

[1]
https://github.com/openstack/fuel-nailgun-agent/blob/master/nailgun-agent.cron

[Maksim Malchuk (mmalchuk)]

agree with additional checks in the nailgun-agent.cron

[Aleksey Zvyagintsev (azvyagintsev)]

Also, problem should be investigated for already deployed system - looks like they also affected

[Maksim Malchuk (mmalchuk)]

they can't be affected because status has changed from 'discovered' to something else

[Aleksey Zvyagintsev (azvyagintsev)]

Nope, they can :
Node discovered => provisioned (not deployed) => nailgun-agent ran, but mcollective still in progress.

[slava valyavskiy (slava-val-al)]

From my point of view this piece of logic should be moved into nailgun agent code.
btw. we already have mcollective related code into nailgun agent code.

[Matthew Mosesohn (raytrac3r)]

This bug cannot be tech-debt and critical. It must be one or the other.

[Maksim Malchuk (mmalchuk)]

@Slava, can You propose a solution in the nailgun?

[slava valyavskiy (slava-val-al)]

@Maksim, check status of mcollective service more correctly than https://github.com/openstack/fuel-nailgun-agent/blob/master/agent#L120

[Timur Nurlygayanov (tnurlygayanov)]

This bug is the root of randomly failed BVT tests, it is marked as Critical for MOS 8.0

[Nastya Urlapova (aurlapova)]

@TImur, could you attach bvt job example which you mentioned ?

[Aleksey Zvyagintsev (azvyagintsev)]

Related:
https://bugs.launchpad.net/fuel/+bug/1538055/comments/8 =>>>
https://bugs.launchpad.net/fuel/+bug/1518306/comments/12

[Dmitry Pyzhov (dpyzhov)]

I'm against fixing it in a week before HCF. Let's not backport it.

[Dmitry Pyzhov (dpyzhov)]

And I don't think that creating of one more workaround around another workaround is a good idea. Workflow needs to be redesigned. Mcollective config is updated by at least three tools. Obviously there is a space for improvement here.

[Aleksey Zvyagintsev (azvyagintsev)]

@DPyznov:
its critical issue, and should be fixed in 8.0 also - otherwise we will always catch random problems with tests.

[Michael Polenchuk (mpolenchuk)]

use in nailgun-agent a kind of the following check:
def get_mco_status
  Process.kill 0, %x(status mcollective).split.last.to_i
rescue
 0
end

# returns 1 if process found and active, otherwise 0

[Dmitry Bilunov (dbilunov)]

We can check if mcollective is available without any additional tools and patches just by inserting a one-line condition into nailgun-agent.cron:

# timeout 5 ruby -rmcollective -e 'include MCollective::RPC; rpcclient("version", :options => { :config => "/etc/mcollective/server.cfg" }).get_version'
# echo $?
0
(now setting invalid configuration in /etc/mcollective/server.cfg)
# timeout 5 ruby -rmcollective -e 'include MCollective::RPC; rpcclient("version", :options => { :config => "/etc/mcollective/server.cfg" }).get_version'
Could not create RPC client: Could not connect to RabbitMQ Server: SIGTERM
# echo $?
124

[Timur Nurlygayanov (tnurlygayanov)]

@Nastya, it looks like my mistake, the BVT on #484 failed because of other issue, https://bugs.launchpad.net/fuel/+bug/1533082

[Dmitry Pyzhov (dpyzhov)]

Guys, nailgun agent creates a record in nailgun, takes id on the node, then updates mcollective config and restarts mcollective. Your extra check will ruin the workflow. Please update tests, add a wait till 'mco ping' returns all nodes.

[slava valyavskiy (slava-val-al)]

@Dmitry, and what is about users? Should we add this ....issue into the release notes?

[Aleksey Zvyagintsev (azvyagintsev)]

@Dmitry, fix fuel-qa didn't fix deployment process
(https://bugs.launchpad.net/fuel/+bug/1538055/comments/4)
Which actually (my assume) root cause of other problems (like ntpd granular timeout)

[Dmitry Pyzhov (dpyzhov)]

@slava, @Aleksey, could you provide use case that affects users? The only case that I see is when a user wants to deploy a cluster automatically as soon as all nodes are available. I guess we can live with workaround for this case until we redesign our flow.

[Dmitry Pyzhov (dpyzhov)]

Technical debt, doesn't affect real users. Removing from Mitaka release.