[centos] mcollective autostart is not ensured on nodes

[Bogdan Dobrelya]

# chkconfig | grep mcollective
mcollective 0:off 1:off 2:off 3:off 4:off 5:off 6:off

This results in nodes became unmanaged for Fuel orchestration after reboot.
Reproduced at 6.1 dev ISO #291

[Alex Schultz (alex-schultz)]

So I looked into this bug on 6.1 and while chkconfig reports mcollective as being off, if you reboot and mcollectived is being started. Are there additional steps required to reboot and result in mcollective in not running?

As far as I can tell, it is being started via rc.local which calls fix-configs-on-startup.

[Alex Schultz (alex-schultz)]

In looking further, the bootstrap centos starts mcollective via the fix-configs-on-startup but if you provision an actual centos environment it does not start mcollective. I'm also not sure if the ubuntu version starts mcollective either, going to continue to dig deeper.

[Bogdan Dobrelya (bogdando)]

There is no such entry in /etc/rc.local at my Centos Fuel 6.1 lab:

# cat /etc/rc.local
flock -w 0 -o /var/lock/agent.lock -c "/opt/nailgun/bin/agent >> /var/log/nailgun-agent.log 2>&1"

Here is init scripts and runlevels details: http://pastebin.com/7HMnZTPX
I guess, it works on ubuntu due to LSB headers - there are matching S/K levels in /etc/rcX.d

As for centos, chckonfig header is ok, but chkonfig is not configured by puppet, so we should re-check out puppet manifests, do they ensure mcollective service enabled?

Note: as I reported, mcollective is running OK after reboot on Ubuntu, and stopped - at Centos

[Bogdan Dobrelya (bogdando)]

We have mcollective service enabled in puppet manifests https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/mcollective/manifests/server.pp#L62.

And if I run puppet apply -e "service { 'mcollective': ensure=>running, enable=>true }" at my lab, chkconfig gets runlevels configured. That is really strange why this don't work at deploy stage...

[Alex Schultz (alex-schultz)]

I found that mcollective is started via the cloud-init process post install and configured there as well. As far as I can tell we aren't actively managing mcollective via puppet as part of the deployment. I'm working on a fix to the fuel-agent cloud-init configuration to address this for centos.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/174005

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/174005
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=5a4556ea0cf943edffa33788994b7fb7abe150b3
Submitter: Jenkins
Branch: master

commit 5a4556ea0cf943edffa33788994b7fb7abe150b3
Author: Alex Schultz <email address hidden>
Date: Wed Apr 15 12:46:45 2015 -0500

    Ensure mcollective starts on boot for CentOS

    Change-Id: I7963dac00795245eb32ec9ad6e57ff7220fd95a4
    Closes-Bug: 1443291

[Anastasia Palkina (apalkina)]

Verified on ISO #329

"build_id": "2015-04-20_22-54-31", "ostf_sha": "df8db1f48f03b18126ce5ec65317a1eb83a5a95f", "build_number": "329", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-20_22-54-31", "ostf_sha": "df8db1f48f03b18126ce5ec65317a1eb83a5a95f", "build_number": "329", "api": "1.0", "nailgun_sha": "69fca3bada96a48ab19b969621e5128390b7052e", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "b4f1ddffd5263489090b65e662173e9e11aafd94", "astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0c0e299dcc0276bb8262256148c6ee975857ee86", "fuellib_sha": "b878f1ae8e1d4c6f2f463e810b1f8014df5b22a5"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "69fca3bada96a48ab19b969621e5128390b7052e", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "b4f1ddffd5263489090b65e662173e9e11aafd94", "astute_sha": "bf1751a4fe0d912325e3b4af629126a59c1b2b51", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0c0e299dcc0276bb8262256148c6ee975857ee86", "fuellib_sha": "b878f1ae8e1d4c6f2f463e810b1f8014df5b22a5"

[root@node-6 ~]# chkconfig | grep mcollective
mcollective 0:off 1:off 2:on 3:on 4:on 5:on 6:off