[10.0-community.main.ubuntu.bvt_2-1422] Mcollective is not started during boot process

[Andrii Kroshchenko]

Job URL: https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1412/
Snapshot URL: https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1412/artifact/logs/fail_error_ceph_rados_gw-fuel-snapshot-2017-04-09_00-25-36.tar

FAIL: Deploy ceph HA with RadosGW for objects
-----
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 509, in ceph_rados_gw
    self.fuel_web.verify_network(cluster_id)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 367, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 204, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 1629, in verify_network
    self.assert_task_success(task, timeout, interval=10)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 204, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 349, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'verify_networks' has incorrect status. error != ready, 'Method verify_networks. Network verification not available because nodes ["1"] not available via mcollective.
Inspect Astute logs for the details'
-----

node-1
-----
ls -la
total 1008
drwxr-xr-x. 5 kaslcrof kaslcrof 4096 Apr 13 11:36 .
drwxr-xr-x. 3 kaslcrof kaslcrof 4096 Apr 13 11:04 ..
-rw-r--r--. 1 kaslcrof kaslcrof 22554 Apr 9 03:07 alternatives.log
drwxr-xr-x. 2 kaslcrof kaslcrof 4096 Apr 13 11:04 apt
-rw-r-----. 1 kaslcrof kaslcrof 13553 Apr 9 03:25 auth.log
-rw-r--r--. 1 kaslcrof kaslcrof 226827 Apr 9 03:05 bootstrap.log
-rw-------. 1 kaslcrof kaslcrof 0 Apr 9 03:05 btmp
-rw-r-----. 1 kaslcrof kaslcrof 31 Apr 9 03:05 dmesg
-rw-r--r--. 1 kaslcrof kaslcrof 386977 Apr 9 03:07 dpkg.log
-rw-r--r--. 1 kaslcrof kaslcrof 3488 Apr 9 03:07 faillog
drwxr-xr-x. 2 kaslcrof kaslcrof 4096 Apr 13 11:04 fsck
-rw-r-----. 1 kaslcrof kaslcrof 46383 Apr 9 03:25 kern.log
-rw-rw-r--. 1 kaslcrof kaslcrof 31828 Apr 9 03:07 lastlog
drwxr-xr-x. 2 kaslcrof kaslcrof 4096 Apr 13 11:04 live
-rw-r-----. 1 kaslcrof kaslcrof 66133 Apr 9 03:25 messages
-rw-r--r--. 1 kaslcrof kaslcrof 52694 Apr 9 03:25 nailgun-agent.log
-rw-r-----. 1 kaslcrof kaslcrof 128713 Apr 9 03:25 syslog
-rw-rw-r--. 1 kaslcrof kaslcrof 6144 Apr 9 03:19 wtmp

mcollective.log - does not exist

grep -i "collective" syslog
-----
2017-04-09T00:19:14Z info kernel: [ 0.000000] Command line: initrd=/images/ubuntu_bootstrap/initrd.img ksdevice=bootif lang= console=ttyS0,9600 console=tty0 toram locale=en_US text fetch=http://10.109.20.2:8080/bootstraps/active_bootstrap/root.squashfs boot=live mco_user=mcollective priority=critical url=http://10.109.20.2:8000/api biosdevname=0 components ip=frommedia ethdevice-timeout=120 net.ifnames=1 panic=60 mco_pass=8i6D4W6y1CZn7Zpv7OKH6BRc auto-install/enable=true priority=critical url=http://10.109.20.2/cblr/svc/op/ks/profile/ubuntu_bootstrap hostname=ubuntubootstrap domain=local.lan suite=xenial BOOTIF=01-64-e7-89-48-34-4f BOOT_IMAGE=/images/ubuntu_bootstrap/vmlinuz
2017-04-09T00:19:14Z notice kernel: [ 0.000000] Kernel command line: initrd=/images/ubuntu_bootstrap/initrd.img ksdevice=bootif lang= console=ttyS0,9600 console=tty0 toram locale=en_US text fetch=http://10.109.20.2:8080/bootstraps/active_bootstrap/root.squashfs boot=live mco_user=mcollective priority=critical url=http://10.109.20.2:8000/api biosdevname=0 components ip=frommedia ethdevice-timeout=120 net.ifnames=1 panic=60 mco_pass=8i6D4W6y1CZn7Zpv7OKH6BRc auto-install/enable=true priority=critical url=http://10.109.20.2/cblr/svc/op/ks/profile/ubuntu_bootstrap hostname=ubuntubootstrap domain=local.lan suite=xenial BOOTIF=01-64-e7-89-48-34-4f BOOT_IMAGE=/images/ubuntu_bootstrap/vmlinuz
-----

MCollective service was not started

But package is installed:
-----
dpkg.log:2017-04-09 00:06:23 install mcollective-common:all <none> 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:06:23 status half-installed mcollective-common:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:06:23 status unpacked mcollective-common:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:06:23 status unpacked mcollective-common:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:06:23 install mcollective:all <none> 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:06:23 status half-installed mcollective:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:06:23 status unpacked mcollective:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:06:23 status unpacked mcollective:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 configure mcollective-common:all 2.6.0+dfsg-2.1 <none>
dpkg.log:2017-04-09 00:07:14 status unpacked mcollective-common:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 status unpacked mcollective-common:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 status unpacked mcollective-common:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 status half-configured mcollective-common:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 status installed mcollective-common:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 configure mcollective:all 2.6.0+dfsg-2.1 <none>
dpkg.log:2017-04-09 00:07:14 status unpacked mcollective:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 status unpacked mcollective:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 status unpacked mcollective:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 status half-configured mcollective:all 2.6.0+dfsg-2.1
dpkg.log:2017-04-09 00:07:14 status installed mcollective:all 2.6.0+dfsg-2.1
-----

node-3
ls -la
total 1036
drwxr-xr-x. 5 kaslcrof kaslcrof 4096 Apr 13 11:49 .
drwxr-xr-x. 3 kaslcrof kaslcrof 4096 Apr 13 11:04 ..
-rw-r--r--. 1 kaslcrof kaslcrof 22554 Apr 9 03:07 alternatives.log
drwxr-xr-x. 2 kaslcrof kaslcrof 4096 Apr 13 11:04 apt
-rw-r-----. 1 kaslcrof kaslcrof 13553 Apr 9 03:25 auth.log
-rw-r--r--. 1 kaslcrof kaslcrof 226827 Apr 9 03:05 bootstrap.log
-rw-------. 1 kaslcrof kaslcrof 0 Apr 9 03:05 btmp
-rw-r-----. 1 kaslcrof kaslcrof 31 Apr 9 03:05 dmesg
-rw-r--r--. 1 kaslcrof kaslcrof 386977 Apr 9 03:07 dpkg.log
-rw-r--r--. 1 kaslcrof kaslcrof 3488 Apr 9 03:07 faillog
drwxr-xr-x. 2 kaslcrof kaslcrof 4096 Apr 13 11:04 fsck
-rw-r-----. 1 kaslcrof kaslcrof 46383 Apr 9 03:25 kern.log
-rw-rw-r--. 1 kaslcrof kaslcrof 31828 Apr 9 03:07 lastlog
drwxr-xr-x. 2 kaslcrof kaslcrof 4096 Apr 13 11:04 live
-rw-r--r--. 1 kaslcrof kaslcrof 30791 Apr 9 03:25 mcollective.log
-rw-r-----. 1 kaslcrof kaslcrof 66133 Apr 9 03:25 messages
-rw-r--r--. 1 kaslcrof kaslcrof 52694 Apr 9 03:25 nailgun-agent.log
-rw-r-----. 1 kaslcrof kaslcrof 126280 Apr 9 03:25 syslog
-rw-rw-r--. 1 kaslcrof kaslcrof 6144 Apr 9 03:19 wtmp

grep -i "collective" syslog
-----
2017-04-09T00:19:00Z info kernel: [ 0.000000] Command line: initrd=/images/ubuntu_bootstrap/initrd.img ksdevice=bootif lang= console=ttyS0,9600 console=tty0 toram locale=en_US text fetch=http://10.109.20.2:8080/bootstraps/active_bootstrap/root.squashfs boot=live mco_user=mcollective priority=critical url=http://10.109.20.2:8000/api biosdevname=0 components ip=frommedia ethdevice-timeout=120 net.ifnames=1 panic=60 mco_pass=8i6D4W6y1CZn7Zpv7OKH6BRc auto-install/enable=true priority=critical url=http://10.109.20.2/cblr/svc/op/ks/profile/ubuntu_bootstrap hostname=ubuntubootstrap domain=local.lan suite=xenial BOOTIF=01-64-e6-07-bd-5b-36 BOOT_IMAGE=/images/ubuntu_bootstrap/vmlinuz
2017-04-09T00:19:00Z notice kernel: [ 0.000000] Kernel command line: initrd=/images/ubuntu_bootstrap/initrd.img ksdevice=bootif lang= console=ttyS0,9600 console=tty0 toram locale=en_US text fetch=http://10.109.20.2:8080/bootstraps/active_bootstrap/root.squashfs boot=live mco_user=mcollective priority=critical url=http://10.109.20.2:8000/api biosdevname=0 components ip=frommedia ethdevice-timeout=120 net.ifnames=1 panic=60 mco_pass=8i6D4W6y1CZn7Zpv7OKH6BRc auto-install/enable=true priority=critical url=http://10.109.20.2/cblr/svc/op/ks/profile/ubuntu_bootstrap hostname=ubuntubootstrap domain=local.lan suite=xenial BOOTIF=01-64-e6-07-bd-5b-36 BOOT_IMAGE=/images/ubuntu_bootstrap/vmlinuz
2017-04-09T00:21:02Z info systemd[1]: Stopped The Marionette Collective.
2017-04-09T00:21:02Z info systemd[1]: Starting The Marionette Collective...
2017-04-09T00:21:02Z info systemd[1]: Started The Marionette Collective.
-----

MCollective was started.