conntrackd task failed

[Vladyslav Drok]

Master pkg systest run 725 on iso 152 https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-ubuntu/725/ failed with the following error:

FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 502, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 455, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 440, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 491, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 498, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 382, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 827, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-keystone/1], Task[conntrackd/1] Stopping the deployment process!'

primary-keystone failure is another bug 1577839, as for the conntrackd, here are some log snippets:

nailgun.test.domain.local/var/log/remote/node-1.test.domain.local/ocf-ns_conntrackd.log is full of the following entries:
2016-05-05T10:09:32.812623+00:00 err: ERROR: Setup problem: couldn't find command: conntrackd

/home/vdrok/Downloads/fuel-snapshot-2016-05-05_11-23-05/nailgun.test.domain.local/var/log/remote/node-1.test.domain.local/pengine.log also contains lots of the following:
2016-05-05T10:09:33.275619+00:00 notice: notice: Preventing master_p_conntrackd from re-starting on node-1.test.domain.local: operation monitor failed 'not installed' (5)

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Kyrylo Galanov (kgalanov)]

After p_conntrackd was configured on any of cluster nodes all other nodes start polling ocf ns_conntrackd script locally. Under certain circumstances this task can be launched on nodes with significant time difference. As a result, pacemaker runs ocf ns_conntrackd script before conntrackd was installed.

This bug has no impact on deployment. I am decreasing the priority to medium.

[Matthew Mosesohn (raytrac3r)]

All pacemaker based services on non-primary controller will report "cannot find command". That's because we join nodes to the cluster and they automatically attempt to start the services that all nodes should run. This error is expected and then the service will be installed, cleaned up in pacemaker, and started. There is nothing to fix here.