Race condition in mcollective puppetd module

[Maksym Strukov]

Test failed with:

Error Message

Found 2 lines like PLUGIN: fuel_plugin_example_v3 - deploy.pp but should be only 1 in /var/log/puppet.log Command 'tail -n +206 /var/log/puppet.log | grep -n "PLUGIN: fuel_plugin_example_v3 - deploy.pp"' executed with exit_code='0'
stdout:
* 127:2016-06-16 05:15:55 +0000 Scope(Class[main]) (notice): PLUGIN: fuel_plugin_example_v3 - deploy.pp

131:2016-06-16 05:15:57 +0000 Scope(Class[main]) (notice): PLUGIN: fuel_plugin_example_v3 - deploy.pp
 *
stderr:
'* *
Stacktrace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.fuel_plugin_example/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.fuel_plugin_example/fuelweb_test/tests/plugins/plugin_example/test_fuel_plugin_example.py", line 217, in deploy_ha_one_controller_neutron_example_v3
    connect_str])
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.fuel_plugin_example/fuelweb_test/helpers/checkers.py", line 1114, in check_log_lines_order
    '\n'.join(result['stderr'])))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Found 2 lines like PLUGIN: fuel_plugin_example_v3 - deploy.pp but should be only 1 in /var/log/puppet.log Command 'tail -n +206 /var/log/puppet.log | grep -n "PLUGIN: fuel_plugin_example_v3 - deploy.pp"' executed with exit_code='0'
stdout:
* 127:2016-06-16 05:15:55 +0000 Scope(Class[main]) (notice): PLUGIN: fuel_plugin_example_v3 - deploy.pp

131:2016-06-16 05:15:57 +0000 Scope(Class[main]) (notice): PLUGIN: fuel_plugin_example_v3 - deploy.pp
 *
stderr:
'* *

Env: 9.0-mos-490
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.fuel_plugin_example/146/testReport/(root)/deploy_ha_one_controller_neutron_example_v3/deploy_ha_one_controller_neutron_example_v3/

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Kyrylo Galanov (kgalanov)]

The bug was not reproduced on CI anymore.

[Vladimir Khlyunev (vkhlyunev)]

It was shadowed by another issue - now it's reproduces:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.fuel_plugin_example/150/testReport/(root)/deploy_ha_one_controller_neutron_example_v3/deploy_ha_one_controller_neutron_example_v3/

Snapshot https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.fuel_plugin_example/150/artifact/logs/fail_error_deploy_ha_one_controller_neutron_example_v3-fuel-snapshot-2016-06-20_05-05-11.tar.gz

Doubled log message can indicate that the task was executed twice

[Kyrylo Galanov (kgalanov)]

Plugin manifest was applied twice because first run reported an error, however actually it was successful:

nailgun.test.domain.local/var/log/astute/astute.log-2016-06-20 04:32:54 DEBUG [29910] Node[1]: Node 1: task fuel_plugin_example_v3-controller-deployment, task status running
nailgun.test.domain.local/var/log/astute/astute.log:2016-06-20 04:32:54 DEBUG [29910] Node 1(fuel_plugin_example_v3-controller-deployment) status: error
nailgun.test.domain.local/var/log/astute/astute.log-2016-06-20 04:32:54 DEBUG [29910] Puppet on node 1 will be restarted. 1 retries remained.
nailgun.test.domain.local/var/log/astute/astute.log-2016-06-20 04:32:54 INFO [29910] Retrying to run puppet for following error nodes: 1

[Kyrylo Galanov (kgalanov)]

https://review.openstack.org/#/c/335319/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/335319
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=8402425be1ad71cd5f41267bda70c064828b077b
Submitter: Jenkins
Branch: master

commit 8402425be1ad71cd5f41267bda70c064828b077b
Author: Kyrylo Galanov <email address hidden>
Date: Thu Jun 30 06:01:31 2016 +0200

    Fix race condition in get puppet status

    Check if puppet is running or not before reading last_summary

    Closes-bug: #1593166
    Change-Id: Ie8d68b570109277ba3324d6404d626bc98eb44ba

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-astute 10.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-astute 10.0.0 release.