hiera hosts aren't the same for different group of roles

[Artem Hrechanychenko]

Detailed bug description:
 hiera hosts aren't the same for different group of roles
 Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/fuelweb_test/tests/tests_separate_services/test_separate_keystone.py", line 283, in separate_keystone_service_add_delete_node
    cmd='hiera corosync_roles')
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/fuelweb_test/helpers/checkers.py", line 1138, in check_hiera_hosts
    ' others'.format(node['hostname']))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Hosts on node node-4 differ from others

Steps to reproduce:
 1. Revert snapshot separate_keystone_service
2. Add one keystone node and re-deploy cluster
3. Run network verification
4. Run OSTF
5. Check hiera hosts are the same for
different group of roles
6. Delete one keystone node
7. Run network verification
8. Run ostf
9. Check hiera hosts are the same for
different group of roles <<<<<<<<<<<<<<failed here

Expected results:
 hiera hosts are the same for
different group of roles
Actual result:
 AssertionError: Hosts on node node-4 differ from others
Reproducibility:
 https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/85/testReport/%28root%29/separate_keystone_service_add_delete_node/separate_keystone_service_add_delete_node/
Workaround:
 -
Impact:
 swarm

[Nastya Urlapova (aurlapova)]

There is other issue since 9.0-294 build, this one moved to invalid.

[Nastya Urlapova (aurlapova)]

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/106/testReport/(root)/separate_keystone_service_add_delete_node/separate_keystone_service_add_delete_node/

[Alexandr Kostrikov (akostrikov-mirantis)]

Reproduced at https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/110/testReport/(root)/separate_keystone_service_add_delete_node/separate_keystone_service_add_delete_node/

Possbile duplicate of https://bugs.launchpad.net/fuel/+bug/1542307 or https://bugs.launchpad.net/fuel/+bug/1485505

[Volodymyr Shypyguzov (vshypyguzov)]

Reproduced again on iso #481

Scenario:

https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/142/console

I seems, that nodes with two custome roles have only one corosync_role
Here is the difference:

[root@nailgun ~]# fuel node | grep keystone
8 | ready | slave-09_standalone-keystone | 1 | 10.109.5.11 | 64:a8:3b:9e:ee:dd | standalone-keystone | | 1 | 1
5 | ready | slave-05_standalone-database_standalone-keystone | 1 | 10.109.5.7 | 64:b6:b1:ba:a1:7a | standalone-database, standalone-keystone | | 1 | 1
3 | ready | slave-04_standalone-database_standalone-keystone | 1 | 10.109.5.6 | 64:a9:c0:f6:29:17 | standalone-database, standalone-keystone | | 1 | 1
7 | ready | slave-06_standalone-database_standalone-keystone | 1 | 10.109.5.8 | 64:b0:b6:69:6f:26 | standalone-database, standalone-keystone | | 1 | 1
[root@nailgun ~]# for id in 3 5 7 8; do ssh node-$id hiera corosync_roles; done
Warning: Permanently added 'node-3' (ECDSA) to the list of known hosts.
["primary-standalone-database", "standalone-database"]
Warning: Permanently added 'node-5' (ECDSA) to the list of known hosts.
["primary-standalone-database", "standalone-database"]
Warning: Permanently added 'node-7' (ECDSA) to the list of known hosts.
["primary-standalone-database", "standalone-database"]
Warning: Permanently added 'node-8' (ECDSA) to the list of known hosts.
["primary-standalone-keystone", "standalone-keystone"]

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 481
cat /etc/fuel_build_number:
 481
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8458.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8458.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-notify-9.0.0-1.mos8458.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8458.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8742.noarch
 fuel-library9.0-9.0.0-1.mos8458.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos936.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8742.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 fuel-openstack-metadata-9.0.0-1.mos8742.noarch

[Matthew Mosesohn (raytrac3r)]

There is no way to define priority for plugins, so their order in /etc/hiera.yaml is non-deterministic. It could appear in any order. The corosync_roles values could be different when combining roles. As long as the roles are deployed correctly (all detached services combined together), deployment is fine.

Since the check is quite simple: checkers.check_hiera_hosts(keystone_nodes,cmd='hiera corosync_roles'), it's best to just disable this test. I would adjust this test to ensure all entries contain the string "standalone-", but that is more effort.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/331145

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/331607

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/331145
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=54173f659d84e03ce74bcc1aeaaae69b7186b882
Submitter: Jenkins
Branch: master

commit 54173f659d84e03ce74bcc1aeaaae69b7186b882
Author: Alexander Kurenyshev <email address hidden>
Date: Fri Jun 17 16:45:58 2016 +0300

    Fix checking of hiera corosync_roles

    Change-Id: Ic8285bda80db59bdf3cc5d64e4e62388f806f505
    Closes-Bug:1573502

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/331607
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=889459e5cc06b95fde5e1a570c7d93a594879ab6
Submitter: Jenkins
Branch: stable/mitaka

commit 889459e5cc06b95fde5e1a570c7d93a594879ab6
Author: Alexander Kurenyshev <email address hidden>
Date: Fri Jun 17 16:45:58 2016 +0300

    Fix checking of hiera corosync_roles

    Change-Id: Ic8285bda80db59bdf3cc5d64e4e62388f806f505
    Closes-Bug:1573502

[Alexander Kurenyshev (akurenyshev)]

So, we have merged fix for a detached keystone plugin, for other tests an original check works fine. But we could modify it to be more stable, suggested change will work like this:
for i in 3 4 7 8; do ssh node-$i grep corosync_roles /etc/hiera/plugins/detach-*.yaml -A2 2>/dev/null; done
where numbers are ids for nodes with separated role.
We will cat the content of plugin's yaml file and compare it with other nodes with the same role. With this check we'll avoid hiera unpredictable output

[Nastya Urlapova (aurlapova)]

Moved to "Fix Released" since test passed check for roles:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.plugins.thread_keystone_separate_services/149/testReport/(root)/separate_keystone_service_add_delete_node/separate_keystone_service_add_delete_node/