Globals yaml incorrectly fetches data from override files

[Alexandr Kostrikov]

During swarm run there were two similar failures [0] [1]

Probably, there is a need to update unlock settings tab to work better with networking issues.

Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[netconfig/1], Task[netconfig/2] Stopping the deployment process!'
Task 'deployment' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[connectivity_tests/1], Task[connectivity_tests/2] Stopping the deployment process!'

[0] https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.unlock_settings_tab_states/16/testReport/(root)/failed_deploy_unlock/failed_deploy_unlock/
[1] https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.unlock_settings_tab_states/16/testReport/(root)/partially_deployed_unlock/partially_deployed_unlock/

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alexander Kurenyshev (akurenyshev)]

I've looked on the one failed test, because it plays with netconfig task by adding an expected fail to its puppet. And I found that test failed on the next step, when we're re-deploying cluster with correct task:
[root@nailgun ~]# cat /etc/puppet/modules/osnailyfacter/modular/netconfig/connectivity_tests.pp
include ::osnailyfacter::netconfig::connectivity_tests

As you can see there is no fail emulation at the output.

So, this bug is not on QA side, something wrong with connectivity tests.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alex Schultz (alex-schultz)]

From the logs...

2016-06-08T16:29:28.943641+00:00 err: Duplicate declaration: L3_clear_route[default] is already declared; cannot redeclare at /etc/puppet/modules/osnailyfacter/manifests/netconfig/netconfi
g.pp:24 on node node-2.test.domain.local

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/327528

[Aleksandr Didenko (adidenko)]

Between those two deployment runs "Assign public network to all nodes" feature was enabled. So during the first deployment default gateway was overrided to vrouter VIP:

2016-06-08 00:56:47 +0000 /Stage[main]/Osnailyfacter::Netconfig::Hiera_default_route/File[/etc/hiera/override/configuration/default_route.yaml]/ensure (notice): defined content as '{md5}c091969d5740a511dfcb9ffd5164bffd'

And during the second run gateway override was removed as expected:

2016-06-08 00:59:49 +0000 /Stage[main]/Osnailyfacter::Netconfig::Hiera_default_route/File[/etc/hiera/override/configuration/default_route.yaml]/ensure (notice): removed

But one thing was not expected - globals.yaml. Task 'globals.pp' is being run before 'netconfig/hiera_default_route.pp' and it "cached" whole network_scheme with both default gateways.

In order to fix this and other LCM cases related to network scheme changes we should remove 'network_scheme' from globals.yaml.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/327569

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/327923

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327787
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=db794bfee433c9ee7b62931934cbff2dba93176e
Submitter: Jenkins
Branch: master

commit db794bfee433c9ee7b62931934cbff2dba93176e
Author: Vladimir Kuklin <email address hidden>
Date: Thu Jun 9 18:56:01 2016 +0300

    Disable all the hiera yamls for globals except for astute.yaml

    We need to ensure that globals uses only astute.yaml to generate
    its helper variables. Othewise, we will have an issue on the second
    deployment if we modify astute.yaml and re-run the deployment

    For globals.pp disable_globals variable is used to redirect hiera
    into non-existing files and thus skip their inclusion.

    Change-Id: I91ab492b44c137c476722f65de3dd99b2dce988a
    Closes-bug: #1590375

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327528
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b041e6eb28a5f65652a0f1954a3c3d44e3ef577a
Submitter: Jenkins
Branch: master

commit b041e6eb28a5f65652a0f1954a3c3d44e3ef577a
Author: Sergey Vasilenko <email address hidden>
Date: Thu Jun 9 12:21:38 2016 +0300

    Remove duplicate save of network_scheme from globals

    Such save leads to cache non-actual values into hiera
    and fails some LCM-cases

    Change-Id: I4c96ba8fe2d262ed9c0ffdb64a375f490029c2a2
    Closes-bug: #1590375

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327569
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4234d043bee7a103eb967241b3b4164cbf731b4e
Submitter: Jenkins
Branch: stable/mitaka

commit 4234d043bee7a103eb967241b3b4164cbf731b4e
Author: Sergey Vasilenko <email address hidden>
Date: Thu Jun 9 12:21:38 2016 +0300

    Remove duplicate save of network_scheme from globals

    Such save leads to cache non-actual values into hiera
    and fails some LCM-cases

    Change-Id: I4c96ba8fe2d262ed9c0ffdb64a375f490029c2a2
    Closes-bug: #1590375

[Ekaterina Shutova (eshutova)]

Now swarm tests are passed:
https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.unlock_settings_tab_states/25/
Verified on:
9.0-mos-490

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/340686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/340686
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0aed35fb32796bb2db3364da114a50c291704dd2
Submitter: Jenkins
Branch: master

commit 0aed35fb32796bb2db3364da114a50c291704dd2
Author: Kyrylo Galanov <email address hidden>
Date: Tue Jul 12 07:22:55 2016 +0200

    Disable all hiera override yamls for globals

    We need to ensure that globals uses only astute.yaml to generate its
    helper variables. Othewise, we will have an issue on the second
    deployment if we modify astute.yaml and re-run the deployment

    Relates to I91ab492b44c137c476722f65de3dd99b2dce988a

    Change-Id: Ieef42f063911ed05ce05f1411bbaa1d4867aa455
    Related-Bug: #1590375
    Closes-bug: #1598163

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/327923
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e852692f4ceca97dd42ba16835f7c756a8bbe6a5
Submitter: Jenkins
Branch: stable/mitaka

commit e852692f4ceca97dd42ba16835f7c756a8bbe6a5
Author: Vladimir Kuklin <email address hidden>
Date: Thu Jun 9 18:56:01 2016 +0300

    Disable all the hiera yamls for globals except for astute.yaml

    We need to ensure that globals uses only astute.yaml to generate
    its helper variables. Othewise, we will have an issue on the second
    deployment if we modify astute.yaml and re-run the deployment

    For globals.pp disable_globals variable is used to redirect hiera
    into non-existing files and thus skip their inclusion.

    Change-Id: I91ab492b44c137c476722f65de3dd99b2dce988a
    Closes-bug: #1590375

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/354064

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/354064
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=376f82c3ad2b442e78026910e507d1ec87226945
Submitter: Jenkins
Branch: stable/mitaka

commit 376f82c3ad2b442e78026910e507d1ec87226945
Author: Kyrylo Galanov <email address hidden>
Date: Tue Jul 12 07:22:55 2016 +0200

    Disable all hiera override yamls for globals

    We need to ensure that globals uses only astute.yaml to generate its
    helper variables. Othewise, we will have an issue on the second
    deployment if we modify astute.yaml and re-run the deployment

    Relates to I91ab492b44c137c476722f65de3dd99b2dce988a

    Change-Id: Ieef42f063911ed05ce05f1411bbaa1d4867aa455
    Related-Bug: #1590375
    Closes-bug: #1598163
    (cherry picked from commit 0aed35fb32796bb2db3364da114a50c291704dd2)