config-download: --start-at-task usually fails b/c gather_facts has been skipped

[James Slagle]

If you use --start-at-task with ansible-playbook and config-download, the task (or a subsequent one) usually fails pretty quickly because gather_facts has been skipped.

We gather_facts once for all hosts as one of the first tasks, and subsequent tasks are heavily dependent on those facts having been gathered. If you then use --start-at-task, there's a good chance you skip gather_facts and your tasks all fail.

Whether or not we gather_facts on each task ought to be configurable with a variable that can be overridden from the cli. It should default to False to preserve the current behavior.

We should also use a fact cache, so that if we do skip gather_facts, things at least have a chance of working. The fact cache should live in the working dir of config-download, which is where the playbooks/tasks have been downloaded (under /var/lib/mistral/... when using the workflow).

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/548651

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/548651
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=b7baf62e0f58a95ae4e48b1e728e3f670a2d5ca4
Submitter: Zuul
Branch: master

commit b7baf62e0f58a95ae4e48b1e728e3f670a2d5ca4
Author: David J Peacock <email address hidden>
Date: Wed Feb 28 11:17:48 2018 -0500

    add option --gather-facts

    Adding option `--gather-facts` for use with `--start-at-task`. Without
    this option, as described in lp1749460, plays usually fail because facts
    are gathered only once at the beginning of a config-download deployment.

    Change-Id: I3d2c16ffa29b521dc5b0c9a920eba7cf04c444ed
    Related-Bug: 1749460

[Bogdan Dobrelya (bogdando)]

I think this issue now came again, complaining about python_cmd unset. And if I ran
it with '-e gather_facts=true -e python_cmd=python3', then it fails cuz of

TASK [Wait for container-puppet tasks (generate config) to finish] **************************************************************************************************************
525400e3-e3d8-a92c-7375-0000000003e3 | TIMING | Run container-puppet tasks (generate config) during step 1 with paunch | 0:00:39.416 | 0.48s
fatal: [standalone]: FAILED! => {"ansible_job_id": "116559345031.256801", "attempts": 1, "changed": true, "cmd": "python3 /var/lib/container-puppet/container-puppet.py", "delta": "0:00:00.045215", "end": "2020-06-02 15:31:53.857886", "finished": 1, "msg": "non-zero return code", "rc": 2, "start": "2020-06-02 15:31:53.812671", "stderr": "python3: can't open file '/var/lib/container-puppet/container-puppet.py': [Errno 2] No such file or directory", "stderr_lines": ["python3: can't open file '/var/lib/container-puppet/container-puppet.py': [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []}

but there is no more paunch in master branch. Weird.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/732730

[Bogdan Dobrelya (bogdando)]

https://review.opendev.org/#/c/732730

[Bogdan Dobrelya (bogdando)]

So with https://review.opendev.org/732730, I was able to run deployment until it failed with

TASK [tripleo_keystone_resources : Create Keystone Projects] ********************************************************************************************************************
task path: /usr/share/ansible/roles/tripleo_keystone_resources/tasks/main.yml:38
525400e3-e3d8-27e3-b6d8-0000000085f4 | TIMING | tripleo_keystone_resources : Create identity admin endpoint | 0:31:27.350 | 1.66s
fatal: [undercloud]: FAILED! => {"msg": "'keystone_resources' is undefined"}

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https://review.opendev.org/732730
Reason: produces no changes after rebase (paunch has been removed)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/734073

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/734073
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=acc12770bbd6e3d54e61f97209fe8b2370923a58
Submitter: Zuul
Branch: stable/train

commit acc12770bbd6e3d54e61f97209fe8b2370923a58
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Jun 2 18:03:32 2020 +0200

    [TRAIN-ONLY] Be explicit when passing vars into deploy steps

    This continues Ia630f08f553bd53656c76e5c8059f15d314a17c0

    Implicit defaults hide issues with overring ansible variables as we
    pass values in from deploy-steps.j2.

    Make no implicit defaults for variables passed into deploy steps via
    ansible vars. Only expect those take the values defined in the caller
    deploy-steps.j2 playbook template. Add missing params and vars for
    templates to propagate ansible values for external deploy/upgrade,
    upgrade/update and post upgrade steps playbooks.

    Fix enable_paunch converted to a boolean.

    Change-Id: I66391d9d193f6475f59544220412f7baaf7fcba5
    Related-Bug: #1799914
    Related-Bug: #1749460
    Signed-off-by: Bogdan Dobrelya <email address hidden>