tripleo-ansible ceph roles badly inherrit ansible environment variables

[John Fulton]

Executions of ceph-related nested ansible in tripleo-ansible can result in unwanted newlines which break command parsing.

For example at point "2020-02-17 05:27:24" in a patche's log [1] the command made by [2] looks like it's getting additional new lines added which breaks the interpolation of the command. At least it looks like an extra newline problem to me because I see additional date entries in the log where I should only see one:

2020-02-17 05:27:24 | cmd: (blah blah blah)
2020-02-17 05:27:24 | ANSIBLE_CACHE_PLUGIN=...
2020-02-17 05:27:24 | ANSIBLE_ROLES_PATH=...

I think (from lines 24/25 of [1]) that the following variables contain new lines:

      - "{{ calling_ansible_environment_variables|join(' ') }}"
      - "{{ ceph_ansible_environment_variables|join(' ') }}"

Most likely the calling_ansible_environment_variables which is basically getting whatever the config-download ansible got.

Either we require these vars to not contain new lines or we modify lines 24/25 to go from this:

  - "{{ calling_ansible_environment_variables|join(' ') }}"

to something like this:

  - "{{ calling_ansible_environment_variables|join(' ')|replace('\n', '') }}"

[1] https://logserver.rdoproject.org/15/706815/7/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/a444f63/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[2] https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_ceph_uuid/tasks/gather.yml

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/708252

[John Fulton (jfulton-org)]

This problem is more widespread than just here:

 https://logserver.rdoproject.org/15/706815/7/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/a444f63/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

It seems to have broken 010 in general. For example:

https://caf2f172638deaa44bce-d94eb819944fd5bc8105d713aef77d0b.ssl.cf1.rackcdn.com/707935/1/gate/tripleo-ci-centos-7-scenario010-standalone/817a10a/logs/undercloud/home/zuul/standalone_deploy.log

Seems to be more than just new lines. Could be this:

 https://review.opendev.org/#/c/705793/21/tripleoclient/utils.py@360

The method of inheritance introduced in I428f6deb416b540dae552b5fc7a778cbc7505e8c is fragile.

[John Fulton (jfulton-org)]

New plan:

1. do not inherit by default (this will fix recent breaks)
2. improve how we inherit
3. probably continue to not inherit by default

[John Fulton (jfulton-org)]

To improve how the env is inherited, Alex pointed out that `env`, as used in [1], is not going to contain quoted variables and now that ANSIBLE_SSH_ARGS is getting set with multiple "-o foo=bar" [2] we need to have those quotes in order for the inheritance to not break. A better way to fix this is to use something like [3].

[1] https://review.opendev.org/#/c/674706/4/tripleo_ansible/roles/tripleo-ceph-common/tasks/main.yml

[2] https://review.opendev.org/#/c/705793/21/tripleoclient/utils.py@360

[3] https://review.opendev.org/#/c/706143/15/roles/tripleo_overcloud_image_build/tasks/main.yml@13

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708252
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=692678dce5b7f4950387c154086a0c1cdc6b887b
Submitter: Zuul
Branch: master

commit 692678dce5b7f4950387c154086a0c1cdc6b887b
Author: John Fulton <email address hidden>
Date: Mon Feb 17 14:51:52 2020 -0500

    Do not inherit the calling ansible env by default

    As per I428f6deb416b540dae552b5fc7a778cbc7505e8c ANSIBLE_*
    environment variables are passed to Ansible executions which
    are used to deploy Ceph.

    The method of inheritance is fragile and possibly unsafe. This
    patch disables inheritance by default. In inheritance feature
    can be made more safe in subsequent patches for Related-Bug but
    but this new default should stay.

    Change-Id: Ifa6d147664fc993ca08c8fb2efa860f7605628cb
    Fixes-Bug: #1863629

[Marios Andreou (marios-b)]

more examples there

                 * https://f1b67923f9334b17f05d-278276a93d7fc7ef4213a4b26089e78a.ssl.cf1.rackcdn.com/708137/1/gate/tripleo-ci-centos-7-scenario004-standalone/70df41a/logs/undercloud/home/zuul/standalone_deploy.log
                        * 2020-02-18 03:08:21 | TASK [tripleo_ceph_uuid : run nodes-uuid command] ******************************
                        * 2020-02-18 03:08:21 | fatal: [undercloud]: FAILED! => changed=true

                * http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_68d/708137/1/gate/tripleo-ci-centos-7-scenario001-standalone/68de2e5/logs/undercloud/home/zuul/standalone_deploy.log
                        * 2020-02-18 03:06:51 | TASK [tripleo_ceph_uuid : run nodes-uuid command] ******************************
                        * 2020-02-18 03:06:52 | fatal: [undercloud]: FAILED! => changed=true
                        * 2020-02-18 03:06:52 | stderr: '/bin/sh: -o: command not found'
                        * 2020-02-18 03:06:53 | Exception: Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2

looks like https://review.opendev.org/#/c/708252/ merged so rechecking my patch thanks for the quick fix folks

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/708485

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708485
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=1a3620532e29a42ad4e05570979f9fee6725ddd7
Submitter: Zuul
Branch: stable/train

commit 1a3620532e29a42ad4e05570979f9fee6725ddd7
Author: John Fulton <email address hidden>
Date: Mon Feb 17 14:51:52 2020 -0500

    Do not inherit the calling ansible env by default

    As per I428f6deb416b540dae552b5fc7a778cbc7505e8c ANSIBLE_*
    environment variables are passed to Ansible executions which
    are used to deploy Ceph.

    The method of inheritance is fragile and possibly unsafe. This
    patch disables inheritance by default. In inheritance feature
    can be made more safe in subsequent patches for Related-Bug but
    but this new default should stay.

    Change-Id: Ifa6d147664fc993ca08c8fb2efa860f7605628cb
    Fixes-Bug: #1863629
    (cherry picked from commit 692678dce5b7f4950387c154086a0c1cdc6b887b)