Tripleo Ceph Deployment - The "unpack tarball to fetch directory" task fails when run config-download manually during ceph deployment.

[Francesco Pantano]

Running overcloud config-download manually with ceph deployment more times it fails executing the following ansible task:

TASK [unpack downloaded ceph-ansible fetch tarball to fetch directory]

According to the doc:

https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ansible_config_download.html#ansible-project-directory

that describe the workflow to deploy an overcloud using the config download feature, even using the workaround:

sudo setfacl -R -m u:$USER:rwx /var/lib/mistral

we got this error:

https://pastebin.com/w3HjkPVf

Also, seems like there is a bug with how fetch_dir is being created

(undercloud) [stack@undercloud fetch_dir]$ find /var/lib/mistral/overcloud/ -name fetch_dir
/var/lib/mistral/overcloud/ceph-ansible/fetch_dir
/var/lib/mistral/overcloud/ceph-ansible/fetch_dir/fetch_dir
/var/lib/mistral/overcloud/ceph-ansible/fetch_dir/fetch_dir/fetch_dir

Here the full log:

https://pastebin.com/KguUwSBb

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/637537

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637537
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=548f54b39d1e5f895af16b93646e9e7925ffd488
Submitter: Zuul
Branch: master

commit 548f54b39d1e5f895af16b93646e9e7925ffd488
Author: fpantano <email address hidden>
Date: Mon Feb 18 13:39:10 2019 +0100

    Added tar option to preserve metadata of existing fetch_dir

    Added tar option to allow the current user to unpack local
    or downloaded data (from swift) into ceph-ansible/fetch_dir,
    preserving the existing metadata.
    Added also a "/*" as trailing character on the *path field* of
    the *archive* ansible task to create a tar.gz containing all
    the fetch_dir content whithout including the parent directory
    in the resulting path: in this way we should be able to avoid
    nested fetch_dir.

    1. Added a step to fix broken environment excluding the nested
    fetch_dir from the archive sections;

    2. Added a new task to delete from filesystem the nested
    section to make everything clean: this task is executed
    according to the output of the previous one (the archive task).

    Change-Id: I45c394bd6b5c09730399202b476ed913e8f334c0
    Related-Bug: 1816094

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/640720

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640720
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5baa88d94e25f887da7a2f7f8103d52795b340ec
Submitter: Zuul
Branch: stable/rocky

commit 5baa88d94e25f887da7a2f7f8103d52795b340ec
Author: fpantano <email address hidden>
Date: Mon Feb 18 13:39:10 2019 +0100

    Added tar option to preserve metadata of existing fetch_dir

    Added tar option to allow the current user to unpack local
    or downloaded data (from swift) into ceph-ansible/fetch_dir,
    preserving the existing metadata.
    Added also a "/*" as trailing character on the *path field* of
    the *archive* ansible task to create a tar.gz containing all
    the fetch_dir content whithout including the parent directory
    in the resulting path: in this way we should be able to avoid
    nested fetch_dir.

    1. Added a step to fix broken environment excluding the nested
    fetch_dir from the archive sections;

    2. Added a new task to delete from filesystem the nested
    section to make everything clean: this task is executed
    according to the output of the previous one (the archive task).

    Change-Id: I45c394bd6b5c09730399202b476ed913e8f334c0
    Related-Bug: 1816094
    (cherry picked from commit 548f54b39d1e5f895af16b93646e9e7925ffd488)

[Cameron.C (cameron-c)]

hi,
Deploy with external ceph, wll get error
|-> Failures for host: undercloud
|--> Task: create temporary ceph-ansible fetch directory tarball for swift backup
|---> _ansible_no_log: false
|---> _ansible_parsed: true
|---> changed: false
|---> expanded_paths: ""
|---> invocation: {
    "module_args": {
        "attributes": null,
        "backup": null,
        "content": null,
        "delimiter": null,
        "dest": "/tmp/temporary_dir_new.tar.gz",
        "directory_mode": null,
        "exclude_path": [
            "/var/lib/mistral/rockycloud/ceph-ansible/fetch_dir/fetch_dir"
        ],
        "follow": false,
        "force": null,
        "format": "gz",
        "group": null,
        "mode": null,
        "owner": null,
        "path": [
            "/var/lib/mistral/rockycloud/ceph-ansible/fetch_dir/*"
        ],
        "regexp": null,
        "remote_src": null,
        "remove": false,
        "selevel": null,
        "serole": null,
        "setype": null,
        "seuser": null,
        "src": null,
        "unsafe_writes": null
    }
}
|---> msg: "Error, no source paths were found"
|---> path: "/var/lib/mistral/rockycloud/ceph-ansible/fetch_dir/*"
|---> state: "absent"

[John Fulton (jfulton-org)]

Hi Cameron,

The issue you mentioned with external ceph is also a bug, but it's a different bug which is tracked by the following:

 https://bugs.launchpad.net/tripleo/+bug/1823759

There's also a patch in the works linked from the above which you could try as a workaround until it lands.

[John Fulton (jfulton-org)]

I see this bug was switched from Fix Released → In Progress though a fix has been released where applicable. There are other related bugs, perhaps introduced by the fix for this bug, but they are separate bugs none the less and are tracked separately. Once which stands out besides 1823759 is the following:

 https://bugs.launchpad.net/tripleo/+bug/1823229

I'm setting this bug back to Fix Released.