Rename compressed text based files to end with txt.gz extension getting skipped on rdo jobs

[chandan kumar]

In almost all the jobs running on review.rdoproject.org, the logs files with .gz files are not getting renamed to .txt.gz file. While viewing the logs on log server, we are ending up downloading the logs locally.

It is making user experience miserable.

https://logserver.rdoproject.org/23/31323/5/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/ef77430/logs/quickstart_collect_logs.log

---
TASK [collect-logs : Rename compressed text based files to end with txt.gz extension] ***
task path: /home/zuul/workspace/.quickstart/share/ansible/roles/collect-logs/tasks/publish.yml:77
Thursday 10 December 2020 07:24:52 +0000 (0:00:00.077) 0:02:06.726 *****
skipping: [undercloud] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}
---
and logs files: https://logserver.rdoproject.org/23/31323/5/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/ef77430/logs/undercloud/var/log/mistral/executor.log.gz

and another example:
https://logserver.rdoproject.org/19/766419/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/5ca63de/logs/quickstart_collect_logs.log

and log files
https://logserver.rdoproject.org/19/766419/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/5ca63de/logs/undercloud/var/log/dnf.rpm.log.gz

The above code is executed as a part of https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/tasks/publish.yml#L77 and it is set to true in toci: https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/collect-logs.yml#L4

and the same config is present in https://logserver.rdoproject.org/19/766419/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/5ca63de/logs/undercloud/var/log/extra/dump_variables_vars.yaml.gz (again needs to download)

artcl_txt_rename: true

not sure what is causing, logging the bug for further investigation

[Sorin Sbarnea (ssbarnea)]

IMHO we should not change the collection logic in order to accommodate what seems to be a hosting configuration issue. Instead this should be addressed at RDO logserver config.

In fact I would say that we should not even pre-compress the files, compression at rest is something that should be done transparently by log server. It is a known issue, raised several times during the last years and every time we were asked to implement workarounds like these which cause more problem than resolving.

For example the already existing upload-logs-s3 role already uploads files while compressing them on the fly and keeping them compressed at rest on S3. Still, the files are served correctly to the user.

The user never sees a ".gz" url, it just grabs the file and the server (S3) will serve that file using correct headers and a compresses stream.

That is a low-overhead and low-cost solution that is transparent. maybe we should just switch to use S3 until RDO is upgraded.

[Marios Andreou (marios-b)]

apparently this is due to config on the rdo server side .. dpawlik is checking that

11:29 < chandankumar> dpawlik|EoD, please comment on the bug alos
                      https://bugs.launchpad.net/tripleo/+bug/1907544
11:29 < bhjf> Title: Bug #1907544 “Rename compressed text based files to end with txt...” : Bugs : tripleo (at bugs.launchpad.net)
11:29 < chandankumar> dpawlik|EoD, thanks :-)
11:29 < chandankumar> marios, ^^
11:31 < marios> chandankumar: thanks ... Bug #1907544 was because of config issue on rdo side?
11:31 < bhjf> Bug https://bugzilla.redhat.com/show_bug.cgi?id=1907544 could not be retrieved: InvalidBugId
11:31 <+dpawlik|EoD> marios: more less yes. Earlier there was multiple rules and it was fine, but I added: (log|sh|yml|yaml)
11:32 < marios> dpawlik|EoD: thanks
11:32 <+dpawlik|EoD> (log|sh|yml|yaml)\.gz$

[daniel.pawlik (daniel-pawlik)]

Patch set: https://softwarefactory-project.io/r/#/c/20299/

[wes hayutin (weshayutin)]

fixed released? I see issues w/ the logs upstream but that may be a different issue