/usr/bin/swift-object-expirer: No such file or directory while deploying standalone

[chandan kumar]

Swift component pipeline master standalone jobs are failing with following error:
https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

```
 Run kolla_set_configs to copy ring files | standalone | item=swift_object_expirer | error={"ansible_loop_var": "item", "changed": true, "cmd": "podman exec -u root swift_object_expirer /usr/local/bin/kolla_set_configs", "delta": "0:00:00.117936", "end": "2021-03-22 17:37:01.342652", "failed_when_result": true, "item": "swift_object_expirer", "msg": "non-zero return code", "rc": 126, "start": "2021-03-22 17:37:01.224716", "stderr": "Error: cannot exec into container that is not running: container state improper", "stderr_lines": ["Error: cannot exec into container that is not running: container state improper"], "stdout": "", "stdout_lines": []}
```

While looking at container logs,
https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/var/log/extra/podman/containers/swift_object_expirer/stdout.log.txt.gz

```
Running command: '/usr/bin/swift-object-expirer /etc/swift/object-expirer.conf'
++ cat /run_command
+ CMD='/usr/bin/swift-object-expirer /etc/swift/object-expirer.conf'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
+ echo 'Running command: '\''/usr/bin/swift-object-expirer /etc/swift/object-expirer.conf'\'''
+ exec /usr/bin/swift-object-expirer /etc/swift/object-expirer.conf
/usr/local/bin/kolla_start: line 18: /usr/bin/swift-object-expirer: No such file or directory
```

the following binary got added in swift https://github.com/rdo-packages/swift-distgit/commit/0014419a4958e39c652a74d6cff8f46464bce8ad#diff-aa7357e145b5970f69d2f6508c5dc1ddc55add13564ade893c32f7e561507d9e 4 days back in RDO.

I will be adding more info regarding packages soon.

[chandan kumar (chkumar246)]

By looking at repo used: https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-train/0138736/logs/undercloud/etc/yum.repos.d/swift-component.repo.txt.gz

[swift]
name=swift
baseurl=https://trunk.rdoproject.org/centos8-train/component/swift/component-ci-testing
gpgcheck=0
enabled=1
priority=1

and following packages are available there:

openstack-swift-2.23.3-0.20210301213807.96e24bb.el8.src.rpm
openstack-swift-account-2.23.3-0.20210301213807.96e24bb.el8.noarch.rpm
openstack-swift-container-2.23.3-0.20210301213807.96e24bb.el8.noarch.rpm
openstack-swift-doc-2.23.3-0.20210301213807.96e24bb.el8.noarch.rpm
openstack-swift-object-2.23.3-0.20210301213807.96e24bb.el8.noarch.rpm
openstack-swift-proxy-2.23.3-0.20210301213807.96e24bb.el8.noarch.rpm
python3-swift-2.23.3-0.20210301213807.96e24bb.el8.noarch.rpm
python3-swift-tests-2.23.3-0.20210301213807.96e24bb.el8.noarch.rpm

are there.

[chandan kumar (chkumar246)]

https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/var/log/extra/podman/containers/swift_container_server/podman_info.log.txt.gz

openstack-swift-container.noarch 2.27.1-0.20210322142351.7e27829.el8 @swift got updated

[chandan kumar (chkumar246)]

Following patches landed recently in dist-git recently then it broke the component line.
https://review.rdoproject.org/r/c/openstack/swift-distgit/+/32374
https://review.rdoproject.org/r/c/openstack/swift-distgit/+/29930
https://review.rdoproject.org/r/c/openstack/swift-distgit/+/29952
https://review.rdoproject.org/r/c/openstack/swift-distgit/+/32542
https://review.rdoproject.org/r/c/openstack/swift-distgit/+/32499

out of which https://review.rdoproject.org/r/c/openstack/swift-distgit/+/32374 Swift to the distributed object expirer -> might caused it.

[Pete Zaitcev (zaitcev)]

That is all fine, but it does not explain why CI is trying to run with the removed object-expirer.conf

[zaitcev@suzdal openstack-swift.rdo]$ git branch
* rpm-master
[zaitcev@suzdal openstack-swift.rdo]$ git grep 'expirer\.conf'
openstack-swift.spec:%{_mandir}/man5/object-expirer.conf.5*
[zaitcev@suzdal openstack-swift.rdo]$ grep expirer *
object-server.conf:[object-expirer]
openstack-swift-object-expirer.service:ExecStart=/usr/bin/swift-object-expirer /etc/swift/object-server.conf
[zaitcev@suzdal openstack-swift.rdo]$

The path must be hardcoded in TripleO or something.

[chandan kumar (chkumar246)]

While configuring tripleo swift services container, it calls this https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/swift/swift-storage-container-puppet.yaml#L277
which is not found.

Copying from above https://review.rdoproject.org/r/c/openstack/swift-distgit/+/32374 yatin's comment here might help.
```
This caused issue in swift component jobs https://bugs.launchpad.net/tripleo/+bug/1920924, as containers and tht still rely on old binary,subpackage and conf file.

tripleo-common:- currently relies on swift-proxy-server container image, which do not contain swift-object rpm so new path of config is not available and thus job fails https://github.com/openstack/tripleo-common/blob/master/container-images/tcib/base/os/swift-base/swift-proxy-server/swift-proxy-server.yaml#L10

tht:- relies on old config file: /usr/bin/swift-object-expirer /etc/swift/object-expirer.conf which is now removed, so need to switch tht to new files. Also as per the change seems it needs to switch to ContainerSwiftObjectImage so required config and binary are available for running expirer service going forward.

But to put a change to tht we would need swift-object container to be promoted(which is currently blocked due to above issue). So we can try following to clear this:
* put a temporary change in rpm to have old paths available(can be added in both proxy and object subpackage)
* have a promotion for new swift object rpms/containers
* push change(rely on new path and swiftobject container image) to tht and get that merged
* remove the temp change in swift at last once everything is clear.

[Pete Zaitcev (zaitcev)]

Note that the first ENOENT was caused by missing binary. It's supplied by a different RPM now, openstack-swift-object, while formerly it was in openstack-swift-proxy. That shold be possible to fix somewhere in whatever installs what RPM.

[Rabi Mishra (rabi)]

Could be fixed with https://review.opendev.org/c/openstack/tripleo-heat-templates/+/782746

[chandan kumar (chkumar246)]

Testing above patch in testproject here: https://review.rdoproject.org/r/c/testproject/+/32717

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/c/openstack/swift-distgit/+/32516 Revert "Swift to the distributed object expirer" is proposed now.

[chandan kumar (chkumar246)]

 object-expirer options should be written to object-server.conf - https://bugs.launchpad.net/puppet-swift/+bug/1921319

[chandan kumar (chkumar246)]

We reverted the spec file changes https://review.rdoproject.org/r/c/openstack/swift-distgit/+/32516 and tested it here: https://review.rdoproject.org/r/c/testproject/+/32717

periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master SUCCESS 1h 16m 49s

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rabi Mishra <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/782746