Ussuri periodic standalone upgrade job is failing with error "stderr": "/bin/sh: docker: command not found"

[Sandeep Yadav]

periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri is failing with error "stderr": "/bin/sh: docker: command not found"

Log Url:-

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/eddc6a6/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
TASK [tripleo-container-tag : Pull trunk.registry.rdoproject.org/tripleotraincentos8/centos-binary-cinder-volume:ce8cbfdd6ebaaf374c54870cbd849fb3 image] ***
Thursday 22 October 2020 05:30:53 +0000 (0:00:00.093) 0:02:17.865 ******
fatal: [standalone]: FAILED! => {"changed": true, "cmd": "docker pull trunk.registry.rdoproject.org/tripleotraincentos8/centos-binary-cinder-volume:ce8cbfdd6ebaaf374c54870cbd849fb3", "delta": "0:00:00.002175", "end": "2020-10-22 05:30:54.128396", "msg": "non-zero return code", "rc": 127, "start": "2020-10-22 05:30:54.126221", "stderr": "/bin/sh: docker: command not found", "stderr_lines": ["/bin/sh: docker: command not found"], "stdout": "", "stdout_lines": []}
~~~

Another Example:-

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/24342fb/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[Alex Schultz (alex-schultz)]

The issue is that the train deploy is failing because it thinks docker (but for centos8 we need podman). We're missing the -e THT/environment/podman.yaml specifically for the train centos8 deployment. The HA in train on centos7 is docker only, but when it's a centos8 job we need to include podman to allow it to continue

[Alex Schultz (alex-schultz)]

https://review.opendev.org/#/c/759325/

[Sandeep Yadav (sandeepyadav93)]

Hello,

We are still hitting the issue:_

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/11cada9/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
TASK [tripleo-container-tag : Pull trunk.registry.rdoproject.org/tripleotraincentos8/centos-binary-cinder-volume:1156b5d8cec9fbd5863009dcfa04a6cf image] ***
Tuesday 27 October 2020 05:08:11 +0000 (0:00:00.091) 0:02:17.675 *******
fatal: [standalone]: FAILED! => {"changed": true, "cmd": "docker pull trunk.registry.rdoproject.org/tripleotraincentos8/centos-binary-cinder-volume:1156b5d8cec9fbd5863009dcfa04a6cf", "delta": "0:00:00.002047", "end": "2020-10-27 05:08:11.297656", "msg": "non-zero return code", "rc": 127, "start": "2020-10-27 05:08:11.295609", "stderr": "/bin/sh: docker: command not found", "stderr_lines": ["/bin/sh: docker: command not found"], "stdout": "", "stdout_lines": []}
~~~

/usr/share/openstack-tripleo-heat-templates/environments/podman.yaml is missing from deployment command.

https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri/11cada9/logs/undercloud/home/zuul/tripleo_deploy.sh.txt.gz

Patch https://review.opendev.org/#/c/759325/ didn't fix the issue because we are parenting off "tripleo-ci-base-standalone-upgrade-rdo-centos-8" and not tripleo-ci-centos-8-standalone-upgrade.

fyi.. tracing .. Periodic job parent chain:- http://paste.openstack.org/show/799457/

Upstream check jobs is not hitting this issue because there we pass 'environments/podman.yaml' using featureset_override in job defination(tripleo-ci-centos-8-standalone-upgrade-ussuri).

Upstream job
~~~
- job:
    name: tripleo-ci-centos-8-standalone-upgrade-ussuri
    parent: tripleo-ci-base-standalone-upgrade-centos-8
    voting: true
    branches: ^stable/ussuri$
    vars:
      featureset: '056'
      featureset_override:
        standalone_environment_files:
          - 'environments/low-memory-usage.yaml'
          - 'environments/docker-ha.yaml'
          - 'environments/podman.yaml' >>>> Here
    irrelevant-files: *upgrade_irrelevant
~~~

We are missing similiar standalone_environment_files(featureset_override) in periodic job definition.

https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/standalone-jobs.yaml#l475
~~~
 474 - job:
 475 name: periodic-tripleo-ci-centos-8-standalone-upgrade-master
 476 parent: tripleo-ci-base-standalone-upgrade-periodic-centos-8
 477 voting: false
 478 vars:
 479 featureset: '056'
 480 release: master
~~~

[Sandeep Yadav (sandeepyadav93)]

We proposed a similar patch[1] for periodic upgrade jobs which fixes the issue

[1] https://review.rdoproject.org/r/#/c/30684/

Testproject run is green:-

https://review.rdoproject.org/r/#/c/28458/
~~~
periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri https://review.rdoproject.org/zuul/build/dca79d57a9f749dbbbe3c716de2623f5 : SUCCESS in 1h 57m 19s (non-voting)
~~~