Failed containers: nova_wait_for_api_service on CS 9 standalone upgrade

[chandan kumar]

39554: DNM standalone upgrade cs9 master testing | https://review.rdoproject.org/r/c/testproject/+/39554 are working on adding standalone upgrade job on CentOS Stream 9.

But the job is failing during standalone upgrade with following errors:

Logs from standalone_upgrade.log.txt.gz [1]

```
+ . kolla_extend_start
2022-02-23 09:10:20 | + echo 'Running command: '\''/usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c '\''/container-config-scripts/pyshim.sh /container-config-scripts/nova_wait_for_api_service.py'\'''\'''
2022-02-23 09:10:20 | + exec /usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c ''\''/container-config-scripts/pyshim.sh' '/container-config-scripts/nova_wait_for_api_service.py'\'''
2022-02-23 09:10:20 | + command -v python3
2022-02-23 09:10:20 | + python3 /container-config-scripts/nova_wait_for_api_service.py
2022-02-23 09:10:20 | 2022-02-23 09:10:20.907605 | fa163eeb-0856-8642-5ae5-000000002661 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4 | standalone | error={"changed": false, "msg": "Failed containers: nova_wait_for_api_service"}
```

On further looking at nova_wait_for_api_service/stdout.log [2]

```
+ python3 /container-config-scripts/nova_wait_for_api_service.py
2022-02-23 09:00:16.517 DEBUG nova_wait_for_api_service Running with parameter insecure = True
2022-02-23 09:00:17.101 INFO nova_wait_for_api_service Waiting for nova-api service
2022-02-23 09:00:27.121 INFO nova_wait_for_api_service Waiting for nova-api service
2022-02-23 09:00:37.139 INFO nova_wait_for_api_service Waiting for nova-api service
2022-02-23 09:00:47.156 INFO nova_wait_for_api_service Waiting for nova-api service
2022-02-23 09:00:57.180 INFO nova_wait_for_api_service Waiting for nova-api service
2022-02-23 09:01:07.197 INFO nova_wait_for_api_service Waiting for nova-api service
2022-02-23 09:01:17.206 INFO nova_wait_for_api_service Waiting for nova-api service
2022-02-23 09:01:27.226 INFO nova_wait_for_api_service Waiting for nova-api service

```

By grepping the logs from errors.txt
```
2022-02-23 08:59:57.290 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova Traceback (most recent call last):
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module>
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova sys.exit(main())
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova File "/usr/lib/python3.9/site-packages/nova/cmd/scheduler.py", line 47, in main
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova server = service.Service.create(
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova File "/usr/lib/python3.9/site-packages/nova/service.py", line 265, in create
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova utils.raise_if_old_compute()
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova File "/usr/lib/python3.9/site-packages/nova/utils.py", line 1095, in raise_if_old_compute
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova raise exception.TooOldComputeService(
2022-02-23 09:00:01.599 ERROR /var/log/containers/nova/nova-scheduler.log: 2 ERROR nova nova.exception.TooOldComputeService: Current Nova version does not support computes older than Xena but the minimum compute service level in your system is 56 and the oldest supported service level is 57.
```

I am not sure ERROR nova nova.exception.TooOldComputeService: Current Nova version does not support computes older than Xena but the minimum compute service level in your system is 56 and the oldest supported service level is 57. is causing this issue.

Filing this bug to track this issue.

Logs:

[1]. https://logserver.rdoproject.org/54/39554/2/check/periodic-tripleo-ci-centos-9-standalone-upgrade-master/049666a/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

[2]. https://logserver.rdoproject.org/54/39554/2/check/periodic-tripleo-ci-centos-9-standalone-upgrade-master/049666a/logs/undercloud/var/log/extra/podman/containers/nova_wait_for_api_service/stdout.log.txt.gz

[3]. https://logserver.rdoproject.org/54/39554/2/check/periodic-tripleo-ci-centos-9-standalone-upgrade-master/049666a/logs/undercloud/var/log/extra/errors.txt.gz

[Rabi Mishra (rabi)]

same as https://bugs.launchpad.net/tripleo/+bug/1947959/comments/2

We can't upgrade from wallaby to master.

[yatin (yatinkarel)]

May be can try to see if just setting workarounds/disable_compute_service_check_for_ffu=True for nova services clears it?

[Rabi Mishra (rabi)]

> May be can try to see if just setting workarounds/disable_compute_service_check_for_ffu=True for nova services clears it?

Yeah worth trying. This workaround[1] seems to have been added recently.

[1] https://review.opendev.org/c/openstack/nova/+/826097

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830740

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830740
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/7ec40a3c5287e1f6bf37a281cf778a23cc11ef1a
Submitter: "Zuul (22348)"
Branch: master

commit 7ec40a3c5287e1f6bf37a281cf778a23cc11ef1a
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu Feb 24 12:24:42 2022 +0530

    Added NovaDisableComputeServiceCheckForFfu var

    https://review.opendev.org/c/openstack/nova/+/826097 adds
    the service version check workaround for FFU.

    In order to reuse this functionality in tripleo deployment,
    NovaDisableComputeServiceCheckForFfu var is added to set
    workarounds/disable_compute_service_check_for_ffu=false
    in nova.conf.

    Depends-On: https://review.opendev.org/c/openstack/puppet-nova/+/830795

    Closes-Bug: #1961966
    Closes-Bug: #1947959

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ifbfd859e77ecf6831ced74e37aea1bd0ccdf4c5c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 16.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845681

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845681
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/3e72a2dff77e0b887285fd44f2df7dc2d6cd6a47
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3e72a2dff77e0b887285fd44f2df7dc2d6cd6a47
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu Feb 24 12:24:42 2022 +0530

    Added NovaDisableComputeServiceCheckForFfu var

    https://review.opendev.org/c/openstack/nova/+/826097 adds
    the service version check workaround for FFU.

    In order to reuse this functionality in tripleo deployment,
    NovaDisableComputeServiceCheckForFfu var is added to set
    workarounds/disable_compute_service_check_for_ffu=false
    in nova.conf.

    Depends-On: https://review.opendev.org/c/openstack/puppet-nova/+/845737

    Closes-Bug: #1961966
    Closes-Bug: #1947959

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ifbfd859e77ecf6831ced74e37aea1bd0ccdf4c5c
    (cherry picked from commit 7ec40a3c5287e1f6bf37a281cf778a23cc11ef1a)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/830741