periodic-tripleo-ci-centos-8-standalone-upgrade-master is consistently failing with "Failed containers: nova_wait_for_api_service"

[Pooja Jadhav]

periodic-tripleo-ci-centos-8-standalone-upgrade-master is consistently failing with "Failed containers: nova_wait_for_api_service" with below trackback :

2021-10-20 18:40:11 | + exec /usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c ''\''/container-config-scripts/pyshim.sh' '/container-config-scripts/nova_wait_for_api_service.py'\'''
2021-10-20 18:40:11 | + command -v python3
2021-10-20 18:40:11 | + python3 /container-config-scripts/nova_wait_for_api_service.py
2021-10-20 18:40:11 | 2021-10-20 18:40:11.484805 | fa163ef3-2b47-b765-8bb2-00000000265b | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4 | standalone | error={"changed": false, "msg": "Failed containers: nova_wait_for_api_service"}
2021-10-20 18:40:11 | 2021-10-20 18:40:11.485923 | fa163ef3-2b47-b765-8bb2-00000000265b | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4 | standalone | 0:25:40.002250 | 634.80s

Detailed Logs :

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-master

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/11f7cb5/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/7ecc8ed/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/5e5d38f/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

[Sandeep Yadav (sandeepyadav93)]

This is still failing:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-master

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/74afd57/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

~~~
2021-11-25 02:46:20 | 2021-11-25 02:46:20.679186 | fa163efc-5d52-5392-87c8-000000002628 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4 | standalone | error={"changed": false, "msg": "Failed containers: nova_wait_for_api_service"}
~~~

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/74afd57/logs/undercloud/var/log/containers/nova/nova-conductor.log.txt.gz

~~~
2021-11-25 02:37:14.507 8 CRITICAL nova [req-60d4bf71-f10e-4073-945a-7ad009f698f3 - - - - -] Unhandled error: nova.exception.TooOldComputeService: Current Nova version does not support computes older than Xena but the minimum compute service level in your system is 56 and the oldest supported service level is 57.
2021-11-25 02:37:14.507 8 ERROR nova Traceback (most recent call last):
2021-11-25 02:37:14.507 8 ERROR nova File "/usr/bin/nova-conductor", line 10, in <module>
2021-11-25 02:37:14.507 8 ERROR nova sys.exit(main())
2021-11-25 02:37:14.507 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 46, in main
2021-11-25 02:37:14.507 8 ERROR nova topic=rpcapi.RPC_TOPIC)
2021-11-25 02:37:14.507 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 264, in create
2021-11-25 02:37:14.507 8 ERROR nova utils.raise_if_old_compute()
2021-11-25 02:37:14.507 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/utils.py", line 1099, in raise_if_old_compute
2021-11-25 02:37:14.507 8 ERROR nova oldest_supported_service=oldest_supported_service_level)
2021-11-25 02:37:14.507 8 ERROR nova nova.exception.TooOldComputeService: Current Nova version does not support computes older than Xena but the minimum compute service level in your system is 56 and the oldest supported service level is 57.
~~~

[Rabi Mishra (rabi)]

We're upgrading from wallaby to master (no xena release) and hence it's failing as Nova services only support computes that are not older than the previous major release[1]

I see nova compute service using wallaby container.

0b16653b4db1 trunk.registry.rdoproject.org/tripleowallaby/openstack-nova-compute:314a674e003a6c38e9de5bd6ac589ac9 kolla_start 44 minutes ago Up 44 minutes ago nova_compute

[1] https://opendev.org/openstack/nova/commit/aa7c6f87699ec1340bd446a7d47e1453847a637f

We probably need a better approach for upgrade.

[Jesse Pretorius (jesse-pretorius)]

Given that we are now diverging from the standard component upgrade model to a 'fast forward upgrade' model, we will need to change how our upgrade jobs execute the upgrade procedure. The upgrade will now need to be offline (from the control plane standpoint), not online.

In other words, we need to change it to:

1. Take all OpenStack services down
2. Upgrade the OpenStack containers
3. Run the OpenStack service database migrations
4. Bring up the OpenStack services

[Sandeep Yadav (sandeepyadav93)]

36890: Temporary remove master standalone upgrade jobs | https://review.rdoproject.org/r/c/rdo-jobs/+/36890

[Marios Andreou (marios-b)]

o/ as discussed a little at the last PTG it was decided in the *last* cycle (so during W release) that we would reduce coverage for the upgrade jobs - so in check/gate we only have undercloud upgrade and minor update.

we removed the standalone upgrade and overcloud upgrade jobs from master, didn't add them for wallaby and kept them only where they were already running in U and V

Indeed @Rabi from comment #2 and the same applies for the undercloud upgrade job which we are planning to keep running in check/gate. As we discussed briefly at PTG and since with upgrades team, that job is going stable/wallaby to M.

If we are going to keep the standalone upgrade for the component and integration lines then it should probably be the upcoming ffu standalone that upgrades is working on.

I don't know if we should spend any time trying to fix this job. As I said earlier it was decided a while ago to dump these (granted we said we'd keep them in periodics, but this is not viable in their present form at least, as we have found). This job in particular has been non voting (not in criteria for promotion) and broken for a while - partially because we are no longer running in check/gate so it was able to break without us noticing (and I'd be surprised if you didn't find another breakage even if you spent the time to fix this one).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830740

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/830740
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/7ec40a3c5287e1f6bf37a281cf778a23cc11ef1a
Submitter: "Zuul (22348)"
Branch: master

commit 7ec40a3c5287e1f6bf37a281cf778a23cc11ef1a
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu Feb 24 12:24:42 2022 +0530

    Added NovaDisableComputeServiceCheckForFfu var

    https://review.opendev.org/c/openstack/nova/+/826097 adds
    the service version check workaround for FFU.

    In order to reuse this functionality in tripleo deployment,
    NovaDisableComputeServiceCheckForFfu var is added to set
    workarounds/disable_compute_service_check_for_ffu=false
    in nova.conf.

    Depends-On: https://review.opendev.org/c/openstack/puppet-nova/+/830795

    Closes-Bug: #1961966
    Closes-Bug: #1947959

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ifbfd859e77ecf6831ced74e37aea1bd0ccdf4c5c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 16.0.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845681

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845681
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/3e72a2dff77e0b887285fd44f2df7dc2d6cd6a47
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3e72a2dff77e0b887285fd44f2df7dc2d6cd6a47
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu Feb 24 12:24:42 2022 +0530

    Added NovaDisableComputeServiceCheckForFfu var

    https://review.opendev.org/c/openstack/nova/+/826097 adds
    the service version check workaround for FFU.

    In order to reuse this functionality in tripleo deployment,
    NovaDisableComputeServiceCheckForFfu var is added to set
    workarounds/disable_compute_service_check_for_ffu=false
    in nova.conf.

    Depends-On: https://review.opendev.org/c/openstack/puppet-nova/+/845737

    Closes-Bug: #1961966
    Closes-Bug: #1947959

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Ifbfd859e77ecf6831ced74e37aea1bd0ccdf4c5c
    (cherry picked from commit 7ec40a3c5287e1f6bf37a281cf778a23cc11ef1a)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/830741