tripleo-ci-centos-9-standalone is failing on stable/wallaby - tempest scenario - failed to reach ACTIVE status and task state "None" within the required time

[Ronelle Landy]

tripleo-ci-centos-9-standalone is failing on stable/wallaby:

tempest.scenario.test_network_basic_ops.TestNetworkBasicOps
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern

Example logs:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_11b/847237/1/gate/tripleo-ci-centos-9-standalone/11b9f5e/logs/undercloud/var/log/tempest/stestr_results.html

tempest.lib.exceptions.TimeoutException: Request timed out
Details: (TestNetworkBasicOps:test_update_instance_port_admin_state) Server 1fd5008b-1027-4d50-b4e0-cfc77c5c012e failed to reach ACTIVE status and task state "None" within the required time (300 s). Server boot request ID: req-602e6c46-236f-4bd6-970e-7084c403ba05. Current status: BUILD. Current task state: scheduling.

Other logs:

https://5bc0b841e9829f2c71ba-d0e3ac2c8767bdf73a2dad3e5ed8b6d1.ssl.cf1.rackcdn.com/850046/2/check/tripleo-ci-centos-9-standalone/2d91f78/logs/undercloud/var/log/tempest/stestr_results.html

https://79521b7af6a207fbab35-460112c8cdf3a162e27bb5a0ba3266a8.ssl.cf1.rackcdn.com/850047/1/check/tripleo-ci-centos-9-standalone/a402aad/logs/undercloud/var/log/tempest/stestr_results.html

[Ronelle Landy (rlandy)]

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-standalone&branch=stable%2Fwallaby&skip=0

failure started on: 2022-07-19 13:00:33

[Ronelle Landy (rlandy)]

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b8b/849839/5/check/tripleo-ci-centos-9-standalone/b8b0cc7/logs/undercloud/var/log/extra/journal_errors.txt

shows:

2022-07-19T17:10:02+0000 standalone.localdomain sshd[218372]: fatal: mm_answer_sign: sign: error in libcrypto

[Ronelle Landy (rlandy)]

We got an new centos 9 stream compose:

last success: CentOS-Stream-9-20220705.0
first failure: CentOS-Stream-9-20220718.0

[Ronelle Landy (rlandy)]

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b8b/849839/5/check/tripleo-ci-centos-9-standalone/b8b0cc7/logs/undercloud/var/log/containers/nova/nova-compute.log shows:

Error: 'QueryContext' object has no attribute 'statement'
Traceback (most recent call last):

  File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 138, in _object_dispatch
    return getattr(target, method)(*args, **kwargs)

  File "/usr/lib/python3.9/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
    return fn(self, *args, **kwargs)

  File "/usr/lib/python3.9/site-packages/nova/objects/instance.py", line 836, in save
    old_ref, inst_ref = db.instance_update_and_get_original(

  File "/usr/lib/python3.9/site-packages/nova/db/api.py", line 683, in instance_update_and_get_original
    rv = IMPL.instance_update_and_get_original(context, instance_uuid, values,

  File "/usr/lib/python3.9/site-packages/nova/db/sqlalchemy/api.py", line 178, in wrapper
    return f(*args, **kwargs)

  File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
    ectxt.value = e.inner_exc

  File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

  File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

  File "/usr/lib/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)

  File "/usr/lib/python3.9/site-packages/nova/db/sqlalchemy/api.py", line 221, in wrapped
    return f(context, *args, **kwargs)

  File "/usr/lib/python3.9/site-packages/nova/db/sqlalchemy/api.py", line 2151, in instance_update_and_get_original
    return (copy.copy(instance_ref), _instance_update(

  File "/usr/lib/python3.9/site-packages/nova/db/sqlalchemy/api.py", line 2232, in _instance_update
    instance_ref = model_query(context, models.Instance,

  File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/orm.py", line 52, in update_on_match
    return update_match.update_on_match(

  File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/update_match.py", line 185, in update_on_match
    pk_value = query.update_returning_pk(values, surrogate_key_arg)

  File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/orm.py", line 42, in update_returning_pk
    return update_match.update_returning_pk(self, values, surrogate_key)

  File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/update_match.py", line 407, in update_returning_pk
    return pk_strategy(query, mapper, values, surrogate_key)

  File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/update_match.py", line 465, in _pk_strategy_mysql_last_insert_id
    update_stmt = _update_stmt_from_query(mapper, query, values)

  File "/usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/update_match.py", line 489, in _update_stmt_from_query
    primary_table = context.statement.froms[0]

AttributeError: 'QueryContext' object has no attribute 'statement'

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850428

[Takashi Kajinami (kajinamit)]

This looks like a problem which was fixed in oslo.db 8.6.0 .

https://github.com/openstack/oslo.db/commit/f2a94b6f1a8d1c2274ff81ae9968d22a22fd8503

8.6.0 is part of xena deliverables thus the fix is not present in wallaby.

Currently sqlalchemy 1.4.37-3.el9 is pulled from the appstream repo, and this is newer than the wallaby upper constraint(1.3.23).

[Alan Pevec (apevec)]

This oslo.db fix could qualify for stable backport, using private APIs is a bad thing:
https://review.opendev.org/c/openstack/oslo.db/+/850550

Not sure if this will be enough to make Wallaby work with SQLAlchemy 1.4 ?

[Takashi Kajinami (kajinamit)]

That fix can be backported as long as it does not break the previous versions.
The serious problem this is revealing is that we are pulling dependencies which have never been tested with wallaby (and WILL never be tested as we won't bump u-c in stable branches)

There is one more fix which looks like a fix for sqlalchemy 1.4 I can find from git log and there might be a few more.
 https://github.com/openstack/oslo.db/commit/f37269ae603bc0e09210379e03a476ff2193c74c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/1b30d001551efa6fd78082039e0e2900e7d6b4eb
Submitter: "Zuul (22348)"
Branch: master

commit 1b30d001551efa6fd78082039e0e2900e7d6b4eb
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Wed Jul 20 12:44:28 2022 +0530

    Downgrade python3-sqlalchemy

    Wallaby tests are showing multiple errors since
    the last centos 9 stream compose was released.

    This patch rolls back some version to avoid
    the latest failures and uses the version from
    wallaby-deps.

    Related-Bug: #1982227
    Related-Bug: #1982195

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: Iae399de94350d0b1a6f5e75f6155e9013919ef8d

[Alan Pevec (apevec)]

> we are pulling dependencies which have never been tested with wallaby

this came in from CS9 baseos, see more info in my comment https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/44082/2//COMMIT_MSG#15

we'll consider preventive action, at least to get an early warning when baseos deps change,
also updating stable u-c is possible with justification

[Takashi Kajinami (kajinamit)]

I'm asking this just based on my curiosity but AFAIK upstream wallaby does not support CentOS Stream 9 but only CentOS 8 (even not stream as it was not available at that time).
What kind of justification we can push to change wallaby u-c for fix the compatibility for unsupported CentOS Stream 9.

Updating u-c affects ALL openstack projects and bump of sqlalchemy is likely to cause the same breackage in mutiple projects. If we bump seqalchemy in wallaby u-c then that means we need to ensure we fix the same problem in literally all projects, including ones not supported downstream or even in RDO. I'm not sure that is doable now.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850428

[Arx Cruz (arxcruz)]

This is no longer been seen in openstack standalone closing