stable/yoga gate blocker: failures in py39 unit tests

[Jay Faulkner]

All jobs in nova.tests.unit.cmd.test_manage.DbCommandsTestCase, as well as test_nova.tests.unit.cmd.test_manage.UtilitiesTestCase.test_format_dict.

One specific example:
```
ft2.1: nova.tests.unit.cmd.test_manage.UtilitiesTestCase.test_format_dicttesttools.testresult.real._StringException: pythonlogging:'': {{{
2022-09-01 17:37:16,772 WARNING [oslo_policy.policy] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html.
2022-09-01 17:37:16,773 WARNING [oslo_policy.policy] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use ``oslopolicy-convert-json-to-yaml`` tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way: https://docs.openstack.org/oslo.policy/latest/cli/oslopolicy-convert-json-to-yaml.html.
2022-09-01 17:37:16,774 WARNING [oslo_policy.policy] Policy Rules ['os_compute_api:extensions', 'os_compute_api:os-floating-ip-pools', 'os_compute_api:os-quota-sets:defaults', 'os_compute_api:os-availability-zone:list', 'os_compute_api:limits', 'project_admin_api', 'project_member_api', 'project_reader_api', 'project_reader_or_admin', 'os_compute_api:limits:other_project', 'os_compute_api:os-lock-server:unlock:unlock_override', 'os_compute_api:servers:create:zero_disk_flavor', 'compute:servers:resize:cross_cell'] specified in policy files are the same as the defaults provided by the service. You can remove these rules from policy files which will make maintenance easier. You can detect these redundant rules by ``oslopolicy-list-redundant`` tool also.
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/unit/cmd/test_manage.py", line 55, in test_format_dict
    self.assertEqual(
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py39/lib/python3.9/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/py39/lib/python3.9/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = '''\
+----------+----------------------+
| Property | Value |
+----------+----------------------+
| bing | bat |
| foo | bar |
| test | {'a nested': 'dict'} |
| wow | a multiline |
| | string |
+----------+----------------------+'''
actual = '''\
+----------+----------------------+
| Property | Value |
+----------+----------------------+
| bing | bat |
| foo | bar |
| test | {'a nested': 'dict'} |
| wow | a multiline |
| | string |
+----------+----------------------+'''

```

This is failing on at least two different changes that are completely unrelated and are unlikely to be causing these failures:
- https://review.opendev.org/c/openstack/nova/+/855025
- https://review.opendev.org/c/openstack/nova/+/854257

I was unable to create an opensearch query to find all these failures, but https://tinyurl.com/mr3pjzyn catches related failures in cross-nova-py38

I was able to reproduce the failure in test_format_dict locally, by downloading 854257 and running tox -e py39. Output from that reproduction is:

```
nova.tests.unit.cmd.test_manage.UtilitiesTestCase.test_format_dict
------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/jay/dev/nova/nova/tests/unit/cmd/test_manage.py", line 55, in test_format_dict
    self.assertEqual(

      File "/home/jay/dev/nova/.tox/py39/lib/python3.9/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)

      File "/home/jay/dev/nova/.tox/py39/lib/python3.9/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error

    testtools.matchers._impl.MismatchError: !=:
reference = '''\
+----------+----------------------+
| Property | Value |
+----------+----------------------+
| bing | bat |
| foo | bar |
| test | {'a nested': 'dict'} |
| wow | a multiline |
| | string |
+----------+----------------------+'''
actual = '''\
+----------+----------------------+
| Property | Value |
+----------+----------------------+
| bing | bat |
| foo | bar |
| test | {'a nested': 'dict'} |
| wow | a multiline |
| | string |
+----------+----------------------+'''
```

[Balazs Gibizer (balazs-gibizer)]

This feels related to https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2022-08-27.log.html

[Balazs Gibizer (balazs-gibizer)]

I searched around and I can confirm it is only visible on stable/yoga and in the py39 job. Also it only appears in the last 4 days
https://paste.opendev.org/show/bxX18pfYNwDcIzq1e0H1/

[Balazs Gibizer (balazs-gibizer)]

Now I realized that launchpad changed the pasted output of the bug report. The difference in the output is the placement of the "Value" header. The in the reference it is left aligned it the actual output it is centered.

So now I'm pretty sure it is related to the prettytable issue reported by flicker during the weekend:
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2022-08-27.log.html
https://zuul.opendev.org/t/openstack/build/2e4535d6aa2d4fd987ddfd0d0bc296d1

[Balazs Gibizer (balazs-gibizer)]

For some reason in py39 in stable/yoga PrettyTable==3.4.0 is installed which is very new, released on 25th of Aug and which has a breaking change about table formatting. So I looked into our global upper constraints file and actually it does not restrict PrettyTable for py39 on yoga[1]. I will push a patch soon to requirements to fix it.

[1]https://github.com/openstack/requirements/blob/52cd6721466b6af58f62ae8a16512b566db887dd/upper-constraints.txt#L154

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/855658

[sean mooney (sean-k-mooney)]

with the new 3.4.1 release of PrettyTable we are no longer impacted by this however to make sure this does not break use going forward I'm going to continue with
https://review.opendev.org/c/openstack/nova/+/855658
and backport it to yoga so that we have captured our depency on the alignment.

if we want to then change the alignment in the future we can update the tests deliberately and not
worry about version difference for now that is out of scope so I'm just going to hardcode the left alignment of the table headers our tests depend on today.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/855658
Committed: https://opendev.org/openstack/nova/commit/6569de43a6482488afb620fd9dd43cb15b316cce
Submitter: "Zuul (22348)"
Branch: master

commit 6569de43a6482488afb620fd9dd43cb15b316cce
Author: Sean Mooney <email address hidden>
Date: Fri Sep 2 13:42:15 2022 +0100

    add header alingment for PrettyTable 3.4.0

    In 3.3.0 the align attirbute applied to both the header
    and the data. In 3.4.0 align only applies to the data.

    This change restores the previous left align behavior
    for both header and data.

    Closes-Bug: #1988482
    Change-Id: Ia77410b10c1706bc6561b11cf5d2ef72b936795e

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 27.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/requirements yoga-eom release.