nova.tests.functional.test_nova_manage.{TestDBArchiveDeletedRowsMultiCell, TestDBArchiveDeletedRows}.test_archive_task_logs failing

Bug #1934519 reported by Lee Yarwood
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
melanie witt

Bug Description

https://zuul.opendev.org/t/openstack/builds?job_name=nova-tox-functional-py38&project=openstack%2Fnova&branch=master&pipeline=check

https://925ac014ce9affd643c5-922c284b53012421c7cb8203c60422be.ssl.cf2.rackcdn.com/780147/23/check/nova-tox-functional-py38/8da78ca/testr_results.html

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/test_nova_manage.py", line 1735, in test_archive_task_logs
    self.assertRegex(self.output.getvalue(), r'\| task_log\s+\| 2')
  File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/unittest2/case.py", line 1304, in assertRegex
    raise self.failureException(msg)
AssertionError: Regex didn't match: '\\| task_log\\s+\\| 2' not found in 'Nothing was archived.\n+----------+-------------------------+\n| Table | Number of Rows Archived |\n+----------+-------------------------+\n| task_log | 1 |\n+----------+-------------------------+\n+----------+-------------------------+\n| Table | Number of Rows Archived |\n+----------+-------------------------+\n| task_log | 1 |\n+----------+-------------------------+\n'

Revision history for this message
Lee Yarwood (lyarwood) wrote :

Randomly this is now passing again... weird.

Changed in nova:
status: New → Incomplete
Revision history for this message
melanie witt (melwitt) wrote :

Hm, I will look into what's going on here. I used oslo.utils time overrides for this test to avoid inconsistencies but I must have missed something somewhere. The audit API time ranges aren't too intuitive [to me], I expect it has something to do with that.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/800313

Changed in nova:
status: Incomplete → In Progress
Revision history for this message
melanie witt (melwitt) wrote :

Indeed, the tests could fail depending on what date the test runs on + the number of days in the current month and next two months because the task log audit periods are calendar months. Go me!

melanie witt (melwitt)
tags: added: testing
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/800313
Committed: https://opendev.org/openstack/nova/commit/87dba379aca6ec4613986cfa4e749a3ada2b44d0
Submitter: "Zuul (22348)"
Branch: master

commit 87dba379aca6ec4613986cfa4e749a3ada2b44d0
Author: melanie witt <email address hidden>
Date: Fri Jul 9 22:22:15 2021 +0000

    Make test_archive_task_logs deterministic

    The 'nova-manage db archive_deleted_rows --task-log' functional tests
    involve manipulating time to assert archive behaviors when the --before
    flag is also used.

    While timedelta was used, set_time_override was not, so depending on
    the date the test ran on + the number of days in the current month and
    next two months, the test could fail. Task log audit periods are one
    calendar month by default and the compute manager calls
    last_completed_audit_period() without specifying a unit.

    This changes the tests to use a time override to ensure predictable
    behavior with regard to the audit period boundaries. The tests were
    moved into their own test case classes in order to override the time
    before services were started, so that the "service up" calculations
    work as expected.

    Closes-Bug: #1934519

    Change-Id: I9b16a3a849937aba5b90ed1ab9a80b7f0103f673

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/802766

Revision history for this message
melanie witt (melwitt) wrote :

This bug is unfortunately back again today and I'm working on a fix again.

Changed in nova:
assignee: nobody → melanie witt (melwitt)
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/803106

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/803106
Committed: https://opendev.org/openstack/nova/commit/39ef1ad774859562183190f74831a115698e6836
Submitter: "Zuul (22348)"
Branch: master

commit 39ef1ad774859562183190f74831a115698e6836
Author: melanie witt <email address hidden>
Date: Sat Jul 31 15:49:12 2021 +0000

    Restore original time override in test_archive_task_logs

    Change I9b16a3a849937aba5b90ed1ab9a80b7f0103f673 attempted to fix a
    non-determinstic failure in the test_archive_task_logs functional test
    by adding a time override. That fix however missed that the
    osloutils_fixture.TimeFixture clears the time override when it exits
    and that does not mean a previous time override will be restored.

    This removes usage of the osloutils_fixture.TimeFixture and instead
    sets time overrides and restores previous time overrides. Restoration
    of the original time overrides also made an adjustment to the --before
    option value necessary in order to pick up the first task_log record.

    Closes-Bug: #1934519

    Change-Id: Ic28d2e53d5f50e89635e19df08699f2b8c5cea84

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.0.0.0rc1

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers