tempest/api/object_storage/test_object_expiry.py tests are brittle
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| tempest |
New
|
Undecided
|
Unassigned | ||
Bug Description
I recently saw a job fail like https:/
(Full logs at https:/
The relevant bit of the client request log had:
2024-01-18 07:22:04,117 92025 INFO [tempest.
1234332/
2024-01-18 07:22:04,118 92025 DEBUG [tempest.
Body: None
Response - Headers: {'date': 'Thu, 18 Jan 2024 07:22:04 GMT', <snip>}
Body: b'<html>
2024-01-18 07:22:04,160 92025 INFO [tempest.
2024-01-18 07:22:04,160 92025 DEBUG [tempest.
Body: None
Response - Headers: {'date': 'Thu, 18 Jan 2024 07:22:04 GMT', <snip>}
Body: b''
That is, the POST request took 30s to complete, so the expiry time of "now + 10s" [0] had already passed by the time it did the subsequent HEAD. The 404 is then expected behavior for Swift: the POST was timestamped as of the initial request time (so it shouldn't 400 with 'X-Delete-At in past') and the HEAD came after expiry had passed.
Note that we've seen similar trouble a few times before:
https:/
https:/
https:/
I'm not sure that the test *can* be made reliable without either
- (sometimes) skipping the "HEAD/GET should still 200 after POST" check (which seems to materially change the nature of the test) or
- using expiration delays on the order of minutes (which has other obvious downsides)
Maybe we could at least track the request time for the POST and retry the setup to that point if it's been more than the delay?
