Delay for safety in ObjectExpiryTest is not sufficient in all test environments

[Daryl Walleck]

In the _test_object_expiry function of ObjectExpiryTest, an extra delay for latency is hard coded in:

https://github.com/openstack/tempest/blob/master/tempest/api/object_storage/test_object_expiry.py#L66

This appears to be sufficient for smaller test environments, but not enough for public environments under load (trial and error led me to determine that 7 seconds was sufficient for Rackspace's public environments). This value could be increased, but that would be a temporary patch until someone with another environment with different latency ran into the same problem. Would it make sense to turn this into a polling loop checking periodically over a configurable period of time? Or would increasing this value to a larger number be sufficient?

[Attila Fazekas (afazekas)]

What is the source of the delay ?

The object was stored and read back with `x-delete-at` , as I remember the server just checks the current time and this header for serving 404.

Is it still working like that?

Is it possible the clocks are not in sync ?

[Ievgeniia Zadorozhna (izadorozhna)]

I suggest adding a configuration parameter, that allows to set this delay in tempest.conf. If it is ann appropriate solution, I can propose a fix for it.

[Daryl Walleck (dwalleck)]

I'm not sure what the exact cause of the delay is. My guess is that it has to do with propagation time, which is what the existing hardcoded leeway of 3 seconds was trying to solve. 7 seconds seems to be enough for my production regions, but that's as arbitrary of a number of 3. We could make this value configurable per environment as mentioned, or should there be no propagation delay and the API should be immediately up to date, in which case the sleep should be removed and a bug with Swift opened. Thoughts?

[Ievgeniia Zadorozhna (izadorozhna)]

I suggest to make this couple of seconds configurable. In tempest.conf in [object-storage] group there is a parameter "container_sync_interval". I suggest to use it instead of creating new parameter. By default it is 5 seconds.
As for me, on my cloud the test passes only when I am adding 10 seconds (so even 7 is not enough).
Review please my commit:
https://review.openstack.org/#/c/192111/

[Ievgeniia Zadorozhna (izadorozhna)]

Changed the logic from the configurable parameter to the checking whether the object still exists and waiting for several seconds in a loop (waiting for 1 minute in whole).