© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
a60fb26
(Get the code!)
Thanks Chandan & Marios for your help!
What I noticed is that Tempest runs sometimes into issues with the eventual consistency due to slow disk writes on some of the nodes. This happens usually on deleting a container that is not empty yet. Tempest sends object delete requests before deleting a container, but these are not finished on all nodes if some of them are still waiting to finish writing the last update to disk.
This is expected behavior due to the eventual consistency. Question is why this was working in the past and is now sometimes failing (the last 3 successful runs are from today[1] with no errors)?
As mentioned above, sometimes there are a few requests pretty slow, and these are the ones that eventually fail. The last failure I was looking into executed object server deletes on average within 18ms, but the failed tests happened when some writes took 1.65, 1.19 and 0.55 seconds and eventual consistency hit in. So something was slowing down disks writes quite a bit.
I'm wondering if anything changed recently in the test environment itself? Slower nodes and/or disks, network issues etc.? Newer Tempest version (I noticed a change that might be an issue, but doesn't explain all the failed tests[2])?
[1] https:/
[2] https:/