Functional tests are unstable

Bug #1991370 reported by Sudeep Bhandari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
charm-cloudsupport
Triaged
Medium
Unassigned

Bug Description

The functional tests for this charm are highly unstable. The test runs fail about 70-80% of the time.
There are 4 test bundles that we're deploying:
  - jammy-yoga
  - focal-yoga
  - jammy-yoga-ovn
  - focal-yoga-ovn
The failures don't look related to any of the bundles, because sometimes the tests pass in all the bundles, and sometimes they fail in any of the bundles.
The failures are seen mostly in the following functions (see file https://git.launchpad.net/charm-cloudsupport/tree/tests/functional/tests/modules/test_cloudsupport.py):
 - test_20_create_instance
 - test_25_test_connectivity

Sometimes, the failures can be avoided by retrying these tests for a few minutes, but sometimes even after retrying for upto 6 minutes, test_25_test_connectivity keeps failing.
Failures in test_20_create_instance have been mostly solved by addressing the issues in errored instance creation(by deleting and recreating the instances).
Some failures are on add_test_image (https://git.launchpad.net/charm-cloudsupport/tree/tests/functional/tests/modules/setup.py) but they disappear completely if we retry a few times.

We need to create more robust tests that pass most of the time, by finding the root causes of the failures, especially on test_25_test_connectivity.

Detailed logs on failures - https://code.launchpad.net/~sudeephb/charm-cloudsupport/+git/charm-cloudsupport/+merge/429864

Tags: bseng-453
Eric Chen (eric-chen)
tags: added: bseng-453
Eric Chen (eric-chen)
Changed in charm-cloudsupport:
importance: Undecided → Medium
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.