test_dns_provider fails during cleanup with no error output

Bug #1931723 reported by Michael Skalka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Charmed Kubernetes Testing
Medium
Cory Johns

Bug Description

On k8s-suite rev 156e373
As seen on this test run: https://solutions.qa.canonical.com/testruns/testRun/cf9c041a-6f84-4b68-8869-bcb876a0a831
Artifacts here: https://oil-jenkins.canonical.com/artifacts/cf9c041a-6f84-4b68-8869-bcb876a0a831/index.html

test_dns_provider passes, however during cleanup it fails with no error output:

-------------------------------- live log call ---------------------------------
Deploying cs:~containers/coredns-15
  Waiting for CoreDNS charm to be ready
  Creating cross-model offer
  Waiting for cross-model offer to be ready
  Consuming cross-model offer
  Adding cross-model relation to CK
  Verifying that stale pod doesn't pick up new DNS provider
  Checking domain www.ubuntu.com
  Verifying DNS works on fresh pod
  Removing DNS pod
  Deploying DNS pod
  Checking domain www.ubuntu.com
  Checking domain kubernetes.default.svc.cluster.local
  Removing cross-model offer
  Removing CoreDNS charm
  Waiting for pods with label juju-app=coredns to be removed
  Verifying that DNS is no longer working
  Removing DNS pod
  Deploying DNS pod
  Checking domain www.ubuntu.com
  Switching back to core-dns from cdk-addons
  Verifying DNS works again
  Removing DNS pod
  Deploying DNS pod
  Checking domain www.ubuntu.com
  Checking domain kubernetes.default.svc.cluster.local
  Removing DNS pod
PASSEDCleaning up k8s model
Disconnecting k8s model
Destroying k8s model

2021-06-10-18:48:27 root ERROR [localhost] STDERR follows:

Traceback (most recent call last):
  File "/usr/local/bin/fce", line 11, in <module>
    load_entry_point('foundationcloudengine', 'console_scripts', 'fce')()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 207, in entry_point
    sys.exit(main(sys.argv[1:]))
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 198, in main
    opts.func(opts)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/command.py", line 94, in run
    self.run_per_layer(layer, args)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 40, in run_per_layer
    build_and_validate_if_needed(
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 85, in build_and_validate_if_needed
    layer.validate()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 200, in validate
    validator.run()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 389, in run
    self.run_inner()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/kubernetes.py", line 265, in run_inner
    self.run_tests(controller, model)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/kubernetes.py", line 294, in run_tests
    local(cmd, output_mode="live", env=my_env)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 227, in local
    return run_cmd(cmd, target_machine, **kwargs)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/remotehelpers.py", line 165, in run_cmd
    raise subprocess.CalledProcessError(ps.returncode, cmd, output=out, stderr=err)
subprocess.CalledProcessError: Command '['timeout', '1800', 'tox', '--workdir', '.tox', '-c', '/home/ubuntu/k8s-validation/tox.ini', '-e', 'py3', '--', 'pytest', '-v', '-s', '--junit-xml=/home/ubuntu/project/generated/kubernetes/k8s-suite/test_dns_provider-junit.xml', '--controller=foundations-maas', '--model=kubernetes', '/home/ubuntu/k8s-validation/jobs/integration/validation.py::test_dns_provider']' returned non-zero exit status 124.

Revision history for this message
Cory Johns (johnsca) wrote :

The 124 exit code is from the timeout command. It was sitting on "Destroying k8s model" for just over 10 minutes, which exceeded the 30m timeout.

PR: https://github.com/charmed-kubernetes/jenkins/pull/713

Changed in charmed-kubernetes-testing:
importance: Undecided → Medium
status: New → In Progress
assignee: nobody → Cory Johns (johnsca)
Revision history for this message
George Kraft (cynerva) wrote :

Fixed in master, commit ece82d9518e092e87374fcd9b2f1543debaf0412. Thanks Cory.

Changed in charmed-kubernetes-testing:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers