test_microbot fails with "Microbot failed to start."

[Joshua Genet]

12/baremetal/var/log/juju/unit-kubernetes-worker-1.log:2020-05-12 11:50:39 DEBUG microbot Error from server (NotFound): services "microbot" not found
12/baremetal/var/log/juju/unit-kubernetes-worker-1.log:2020-05-12 11:50:39 DEBUG microbot Error from server (NotFound): deployments.apps "microbot" not found
12/baremetal/var/log/juju/unit-kubernetes-worker-1.log:2020-05-12 11:50:39 DEBUG microbot Error from server (NotFound): ingresses.extensions "microbot-ingress" not found

---

Links to the artifacts of 2 different runs that hit this (we also had a successful run so may be a race condition)

https://oil-jenkins.canonical.com/artifacts/99ed933e-1aef-4356-b5af-87a7ba87bb25/index.html
https://oil-jenkins.canonical.com/artifacts/642a8a72-1667-4091-8c55-3de3f29f1371/index.html

---

Microbot being down on kubernetes-worker-1 is causing test_microbot to fail:

---

============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /home/ubuntu/k8s-validation/.tox/py3/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.6.9', 'Platform': 'Linux-4.15.0-74-generic-x86_64-with-Ubuntu-18.04-bionic', 'Packages': {'pytest': '5.4.1', 'py': '1.8.1', 'pluggy': '0.13.1'}, 'Plugins': {'flaky': '3.6.1', 'asyncio': '0.12.0', 'metadata': '1.9.0', 'html': '2.1.1'}}
rootdir: /home/ubuntu/k8s-validation, inifile: pytest.ini
plugins: flaky-3.6.1, asyncio-0.12.0, metadata-1.9.0, html-2.1.1
collecting ... collected 1 item

jobs/integration/validation.py::test_microbot FAILED

=================================== FAILURES ===================================
________________________________ test_microbot _________________________________
Traceback (most recent call last):
  File "/home/ubuntu/k8s-validation/jobs/integration/validation.py", line 331, in test_microbot
    raise MicrobotError("Microbot failed to start.")
integration.validation.MicrobotError: Microbot failed to start.

[John George (jog)]

In a related failure, test_microbot was running for over 2 1/2 hours, at which point our CI timeout limit aborted the tests.

https://solutions.qa.canonical.com/#/qa/testRun/99ed933e-1aef-4356-b5af-87a7ba87bb25

[Adam Stokes (adam-stokes)]

Here is a crashdump from one of our 1.18 runs on bionic http://jenkaas.s3-website-us-east-1.amazonaws.com/87d8234c-92d2-4a8b-b133-4e722232ed38/artifacts.tar.gz

[George Kraft (cynerva)]

This looks like a test issue. test_microbot was only waiting 30 seconds for microbot to come up, but from log output, I do not think microbot came up in that time.

Fixed via https://github.com/charmed-kubernetes/jenkins/commit/11a0b52a41eca2f23de7685b3eaff23a933272a5

> In a related failure, test_microbot was running for over 2 1/2 hours, at which point our CI timeout limit aborted the tests.

I've created a new issue for this: https://bugs.launchpad.net/charmed-kubernetes-testing/+bug/1878428