metrics server failure missed by tests

[Tim Van Steenburgh]

Opened by hyperbolic2346 on 2019-01-23 16:48:13+00:00 at https://github.com/charmed-kubernetes/jenkins/issues/302

------------------------------------------------------------

We had an issue crop up in 1.11 where the metrics server fails to connect. We need to verify this functionality in some way in our testing. In this specific case, kubelet stopped listening on the port metrics server is trying to reach. This was a known thing in 1.12+ and was fixed in those versions, but 1.11 did not have the fix and this behavior was changed in a point release.

[Adam Stokes (adam-stokes)]

We do test metrics in some capacity now:

@pytest.mark.asyncio
@pytest.mark.flaky(max_runs=5, min_passes=1)
async def test_toggle_metrics(model, tools):
    """Turn metrics on/off via the 'enable-metrics' config on kubernetes-master,
    and check that workload status returns to 'active', and that the metrics-server
    svc is started and stopped appropriately.

    """

    async def check_svc(app, enabled):
        unit = app.units[0]
        if enabled:
            await retry_async_with_timeout(
                verify_ready,
                (unit, "svc", ["metrics-server"], "-n kube-system"),
                timeout_msg="Unable to find metrics-server svc before timeout",
            )
        else:
            await retry_async_with_timeout(
                verify_deleted,
                (unit, "svc", "metrics-server", "-n kube-system"),
                timeout_msg="metrics-server svc still exists after timeout",
            )

    app = model.applications["kubernetes-master"]

    k8s_version_str = app.data["workload-version"]
    k8s_minor_version = tuple(int(i) for i in k8s_version_str.split(".")[:2])
    if k8s_minor_version < (1, 16):
        click.echo("skipping, k8s version v" + k8s_version_str)
        return

    config = await app.get_config()
    old_value = config["enable-metrics"]["value"]
    new_value = not old_value

    await set_config_and_wait(
        app, {"enable-metrics": str(new_value)}, tools, timeout_secs=240
    )
    await check_svc(app, new_value)

    await set_config_and_wait(
        app, {"enable-metrics": str(old_value)}, tools, timeout_secs=240
    )
    await check_svc(app, old_value)

If this is enough we can close this bug

[George Kraft (cynerva)]

That test checks to see if the metrics-server Service gets created, but it doesn't really test to make sure metrics are working. It wouldn't cover the case that was originally reported.