Tigera units do not become active after the first installation of the bundle

[Ebrar Leblebici]

Hi,

After the deployment of the bundle, most of the tigera units become stuck in "waiting" state and some of them become stuck in "error" state.

For the "waiting" units we have the message:

tigera-operator POD not found

And for the "error" units we have the message:

hook-failed: "calico-enterprise-relation-changed"

We have the error here for the unit in error state: https://pastebin.ubuntu.com/p/xrYrTXK3JV/

It seems it is failing when it tries to label the nodes because of the apiserver has not running yet.

After running the "juju resolved" command for the units in "error" state, after approx. 10-15 minutes all the tigera units become "active" and "idle". So, there may be a race condition?

Regards,
Ebrar

[Adrian Flynn (flynna)]

This bug requires manual user intervention and is a blocker to deploying Charmed K8S in a nightly automated cluster builds.

[Kevin W Monroe (kwmonroe)]

Thanks for the report. Will you share the bundle used to hit this? Or otherwise let me know the steps to reproduce?

[Adrian Flynn (flynna)]

Unable to share bundle on a public accessible site.

[Kevin W Monroe (kwmonroe)]

Taking a closer look at the failure from the description, I can tell the charm is going through lifecycle hooks faster than it should; e.g. attempting to use kubectl before the api server is up.

This charm has significant config requirements, and if they're not all provided at deployment (and if the hook timing isn't quite right), we'll end up in the described failed state.

I made adjustments to the charm so that it only proceeds as far as the provided config allows. IOW, if you haven't provided image registry credentials, it won't try to pull images. If you haven't provided CIDR ranges, it won't attempt to configure bgp peering data.

PR for review:

https://github.com/charmed-kubernetes/charm-calico-enterprise/pull/5

[Adrian Flynn (flynna)]

I have over the last 24 hours had to deploy a charmed cluster 4 times before it would successfully deploy. This is really not a place we want to be when deploying Kubernetes clusters.

Is the PR above likely to fix things? Any ETA when the fix will be available?

I had updated support case 00380420.

Thanks

Regards

Adrian

[Kevin W Monroe (kwmonroe)]

cherry picks to release_1.29:

https://github.com/charmed-kubernetes/charm-calico-enterprise/commit/e3c1316c21d5ca01a5f8615510d61b2190d9db74

https://github.com/charmed-kubernetes/charm-calico-enterprise/commit/5a696f268f97ec9ec22440a891d6b6445f50e3e4

[Kevin W Monroe (kwmonroe)]

overnight builds should get this landed in the 1.29/candidate channel, with promotion to 1.29/stable shortly after (pending ci for the whole 1.29+ck1 release).

[Ebrar Leblebici (birru2)]

Thank you Kevin for your effort on this one. But we need a backport to 1.28. Can you please help on this one, too?