Cannot upgrade to 1.24, stuck on waiting for kubelet

[Hans van den Bogert]

Following the official documentation and doing a inplace upgrade to 1.24 result in a juju status with the following:

    kubernetes-worker/86* waiting idle 56 192.168.1.203 80/tcp,443/tcp Waiting for kubelet to start.

Looking at journalctl for the kubelet on that node, shows:

```
Aug 07 16:42:16 node2 kubelet.daemon[24423]: Error: failed to parse kubelet flag: unknown flag: --network-plugin
Aug 07 16:42:16 node2 kubelet.daemon[24423]: Usage:
Aug 07 16:42:16 node2 kubelet.daemon[24423]: kubelet [flags]
Aug 07 16:42:16 node2 kubelet.daemon[24423]: Flags:
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --add-dir-header If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --address ip The IP address for the Kubelet to serve on (set to '0.0.0.0' or '::' for listening in all interfaces and IP families) (default 0.0.0.0) (DEPRECATED: This parameter should be set via the config file spec
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --allowed-unsafe-sysctls strings Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --alsologtostderr log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --anonymous-auth Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymou
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --application-metrics-count-limit int Max number of application metrics to store (per container) (default 100) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --authentication-token-webhook Use the TokenReview API to determine authentication for bearer tokens. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/d
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --authentication-token-webhook-cache-ttl duration The duration to cache responses from the webhook token authenticator. (default 2m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://k
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --authorization-mode string Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. (default "AlwaysAllow") (DEPRECATED: This param
Aug 07 16:42:16 node2 kubelet.daemon[24423]: --authorization-webhook-cache-authorized-ttl duration The duration to cache 'authorized' responses from the webhook authorizer. (default 5m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet
```

[George Kraft (cynerva)]

This looks like it was fixed by https://github.com/charmed-kubernetes/layer-kubernetes-common/pull/27 for CK 1.24.

What revision of the kubernetes-worker charm did you encounter this with? I suspect you're on an older revision that doesn't support k8s 1.24, in which case you will need to upgrade the charm.

[Hans van den Bogert (hbogert)]

I think this suffices ?

    $ juju status | grep worker | head -n1
    kubernetes-worker 1.24.3 blocked 5 kubernetes-worker stable 838 yes Needs manual upgrade, run the upgrade action

[George Kraft (cynerva)]

Thanks, yeah, that's charm revision 838 which is part of Charmed Kubernetes 1.23, which doesn't support Kubernetes 1.24. You'll need to upgrade to Charmed Kubernetes 1.24. The upgrade guide for that is here: https://ubuntu.com/kubernetes/docs/1.24/upgrading

[Hans van den Bogert (hbogert)]

Yeah I figured it out probably when you were typing the cause.

Why try to have a general upgrade process in https://ubuntu.com/kubernetes/docs/upgrading which shouldn't really be used, and not just force people to use the actual upgrade process they should've used without ambiguity?!

I really thought I was using the specific upgrade documentation for 1.24, as it referenced everything with 1.24, and also had the following command

    juju upgrade-charm kubernetes-master --switch kubernetes-control-plane --channel=1.24/stable

that one really made me think I was using the specific upgrade notes for 1.24.

[George Kraft (cynerva)]

Yeah, I agree, the general upgrading page does seem to be more confusing than it is helpful. I've opened an issue against the docs to follow up on that: https://github.com/charmed-kubernetes/kubernetes-docs/issues/707

[Hans van den Bogert (hbogert)]

That's the best reaction to this issue! Thanks