Charm managed basic_auth.csv got reset, losing non-admin login credentials

[Paul Goins]

In one of our production k8s environments, after upgrading the easyrsa charm to a new revision, the customer reported an inability to pass through the basic auth prompt prior to reaching the k8s dashboard.

It appears that the /root/cdk/basic_auth.csv file got reset. It only contains the admin credentials; the other credentials which were added got dropped.

Note that it seems kubernetes-master was also upgraded about 5 days prior, however due to the overall issues re: accessing the k8s dashboard, it's possible issues which occurred during the k-m upgrade may not have been noticed until after the easyrsa upgrade. (In other words, easyrsa _might_ be a red herring here.)

[Paul Goins (vultaire)]

subscribed ~field-critical

[Tim Van Steenburgh (tvansteenburgh)]

basic_auth.csv will be naively overwritten, inadvertently deleting additions, any time setup_basic_auth() (see https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/master/reactive/kubernetes_master.py#L1721) is called.

As far as I can tell, this will happen when either:
1. The `client_password` config is changed on kubernetes-master
2. kubernetes-master is upgraded

This means:
1. You can workaround the problem right now by reapplying edits to basic_auth.csv *on the kubernetes-master leader unit* (the leader will distribute to other masters). Your edits will remain in place as long as you don't do either of the two things listed above.
2. The actual fix is to update the setup_basic_auth() function to only modify (or add) a single line to basic_auth.csv, instead of overwriting the entire file.

[Paul Goins (vultaire)]

unsubscribed ~field-critical

[Paul Goins (vultaire)]

subscribed ~field-high

[Tim Van Steenburgh (tvansteenburgh)]

https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/10

[Seyeong Kim (seyeongkim)]

@Tim

/root/cdk/basic_auth.csv is the right file to be changed?

I changed it on leader unit but it is not spreaded to the other kubernetes-mastser/x node

If I changed it on the other node, it is reverted soon.

[Tim Van Steenburgh (tvansteenburgh)]

Yeah, it appears there's second part to this bug, and that's it - the file changes are not distributed to non-leaders reliably. Working on a fix for that too.

[Paul Goins (vultaire)]

In case it helps, we ran this to propagate changes to that file: juju run --unit <kubernetes-master leader> charms.reactive clear_flag authentication.setup

[Tim Van Steenburgh (tvansteenburgh)]

PR has been updated with a fix that will ensure that changes to basic_auth.csv on the leader will be propagated to the other master units automatically (and immediately).

[George Kraft (cynerva)]

Follow-up PR (merged already): https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/13

[George Kraft (cynerva)]

PR to stable: https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/21