EasyRSA scale out broken

[Tim Van Steenburgh]

Opened by jacekn on 2017-03-23 10:05:00+00:00 at https://github.com/juju-solutions/layer-easyrsa/issues/9

------------------------------------------------------------

I use easyrsa with k8s charms. I wanted to scale it out for HA but juju-add-unit easyrsa caused the following hook error:
# ./hooks/client-relation-changed

Easy-RSA error:

Missing expected CA file: serial (perhaps you need to run build-ca?)
Run without commands for usage and command help.
Traceback (most recent call last):
  File "./hooks/client-relation-changed", line 19, in <module>
    main()
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/__init__.py", line 78, in main
    bus.dispatch()
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 434, in dispatch
    _invoke(other_handlers)
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 417, in _invoke
    handler.invoke()
  File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 291, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-easyrsa-1/charm/reactive/easyrsa.py", line 226, in create_server_cert
    server_cert, server_key = create_server_certificate(cn, sans, name)
  File "/var/lib/juju/agents/unit-easyrsa-1/charm/reactive/easyrsa.py", line 263, in create_server_certificate
    check_call(split(server))
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./easyrsa', '--batch', '--req-cn=10.25.61.29', '--subject-alt-name=IP:10.25.61.29,IP:10.25.61.29,DNS:juju-555a14-juju2-is-kubernetes-105-6', 'build-server-full', 'kubernetes-worker_0', 'nopass', '2>&1']' returned non-zero exit status 1

====================== COMMENTS ============================

Comment created by mbruzek on 2017-03-23 16:07:29+00:00

It looks like the other easyrsa unit does not have the CA. We may need to change the code so only the leader signs the keys and certs and the other easyrsa units are ready in standby mode. However I do not know if easyrsa supports importing other server's PKI and signing certs. We need to come up with a strategy of how it should work and more testing and investigation are needed here.

@jacekn If you have more experience with easyrsa or have an architectural design that could help this issue, please leave a comment that would be appreciated.

------------------------------------------------------------

Comment created by hansbogert on 2018-01-01 22:49:34+00:00

In my case the common name of the CA is an ip address of the initial host which create the CA certificate, so if that's correct, then high-availability is even more problematic. In fact I experienced a "unknown certificate authority" when trying to do a manual switch from one easyrsa unit to a different one, on a different MAAS node.

[Xav Paice (xavpaice)]

I've added the canonical-bootstack tag here, as this affects our production Bootstack environments in the following way:

Firstly, we run K8s clouds, and have a single easyrsa unit there which is a single point of failure. I've not dug into recovery options for this as yet.

Secondly, the Openstack deployments use Easyrsa to provide a TLS cert for etcd, which is used by Vault, which stores the LUKs keys for Ceph. With a single unit of easyrsa, if we lose the host it resides on, re implementing a new easyrsa unit breaks the etcd cluster rendering it unusable, which in turn would do nasty things to Vault.

[Xav Paice (xavpaice)]

adding field-high, as the current implementation needs some design review

[Tim Van Steenburgh (tvansteenburgh)]

Thanks Xav. This is an old bug that should actually be closed now, since in the meantime we've provided a new recommended way to establish a HA certificate authority, using Vault instead of EasyRSA. Read https://ubuntu.com/kubernetes/docs/using-vault for details on using Vault with Charmed Kubernetes.

There are no plans to provide a HA configuration for EasyRSA; Vault should be used if HA is needed.

[James Troup (elmo)]

Tim Van Steenburgh <email address hidden> writes:

> Thanks Xav. This is an old bug that should actually be closed now, since
> in the meantime we've provided a new recommended way to establish a HA
> certificate authority, using Vault instead of EasyRSA. Read
> https://ubuntu.com/kubernetes/docs/using-vault for details on using
> Vault with Charmed Kubernetes.
>
> There are no plans to provide a HA configuration for EasyRSA; Vault
> should be used if HA is needed.

Err, I have so many questions about this, but let's put them to one
side for now.

Canonical OpenStack supports full disk encryption. It does this using
vault. vault uses etcd as a backing store. etcd gets its TLS from
easyrsa.

I don't think having an EasyRSA charm which doesn't support scale out
is a super reasonable position, TBH.

--
James

[Tim Van Steenburgh (tvansteenburgh)]

>
> Canonical OpenStack supports full disk encryption. It does this using
> vault. vault uses etcd as a backing store. etcd gets its TLS from
> easyrsa.
>

In this case I suspect that EasyRSA is there to bootstrap etcd certs, since
etcd must be up-and-running in order for Vault to enter HA mode. But you
can also achieve the same result without EasyRSA, by starting with a single
unit of Vault (which provides certs to etcd), and then scaling Vault up
after etcd is running.

[James Hebden (ec0)]

Tim - Are you aware of a migration path between the easyrsa charm and the vault charm? Are users wanting to migrate from a non-HA deployment to a HA deployment based on Vault considered a supported use case?

[Dean Henrichsmeyer (dean)]

There has to be a migration path. A solution involving new technology is not a solution for existing users unless there is a documented, tested migration path.

[Tim Van Steenburgh (tvansteenburgh)]

James H - Yes and yes, but we're missing docs for that migration. Working on that now.

[Xav Paice (xavpaice)]

For the sake of clarity, there's two models right now where we need to have a migration path:

- OpenStack models, where we use Vault for encryption at rest, with a backing data store of etcd (which is bootstrapped using certs from easyrsa)
- Kubernetes models, where all certs are provided by easyrsa and there's no Vault.

For the first scenario, the existing Vault could be migrated to use a fresh store, possibly mysql rather than etcd even, and that would allow us to remove the easyrsa charm when switching to HA.

For the second, easyrsa is more tightly knitted into the environment and we need to determine exactly what and how to migrate from easyrsa to vault.

[Cory Johns (johnsca)]

https://github.com/charmed-kubernetes/kubernetes-docs/pull/223 adds instructions to transition a CDK cluster from EasyRSA to Vault and improves the documentation for making Vault HA.

You can preview the changes here: https://deploy-preview-223--cdk-docs-next.netlify.com/kubernetes/docs/using-vault#transitioning-an-existing-cluster-from-easyrsa-to-vault (the HA instructions are just a bit further down)

[George Kraft (cynerva)]

In testing, we found that the TLS client charms do not handle the transition from easyrsa to vault properly. I've added all of the affected components to this issue.

Cory opened initial PRs (linked below) but the issue has since been handed off to me. I'm still working through some issues with the transition on kubernetes-master.

PRs (WIP):
https://github.com/charmed-kubernetes/layer-etcd/pull/158
https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/pull/2
https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/34
https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/22
https://github.com/charmed-kubernetes/charm-flannel/pull/55
https://github.com/charmed-kubernetes/layer-calico/pull/38
https://github.com/charmed-kubernetes/layer-canal/pull/35
https://github.com/charmed-kubernetes/layer-tigera-secure-ee/pull/17

[George Kraft (cynerva)]

Another PR: https://github.com/charmed-kubernetes/cdk-addons/pull/133

Still working on this. Testing has been slow as there are a lot of components impacted by the CA change, many of which do not show obvious symptoms of failure until logs have been inspected.

[George Kraft (cynerva)]

The cdk-addons part of this fix has been backported to release-1.15, release-1.14, release-1.13.

https://github.com/charmed-kubernetes/cdk-addons/pull/134
https://github.com/charmed-kubernetes/cdk-addons/pull/135
https://github.com/charmed-kubernetes/cdk-addons/pull/136

[George Kraft (cynerva)]

Charm fixes cherry-picked to stable branches:

https://github.com/charmed-kubernetes/layer-calico/commit/61195204210763e2e9c122fa75e85d4eafeca1c2
https://github.com/charmed-kubernetes/layer-canal/commit/1c3be100d9234f9c66c10fd8894b39db1cd18778
https://github.com/charmed-kubernetes/layer-etcd/commit/1dce7d1d39269053d6560d469768ddb8fc611c73
https://github.com/charmed-kubernetes/charm-flannel/commit/1114dc1068eb6a4fba8a5f6b8c44bae9615ab5fe
https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/commit/86ef8861a7efb532a64c9e3c54271960934ff7c5
https://github.com/charmed-kubernetes/charm-kubernetes-master/commit/76a309e8c55068e2e3c47ccf6161ab6d6b0ffd5c
https://github.com/charmed-kubernetes/charm-kubernetes-worker/commit/3f690ce22dbf3a084448773ed46eacbeb26bd1cf
https://github.com/charmed-kubernetes/layer-tigera-secure-ee/commit/ec615c241ceb6e446d53fcc56b7613dad4f5606b