consolidated database backend for Vault HA

[Nobuto Murata]

Vault itself supports to use etcd for both storage and ha_storage backends. However, our charm's architecture assumes to have both mysql and etcd for respective purpose.

Would be nice to have a functionality to select either MySQL (for OpenStack deployment scenario) or etcd (for Kubernetes scenario) so that we can have less number of databases in one model.
- https://www.vaultproject.io/docs/configuration/storage/etcd
- https://www.vaultproject.io/docs/configuration/storage/mysql

Or going to the embedded raft storage in Vault would be a way:
https://www.vaultproject.io/docs/configuration/storage/raft

[Nobuto Murata (nobuto)]

https://review.opendev.org/c/openstack/charm-vault/+/751692

[Andre Ruiz (andre-ruiz)]

How about mysql only instead of etcd only?

[Samuel Allan (samuelallan)]

Working on a patch at https://review.opendev.org/c/openstack/charm-vault/+/838437

[Nobuto Murata (nobuto)]

> How about mysql only instead of etcd only?

etcd only is useful for Kubernetes models, and mysql only is useful for OpenStack models. OpenStack deployment assumes MAAS provider by nature so it has less resource limitation since etcd can be deployed as LXD containers. So from a resource allocation point of view, etcd only backend has some priority IMHO. Anyway, the mysql one is tracked as:
https://bugs.launchpad.net/vault-charm/+bug/1835356

[Samuel Allan (samuelallan)]

> How about mysql only instead of etcd only?

With my implementation of the fix to this bug, I'm aiming to make it easy to add support for other storage and HA backends in the future. So I think we can be optimistic about supporting mysql only later. :)

[Samuel Allan (samuelallan)]

Nobuto, Billy suggested we consider switching to Vault's integrated storage to simplify things and reduce dependencies even further:

> I'm wondering if we should actually support etcd as a backend, or just go straight to the raft based integrated storage provided with Vault in newer versions (see [1] and [2]). This would further remove additional requirements of applications.

> The move to integrated storage doesn't necessarily remove the need to support different backends in a flexible way - but may not run into some of the various tangles involved with the current interfaces without running into awkwardness in the migration paths, etc.

> [1] - https://www.vaultproject.io/docs/internals/integrated-storage
> [2] - https://www.vaultproject.io/docs/configuration/storage/raft

What are your thoughts on this?

I can see two potential challenges:

1. requires vault to be a new enough version to support it. Not sure how it works with charms and enforcing software versions, or how this would work with updating the charm.

2. potentially breaks backwards compatibility with the charm.

[Billy Olsen (billy-olsen)]

> I can see two potential challenges:

> 1. requires vault to be a new enough version to support it. Not sure how it works with charms and > enforcing software versions, or how this would work with updating the charm.

There shouldn't be any problems with a new enough version of Vault, as we have several versions via snap and the charm supports different channels/versions of vault.

> 2. potentially breaks backwards compatibility with the charm.

The charm can check the snap channel to determine behavior if we really want to. Alternatively, the charm can default to the internal storage and if the db relation is added, it falls back to its previous behavior. This allows the upgrade path and the new direction. Can also do the same with etcd since they are used for two separate purposes today, you can have different behaviors.

[Nobuto Murata (nobuto)]

Here is my personal view.

Using Vault with embedded raft storage is the way to go *for the future* for sure. Because we may want to have Vault even before having MAAS for example to save some secrets into Vault. In that case, bootstrapping Vault by itself is ideal without requiring any external storage.

If the enablement comes in a few weeks, that would be great. But in reality it would take some time. The request here as using consolidated external backend storage such as etcd only is a stop-gap solution as something we can get now. i.e. it's definitely better than the current state but it shouldn't require too much effort.

[Felipe Reyes (freyes)]

> Alternatively, the charm can default to the internal storage and if the db relation is added, it falls back to its previous behavior.

I was thinking in a similar behavior.

This is the way I think things could work and allow us to have a different set of storage backends.

you deploy vault, add the relations you want (if any), then you initialize it, when the charm detects that vault was initialized, then at that point considers that those are the relations that will be used from now now. At the moment there is no action to initialise vault and this happens outside the charm's control, this should change probably.

After the action "initialise" is run the charm would get into waiting state if any of the relations is removed.

Scenario 1:

juju deploy vault
juju add-relation etcd vault
juju run-action vault/0 initialise # at this point etcd is used for storage

Scenario 2:

juju deploy vault
juju add-relation etcd vault
juju add-relation vault-mysql-router vault
juju run-action vault/0 initialise # at this point etcd is used for leadership and mysql for storage.

Scenario 3:

juju deploy vault
juju add-relation vault-mysql-router vault
juju run-action vault/0 initialise # at this point mysql is used for storage

Scenario 4:

juju deploy vault
juju run-action vault/0 initialise # at this point vault's internal storage is used.

[Corey Bryant (corey.bryant)]

Could we add config options as an alternative to adding an initialize action? In other words this would be very similar to what Felipe mentioned above except that the config options explicitly specify what backend to use.

storage-backend:
  type: string
  default: mysql
  description: could be "etcd", "mysql", "raft"

ha-storage-backend
  type: string
  default: etcd
  description: could be "etcd", "mysql", "raft"

[Billy Olsen (billy-olsen)]

I believe that the detailed scenario laid out by @freyes in comment #9 is the way to go. It has a nice default for moving forward, but allows for the legacy path and more complicated scenarios. Let's move in that direction.

As far as the options are concerned (comment 10), I think that would be okay - but it adds a bit of complexity as far as validating and verifying what the right type of configuration to have is. It does have the benefit of being able to block and indicate the user has not provided a valid configuration, but its almost nicer if it just works for basic installs and users can adopt the more complexity if their environment requires it.

[Nobuto Murata (nobuto)]

Having the new "initialize" action is cleaner but it requires yet another human intervention and also breaks the backward compatibility. i.e. existing bundles and process stop working. We thought that having an additional config option can keep the current behavior if no additional config is supplied.

[Felipe Reyes (freyes)]

On Sat, 2022-06-18 at 00:07 +0000, Nobuto Murata wrote:
> Having the new "initialize" action is cleaner but it requires yet
> another human intervention and also breaks the backward compatibility.
> i.e. existing bundles and process stop working.

For already deployed environments going through "juju upgrade-charm", on the
upgrade-charm hook we can mark go through the "initialize" logic (e.g.
set_flag() ) to stamp/lock what the storage backends will be from now on.

When it comes to bundles that "just work" today, you are right, they would need
to run a new action before running "vault operator init".

Maybe the vault charm could rely on "goal-state" to determine what will be its
storage configuration.

[Samuel Allan (samuelallan)]

For reference, we decided to pause development on this and implement raft support in the vault charm instead. This will be more work, but should support the same end goal: reduce number of required services in a mysql-only or etcd-only environment.

[Nobuto Murata (nobuto)]

The patch by Samuel has been merged into the master branch.
https://review.opendev.org/c/openstack/charm-vault/+/847964
https://opendev.org/openstack/charm-vault/commit/61fdf9ca773e1e19fed04a630bab13e17c5bdfbb

I don't know why there is no automated message so I'm posting it by hand. The bug link in the commit message above is linked to:
https://bugs.launchpad.net/charm-vault/+bug/1883242
instead of
https://bugs.launchpad.net/vault-charm/+bug/1883242
and that might be a reason.