Vault SEGV with mysql-router 8.0.32

[Peter Colledge]

Below output is for latest/edge in focal containers however also fails with stable charm channels (eg. mysql-version: 8.0/stable, vault-version: 1.7/stable, 1.8/stable) and jammy series. Manually starting vault long after model deployment has settled still reproduces issue. Manually downgrading mysql-router deb package within the vault charm to 8.0.19 and restarting vault service resolves the issue.

Charm journalctl output:

---
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: Started HashiCorp Vault.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: Started snap.vault.vault.2156caa5-0d8a-495a-a2fb-264d83f746a2.scope.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: 2023-01-27T17:58:05.181Z [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: 2023-01-27T17:58:05.183Z [WARN] storage.mysql: No TLS specified, credentials will be sent in plaintext. To mute this warning add 'plaintext_connection_allowed' with a true value to your MySQL configuration in your config file.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: panic: runtime error: invalid memory address or nil pointer dereference
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x29fd09b]
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: goroutine 1 [running]:
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: github.com/go-sql-driver/mysql.(*mysqlConn).handleAuthResult(0xc0005ecc60, 0xc000152468, 0x14, 0x14, 0xc000152450, 0x15, 0x0, 0x0)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /build<email address hidden>/auth.go:376 +0x65b
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: github.com/go-sql-driver/mysql.(*connector).Connect(0xc00040c240, 0x5e25160, 0xc0000760c0, 0x0, 0x0, 0x0, 0x0)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /build<email address hidden>/connector.go:109 +0x596
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: database/sql.(*DB).conn(0xc00077b450, 0x5e25160, 0xc0000760c0, 0x40f901, 0x6c00000075, 0x74, 0x0)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /snap/go/9026/src/database/sql/sql.go:1301 +0x184
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: database/sql.(*DB).query(0xc00077b450, 0x5e25160, 0xc0000760c0, 0x55e790f, 0x49, 0xc001036fd8, 0x1, 0x1, 0xc000674401, 0xc00077a0d0, ...)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /snap/go/9026/src/database/sql/sql.go:1623 +0x66
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: database/sql.(*DB).QueryContext(0xc00077b450, 0x5e25160, 0xc0000760c0, 0x55e790f, 0x49, 0xc001036fd8, 0x1, 0x1, 0xc001036df8, 0x203000, ...)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /snap/go/9026/src/database/sql/sql.go:1605 +0xd4
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: database/sql.(*DB).Query(...)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /snap/go/9026/src/database/sql/sql.go:1619
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: github.com/hashicorp/vault/physical/mysql.NewMySQLBackend(0xc0004fc2d0, 0x5e85cb8, 0xc00016c030, 0x0, 0x0, 0x0, 0x0)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /build/vault/parts/vault/build/physical/mysql/mysql.go:97 +0x36b
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: github.com/hashicorp/vault/command.(*ServerCommand).setupStorage(0xc00033ea80, 0xc000b02000, 0xc0007662d0, 0x0, 0x0, 0x0)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /build/vault/parts/vault/build/command/server.go:874 +0x1be
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: github.com/hashicorp/vault/command.(*ServerCommand).Run(0xc00033ea80, 0xc0000700a0, 0x2, 0x2, 0x0)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /build/vault/parts/vault/build/command/server.go:1189 +0xc2d
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: github.com/mitchellh/cli.(*CLI).Run(0xc000a30780, 0xc000a30780, 0xc000a9a768, 0xc00076a140)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /<email address hidden>/cli.go:262 +0x41a
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: github.com/hashicorp/vault/command.RunCustom(0xc000070090, 0x3, 0x3, 0xc000766190, 0xc000078118)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /build/vault/parts/vault/build/command/main.go:180 +0x86f
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: github.com/hashicorp/vault/command.Run(...)
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /build/vault/parts/vault/build/command/main.go:88
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: main.main()
Jan 27 17:58:05 juju-62dd11-1-lxd-3 vault[18429]: /build/vault/parts/vault/build/main.go:10 +0x71
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: vault.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: vault.service: Failed with result 'exit-code'.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: snap.vault.vault.2156caa5-0d8a-495a-a2fb-264d83f746a2.scope: Succeeded.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: vault.service: Scheduled restart job, restart counter is at 5.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: Stopped HashiCorp Vault.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: vault.service: Start request repeated too quickly.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: vault.service: Failed with result 'exit-code'.
Jan 27 17:58:05 juju-62dd11-1-lxd-3 systemd[1]: Failed to start HashiCorp Vault.

juju export-bundle

applications:
  mysql-innodb-cluster:
    charm: mysql-innodb-cluster
    channel: latest/edge
    revision: 29
    series: focal
    resources:
      mysql-shell: -1
    num_units: 3
    to:
    - lxd:1
    - lxd:1
    - lxd:1
    options:
      source: distro
    constraints: arch=amd64 mem=3072
  vault:
    charm: vault
    channel: latest/edge
    revision: 79
    series: focal
    resources:
      core: -1
      vault: -1
    num_units: 1
    to:
    - lxd:1
    constraints: arch=amd64
  vault-mysql-router:
    charm: mysql-router
    channel: latest/edge
    revision: 46
    series: focal
machines:
  "1":
    series: jammy
relations:
- - vault:shared-db
  - vault-mysql-router:shared-db
- - vault-mysql-router:db-router
  - mysql-innodb-cluster:db-router

juju status --relations

Model Controller Cloud/Region Version SLA Timestamp
default lxd-controller localhost/localhost 2.9.38 unsupported 18:18:45Z

App Version Status Scale Charm Channel Rev Exposed Message
mysql-innodb-cluster 8.0.32 active 3 mysql-innodb-cluster latest/edge 29 no Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
vault blocked 1 vault latest/edge 79 no Vault service not running
vault-mysql-router 8.0.32 active 1 mysql-router latest/edge 46 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/0* active idle 1/lxd/0 192.168.10.203 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/1 active idle 1/lxd/1 192.168.10.174 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/2 active idle 1/lxd/2 192.168.10.148 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
vault/0* blocked idle 1/lxd/3 192.168.10.101 8200/tcp Vault service not running
  vault-mysql-router/0* active idle 192.168.10.101 Unit is ready

Machine State Address Inst id Series AZ Message
0 started a.example.com manual:a.example.com jammy Manually provisioned machine
1 started b.example.com manual:b.example.com jammy Manually provisioned machine
1/lxd/0 started 192.168.10.203 juju-62dd11-1-lxd-0 focal Container started
1/lxd/1 started 192.168.10.174 juju-62dd11-1-lxd-1 focal Container started
1/lxd/2 started 192.168.10.148 juju-62dd11-1-lxd-2 focal Container started
1/lxd/3 started 192.168.10.101 juju-62dd11-1-lxd-3 focal Container started

Relation provider Requirer Interface Type Message
mysql-innodb-cluster:cluster mysql-innodb-cluster:cluster mysql-innodb-cluster peer
mysql-innodb-cluster:coordinator mysql-innodb-cluster:coordinator coordinator peer
mysql-innodb-cluster:db-router vault-mysql-router:db-router mysql-router regular
vault-mysql-router:shared-db vault:shared-db mysql-shared subordinate
vault:cluster vault:cluster vault-ha peer

[Felipe Reyes (freyes)]

this looks like an occurrence of the issue tracked at https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/2003835

[Peter Colledge (pcamd)]

Can confirm fixed by just released mysql-router=8.0.32-0ubuntu0.22.04.2. Bug #2003835 indeed.

[Dongwon Cho (dongwoncho)]

https://pastebin.ubuntu.com/p/5JGRn36968/
I got the same as above but downgrading vault-mysql-router charm to 8.0.19/stable didn't work.

[Peter Colledge (pcamd)]

Dongwon Cho, the '8.0.19/stable' charm tag is broken by design (see reply to my Bug #2004046). As mentioned above, only the deb package versions can be trusted here. For focal, ensure your mysql-router deb package is 8.0.32-0ubuntu0.20.04.2 (which is in the 'security' repo and may not have had time to propagate to your local mirror at the time of your test).

[Dongwon Cho (dongwoncho)]

Now it works with '8.0/stable'. Thank you!