Comment 2 for bug 1988448

It looks like vault had an internal server error due to a dropped mysql connection:

Sep 1 07:46:51 ip-172-31-46-20 vault[50291]: [mysql] 2022/09/01 07:46:51 packets.go:36: unexpected EOF
Sep 1 07:46:51 ip-172-31-46-20 vault[50291]: 2022-09-01T07:46:51.279Z [ERROR] core: failed to create token: error="failed to persist accessor index entry: invalid connection"
Sep 1 07:46:52 ip-172-31-46-20 vault[50291]: [mysql] 2022/09/01 07:46:52 packets.go:122: closing bad idle connection: EOF

and I think the mysql connection dropped because the primary was in the process of shutting down at that time:

2022-09-01T07:46:07.179532Z 144 [System] [MY-011510] [Repl] Plugin group_replication reported: 'This server is working as primary member.'
2022-09-01T07:46:50.168380Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-09-01T07:46:53.271039Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
2022-09-01T07:46:55.272634Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 219 user: 'vault'.

and it looks like the primary was shutting down because the mysql-innodb-cluster charm was in the middle of a rolling restart across its units, to add certificates to their configuration.

This looks like a transient issue that kubernetes-control-plane got hung up on. We probably just need to make kubernetes-control-plane a little more resilient against this InternalServerError exception. We can catch it and retry.