snap refresh fails when upgrading Charm from 1.7/stable to 1.8/stable

[DUFOUR Olivier]

Environment :
Ubuntu : Jammy 22.04
Juju : 2.9.42
MaaS : 3.3
Original channel of Vault : 1.7/stable
Openstack : Yoga
Vault is providing the CA certificate and all certificates to related units.

When running :
juju refresh vault --channel 1.8/stable
The Vault unit are staying in Active / executing state with the error seeing below :
ubuntu@maas:~$ juju status vault
Model Controller Cloud/Region Version SLA Timestamp
yoga juju-default juju/default 2.9.42 unsupported 04:25:17Z

App Version Status Scale Charm Channel Rev Exposed Message
vault 1.7.9 active 1 vault 1.8/stable 108 no Unit is ready (active: true, mlock: disabled)
vault-mysql-router 8.0.33 active 1 mysql-router 8.0/stable 35 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
vault/0* active executing 0/lxd/12 192.168.29.143 8200/tcp (upgrade-charm) Unit is ready (active: true, mlock: disabled)
  vault-mysql-router/0* active idle 192.168.29.143 Unit is ready

2023-06-06 04:28:14 WARNING unit.vault/0.upgrade-charm logger.go:60 All snaps up to date.
2023-06-06 04:28:15 INFO unit.vault/0.juju-log server.go:316 Invoking reactive handler: reactive/vault_handlers.py:339:upgrade_charm
2023-06-06 04:28:15 INFO unit.vault/0.juju-log server.go:316 Invoking reactive handler: reactive/vault_handlers.py:188:snap_refresh
2023-06-06 04:28:15 INFO unit.vault/0.juju-log server.go:316 Refreshing vault from store
2023-06-06 04:28:15 ERROR unit.vault/0.juju-log server.go:316 Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 195, in snap_refresh
    snap.refresh('vault', channel=channel)
  File "/var/lib/juju/agents/unit-vault-0/charm/lib/charms/layer/snap.py", line 113, in refresh
    _refresh_store(snapname, **kw)
  File "/var/lib/juju/agents/unit-vault-0/charm/lib/charms/layer/snap.py", line 421, in _refresh_store
    out = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['snap', 'refresh', '--amend', '--channel=1.8/stable', 'vault']' returned non-zero exit status 1.

When running manually the snap refresh, snap will complain Vault is currently running and should be stopped before refreshing the snap :
ubuntu@juju-74c2fd-0-lxd-12:~$ sudo snap refresh vault --channel 1.8/stable
error: cannot refresh "vault": snap "vault" has running apps (vault), pids: 44110

The charm should preemptively attempt to stop Vault service first before refreshing the snap itself.

Current manual workaround is to stop Vault snap and then refresh it (or let the charm re-attempt the refresh) :
$ sudo systemctl stop vault
$ sudo snap refresh vault --channel 1.8/stable

This might be related to the issue #2008760 and partially related to #2007587