certificates-relation-departed breaks on error loop after removing barbican

[Pedro Guimarães]

Using Bionic/stein (stable) charms on Juju 2.6.10.

To redeploy some applications, I ran the following:
juju remove-application --force octavia barbican

After that Vault moved to error state with following logs:
subprocess.CalledProcessError: Command '['relation-get', '--format=json', '-r', 'certificates:323', '-', 'barbican/0']' returned non-zero exit status 1.

More complete logs: https://pastebin.canonical.com/p/4b4rdNmZYR/

I've seen that happening before and I don't recall using --force all the time.
I am not entirely sure if that flag influenced or not this issue.

[Aurelien Lourot (aurelien-lourot)]

Thanks for reporting! I can't find a specific charms.reactive or charm-helpers commit that would be the fix for this, yet it seems to have been fixed meanwhile. We see in the original logs [0] that in order to reproduce this we need a bundle with barbican connected to vault over the `certificates` relation. Thus I created a bionic-stein.yaml bundle [1] and did this a few times:

juju deploy ./bionic-stein.yaml
juju remove-application --force barbican

Each time in my case the vault unit managed to run the certificates-relation-departed and certificates-relation-joined with no issues. Also the `--force` flag shouldn't have any impact on the errors of applications not being removed. For the record I'm using Juju 2.8.7.

Closing this bug for now. Don't hesitate to re-open if you still encounter this and please provide more information (e.g. charm versions and bundle). Thanks!

[0] https://pastebin.canonical.com/p/4b4rdNmZYR/
[1] https://paste.ubuntu.com/p/nrsrSnQrMw/

[Reginald Carey (reggiecarey)]

This has occurred with me as well.

Attempting to force remove barbican to hopefully induce vault to exit hook failed:"certificates-relation-departed" state.

What is the resolution to the

hook failed:"certificates-relation-departed"

message that vault finds itself stuck in?

[Billy Olsen (billy-olsen)]

You should be able to get past the hook failure with:

juju resolved --no-retry vault/<errored_unit>

This will cause juju to mark the error as resolved and *not* retry the hook.