Block-storage-broker flakiness?

[Robert Bruce Park]

So I've recently set up a postgres instance using block-storage-broker for persistent storage, and I had the following experience:

1. Deployed the instance, created some tables and rows.

2. Destroyed the instance,

3. Deployed the instance again, confirmed that the same tables and rows still existed (confirmed block-storage-broker working).

4. Destroyed again,

5. Deployed again, observed that all tables/rows were missing inexplicably (no errors from juju or block-storage-broker whatsoever)

6. Destroyed again

7. Deployed again, observed that all tables/rows were back from persistent storage as expected.

Is there flakiness/raciness in the way block-storage-broker is handled in the charm?

[Stuart Bishop (stub)]

The easiest explanation for this behaviour is that, in the failed attempt, the mount was not yet available or the storage hooks had not yet fired. For the tables to not exist, the initial empty database is still being used (the install hook can't see the future and doesn't know storage hooks will be run). Or a bug caused the database to be destroyed, but we know that is not the case as it came back.

There is also the chance there is a bug in the postgresql charm or the block-storage-broker charm that causes the hooks to not be fired. However, the protocol is simple and I think it is unlikely.

I recommend using the juju-wait plugin available in ppa:stub/juju. This is the only known wait implementation that actually waits for a stable state and for all hooks to have completed firing. Without using this, you will always get unexpected results as your system will be in an unknown state.

Whatever the cause, any raciness will go away with the Juju 1.24 storage feature so I'm going to WONTFIX this.

[Robert Bruce Park (robru)]

Stuart, thanks for the response. This is the first I've heard of the "Juju 1.24 storage feature", can you point me towards some more info about that? do you know an ETA? Is it reasonable for me to just wait for that or will that be months and months away?

[Steve Langasek (vorlon)]

Robert, fixing this for the production deployment should not block on the availability of juju 1.24.

[Stuart Bishop (stub)]

@robru - see the Juju 1.24 release notes https://launchpad.net/juju-core/+milestone/1.24.0 . It is still considered experimental, so not yet in the main docs. It already offers a number of advantages over the BSB model, and easy to migrate charms to.