A slave won't use the storage mountpoint

[Andreas Hasenack]

A postgresql slave, when used with the storage broker charm and cinder volumes, did the following when coming up:
a) cloned the master
b) mounted the cinder volume
c) rsynced /var/lib/postgresql/9.3/main to /srv/data/postgresql/9.3/main/
d) started the slave but with data_directory set to /var/lib/postgresql/9.3/main

I think there are two bugs here:
a) it cloned the master before mounting the volume, therefore using the root filesystem. There might not be enough space there
b) it restarted the slave pointing at the original /var/lib directory instead of the new mount point

It could also be optimized to perhaps use the new mountpoint for the cloning in the first place, then it wouldn't need the rsync step.

[Andreas Hasenack (ahasenack)]

We have this scenario live right now. Sorry for not attaching logs, I wanted to get this filed before EOD (and before my battery died, it's at 7% now!! :)

[Stuart Bishop (stub)]

A unit has no way of knowing that there will ever be storage attached. To fix a), we need to change this and add a configuration option that blocks at the cloning step until storage is available. Using required Juju native storage would fix the problem too, but we can't use that until Bug #1504658 is addressed.

b) is normal behavior, as /var/lib/postgresql/9.3/main is replaced with a symlink. I don't trust every tool to understand that $PGDATA is relocatable.

This issue also affects optional Juju native storage, which will be landed as soon as I have fixed Juju stability issues in my testing environment.

[Andreas Hasenack (ahasenack)]

Regarding the unit not knowing that storage will be attached, this was a add-unit operation and the service (postgresql) was already related to the storage subordinate. It might be an optimization, but it kind of knows already that it will get storage.

Regarding (b), interesting trick, I didn't know about it:
rwxrwxrwx 1 root root 29 Aug 19 22:06 main -> /srv/data/postgresql/9.3/main
drwx------ 15 postgres postgres 4096 Aug 19 22:04 main-1471644281

So that seems to be ok.

[Stuart Bishop (stub)]

Yes, for the storage subordinate a leader setting would be enough to flag that new units should wait for storage.

For native storage, I'll need the setting since you add storage individually to units rather than declare all units get particular storage.

[Andreas Hasenack (ahasenack)]

Bug #1504658 is marked as wontfix for juju-1.25 :(
I'll bring it up in the next juju cross-team meeting.