Race in Postgres on master node in 5.0

[Sergii Golovatiuk]

When master node is virtualized and host has pretty high load. I see the race condition as

Notice: /Stage[main]/Main/Postgresql::Db[ostf]/Postgresql::Database[ostf]/Exec[/usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf']/returns: createdb: could not connect to database template1: FATAL: the database system is starting up
Error: /usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf' returned 1 instead of one of [0]
Error: /Stage[main]/Main/Postgresql::Db[ostf]/Postgresql::Database[ostf]/Exec[/usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf']/returns: change from notrun to 0 failed: /usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf' returned 1 instead of one of [0]
Notice: /Stage[main]/Main/Postgresql::Db[ostf]/Postgresql::Database[ostf]/Postgresql::Psql[REVOKE CONNECT ON DATABASE ostf FROM public]/Exec[/bin/echo "REVOKE CONNECT ON DATABASE ostf FROM public" | /usr/bin/psql --no-password --tuples-only --quiet --dbname postgres |egrep -v -q '^$']: Dependency Exec[/usr/bin/createdb --template=template0 --encoding 'utf8' --locale=C 'ostf'] has failures: true

It means that Postgres started but DBs were not attached though puppet ran next resource though DBs were not ready.

the problem was fixed in 5.1.1 by updating Postgres manifests to latest though it's present in 5.0 (all branches) and 5.1 (only 5.1)

The actual fixes are:
https://review.openstack.org/#/c/131076/4/deployment/puppet/nailgun/examples/postgres-only.pp

Here is a root cause of issue
https://github.com/puppetlabs/puppetlabs-postgresql/pull/292/files

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/141325

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/141345

[Ryan Moe (rmoe)]

Marking invalid for all but 5.0.3 as the description mentions that this only a problem on 5.0.

[Matthew Mosesohn (raytrac3r)]

This bug will still affect 5.0.3. I couldn't merge this because we don't have a working 5.0 CI.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthew Mosesohn (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/141345

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matthew Mosesohn (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/141325