Upgrade to stein fails on creating pacemaker cluster

[Jiří Stránský]

TASK [Debug output for task: Run puppet host configuration for step 1] *********

.. snip ..

        "<13>Jun 24 11:38:27 puppet-user: Error: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]/returns: change from 'notrun' to ['0'] failed: '/sbin/pcs cluster setup tripleo_cluster cont
roller-0 addr=172.17.1.41' returned 1 instead of one of [0]",

When run manually on the node:

[root@controller-0 ~]# /sbin/pcs cluster setup tripleo_cluster controller-0 addr=172.17.1.41
Error: Host 'controller-0' is not known to pcs, try to authenticate the host using 'pcs host auth controller-0' command
Error: None of hosts is known to pcs.
Error: Errors have occurred, therefore pcs is unable to continue

After running `pcs host auth controller-0` and providing the user+password taken from hiera and re-running the upgrade step, the cluster gets created.

[Jiří Stránský (jistr)]

The hiera key for the generated password is "hacluster_pwd".

[Michele Baldessari (michele)]

Thanks for the heads-up Jiri, so yeah pcs has this new-fangled way of setting up the cluster. It was mandatory for remotes and that is where we actually added its support via https://github.com/openstack/puppet-pacemaker/commit/fde568564d43c78bf2828d3c981d5cc685fb3369

I wonder though why it only barfs after an upgrade and not on fresh deploys

[Jiří Stránský (jistr)]

@Michele Before upgrading the operating system we do `pcs cluster stop` and `pcs cluster destroy`, not sure if this can somehow cause it being different from fresh deploy. It's still possible that it was environmental issue and i was missing something, i did hit it on 2 different attempts though.

[Jiří Stránský (jistr)]

@Michele oh and one more important thing, we utilize these overrides:

https://github.com/openstack/puppet-tripleo/blob/0ad12ff23ab79e2e25baff14c7215e21f275570b/manifests/profile/base/pacemaker.pp#L124

https://github.com/openstack/puppet-tripleo/blob/0ad12ff23ab79e2e25baff14c7215e21f275570b/manifests/profile/base/pacemaker.pp#L143

But then again, when i upgraded the last controller (controller-2), the overrides should already be cleared when Puppet runs, and it still didn't authenticate on controller-2 automatically.

[Jiří Stránský (jistr)]

Stale bug, let's reopen if necessary.