Couchbase Root Password Can Go Out Of Sync Due to Cluster Stitching

[Auston McReynolds]

if a user goes to the web console of a couchbase instance, and adds another couchbase instance as a server to create a cluster, the root password for the second server is replaced with the first server's password (they're sync'd).

this is problematic because the ping check today relies on the password in /etc/couchbase/secret_key, which is only set on prepare(). this means that the status for the second server (and third, and etc.) flips to SHUTDOWN because the state of the database can't be ascertained.

the solution is to retrieve the current password from config.dat for every ping check:

sudo /opt/couchbase/bin/erl -noinput -eval 'case file:read_file("/opt/couchbase/var/lib/couchbase/config/config.dat") of {ok, B} -> io:format("~p~n", [binary_to_term(B)]) end.' -run init stop | grep '\[{"root",\[{password,' | awk -F\" '{print $4}'

snippet of config.dat in term format:

  {rest_creds,
      [{'_vclock',[{'ns_1@127.0.0.1',{1,63580474021}}]},
       {creds,
           [{"root",[{password,"AxmFYh687UT9nC7u8XhsjZFKjxEy4DPDPaNN"}]}]}]},
  {rest,[{port,8091}]},
  {auto_failover_cfg,

alternatively, if this approach is deemed too expensive, perhaps the error on the ping check can be interpreted and attempt a refresh of /etc/couchbase/secret_key.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/129095

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/129095
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=3190c0ac5b37506fbe658244d09be2a3d9b8273e
Submitter: Jenkins
Branch: master

commit 3190c0ac5b37506fbe658244d09be2a3d9b8273e
Author: amcrn <email address hidden>
Date: Thu Oct 16 17:35:26 2014 -0700

    Couchbase Root Password Can Go Out Of Sync

    if a user goes to the web console of a couchbase instance and adds
    another couchbase instance as a server to create a cluster, the root
    password for the second server is replaced with the first server's
    password (aka they're synced).

    this is problematic because the ping check today relies on the
    password in /etc/couchbase/secret_key, which is only set on
    prepare(). this means that the status for the second server (and
    the third, and etc.) flips to 'shutdown' because the state of the
    database cannot be ascertained.

    the solution is to retrieve the current password from config.dat if
    the status check fails with the stored value in secret_key. if the
    password parsed from config.dat works, update the stored value in
    secret_key.

    Change-Id: Ia39b9c57df88b6784d13d8f3fd86967c7cffa743
    Closes-Bug: #1380984