pcs cluster auth unable to connect

[ndsipa pomu]

After installing PCS on (x)Ubuntu 16.04, I'm unable to successfully run "pcs auth cluster".
I was following the documentation from http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/_enable_pcs_daemon.html

To reproduce:

install Ubuntu 16.04
set up networking
install pcs: apt-get -y install pcs
start pcsd: systemctl start pcsd
(optionally enable it on boot: systemctl enable pcsd)
set password for "hacluster": passwd hacluster
attempt cluster authorisation (which fails): pcs cluster auth <hostname>

example:
root@uk01pvmh020:~# pcs cluster auth uk01pvmh020
Username: hacluster
Password:
Error: Unable to communicate with uk01pvmh020
root@uk01pvmh020:~#

Following the same procedure with RHEL7 actually works, so as a workaround I could eschew the use of Ubuntu for clustering services (am looking to get clvm working).

Here's the output with "--debug" enabled:

root@uk01pvmh020:~# pcs --debug cluster auth uk01pvmh020
Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb read_tokens
--Debug Input Start--
{}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
  },
  "log": [
    "I, [2016-05-21T17:38:52.731105 #14594] INFO -- : PCSD Debugging enabled\n",
    "D, [2016-05-21T17:38:52.731174 #14594] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-05-21T17:38:52.731207 #14594] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-05-21T17:38:52.731229 #14594] INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-05-21T17:38:52.740894 #14594] DEBUG -- : [\"totem.cluster_name (str) = debian\\n\"]\n",
    "D, [2016-05-21T17:38:52.741017 #14594] DEBUG -- : []\n",
    "D, [2016-05-21T17:38:52.741059 #14594] DEBUG -- : Duration: 0.009645066s\n",
    "I, [2016-05-21T17:38:52.741155 #14594] INFO -- : Return Value: 0\n",
    "W, [2016-05-21T17:38:52.741423 #14594] WARN -- : Cannot read config 'tokens' from '/var/lib/pcsd/tokens': No such file or directory @ rb_sysopen - /var/lib/pcsd/tokens\n",
    "E, [2016-05-21T17:38:52.741508 #14594] ERROR -- : Unable to parse tokens file: A JSON text must at least contain two octets!\n"
  ]
}
--Debug Output End--

Sending HTTP Request to: https://uk01pvmh020:2224/remote/check_auth
Data: None
Response Reason: Tunnel connection failed: 403 Forbidden
Username: hacluster
Password:
Running: /usr/bin/ruby -I/usr/share/pcsd/ /usr/share/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"username": "hacluster", "local": false, "nodes": ["uk01pvmh020"], "password": "retsulcah", "force": false}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "uk01pvmh020": {
        "status": "noresponse"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [

    ],
    "sync_responses": {
    }
  },
  "log": [
    "I, [2016-05-21T17:39:00.392737 #14611] INFO -- : PCSD Debugging enabled\n",
    "D, [2016-05-21T17:39:00.392806 #14611] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-05-21T17:39:00.392838 #14611] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-05-21T17:39:00.392860 #14611] INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-05-21T17:39:00.402354 #14611] DEBUG -- : [\"totem.cluster_name (str) = debian\\n\"]\n",
    "D, [2016-05-21T17:39:00.402461 #14611] DEBUG -- : []\n",
    "D, [2016-05-21T17:39:00.402513 #14611] DEBUG -- : Duration: 0.009475549s\n",
    "I, [2016-05-21T17:39:00.402595 #14611] INFO -- : Return Value: 0\n",
    "W, [2016-05-21T17:39:00.403098 #14611] WARN -- : Cannot read config 'tokens' from '/var/lib/pcsd/tokens': No such file or directory @ rb_sysopen - /var/lib/pcsd/tokens\n",
    "E, [2016-05-21T17:39:00.403238 #14611] ERROR -- : Unable to parse tokens file: A JSON text must at least contain two octets!\n",
    "I, [2016-05-21T17:39:00.403273 #14611] INFO -- : SRWT Node: uk01pvmh020 Request: check_auth\n",
    "E, [2016-05-21T17:39:00.403298 #14611] ERROR -- : Unable to connect to node uk01pvmh020, no token available\n",
    "I, [2016-05-21T17:39:00.409109 #14611] INFO -- : No response from: uk01pvmh020 request: /auth, exception: 403 \"Forbidden\"\n"
  ]
}
--Debug Output End--

Error: Unable to communicate with uk01pvmh020
root@uk01pvmh020:~#

For reference, here's the debug output from a RHEL7/OL7 machine:

[root@uk01vort003 pcsd]# pcs --debug cluster auth uk01vort003
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb read_tokens
--Debug Input Start--
{}
--Debug Input End--

Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
  },
  "log": [
    "I, [2016-05-21T17:43:55.516093 #28868] INFO -- : PCSD Debugging enabled\n",
    "D, [2016-05-21T17:43:55.516217 #28868] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-05-21T17:43:55.516290 #28868] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-05-21T17:43:55.516361 #28868] INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-05-21T17:43:55.520897 #28868] DEBUG -- : []\n",
    "D, [2016-05-21T17:43:55.520995 #28868] DEBUG -- : Duration: 0.004518704s\n",
    "I, [2016-05-21T17:43:55.521117 #28868] INFO -- : Return Value: 1\n",
    "W, [2016-05-21T17:43:55.521284 #28868] WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file or directory - /etc/corosync/corosync.conf\n",
    "W, [2016-05-21T17:43:55.521740 #28868] WARN -- : Cannot read config 'tokens' from '/var/lib/pcsd/tokens': No such file or directory - /var/lib/pcsd/tokens\n",
    "E, [2016-05-21T17:43:55.521844 #28868] ERROR -- : Unable to parse tokens file: A JSON text must at least contain two octets!\n"
  ]
}

--Debug Output End--

Sending HTTP Request to: https://uk01vort003:2224/remote/check_auth
Data: None
Response Reason: Tunnel connection failed: 403 Forbidden
Username: hacluster
Password:
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"username": "hacluster", "local": false, "nodes": ["uk01vort003"], "password": "retsulcah", "force": false}
--Debug Input End--

Return Value: 0
--Debug Output Start--
{
  "status": "ok",
  "data": {
    "auth_responses": {
      "uk01vort003": {
        "status": "ok",
        "token": "0d262df4-7f1b-4687-acc2-73e1febec81d"
      }
    },
    "sync_successful": true,
    "sync_nodes_err": [

    ],
    "sync_responses": {
    }
  },
  "log": [
    "I, [2016-05-21T17:44:02.473670 #28892] INFO -- : PCSD Debugging enabled\n",
    "D, [2016-05-21T17:44:02.473833 #28892] DEBUG -- : Did not detect RHEL 6\n",
    "I, [2016-05-21T17:44:02.473904 #28892] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
    "I, [2016-05-21T17:44:02.473974 #28892] INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-05-21T17:44:02.480170 #28892] DEBUG -- : []\n",
    "D, [2016-05-21T17:44:02.480552 #28892] DEBUG -- : Duration: 0.006169278s\n",
    "I, [2016-05-21T17:44:02.480737 #28892] INFO -- : Return Value: 1\n",
    "W, [2016-05-21T17:44:02.481035 #28892] WARN -- : Cannot read config 'corosync.conf' from '/etc/corosync/corosync.conf': No such file or directory - /etc/corosync/corosync.conf\n",
    "W, [2016-05-21T17:44:02.481756 #28892] WARN -- : Cannot read config 'tokens' from '/var/lib/pcsd/tokens': No such file or directory - /var/lib/pcsd/tokens\n",
    "E, [2016-05-21T17:44:02.481865 #28892] ERROR -- : Unable to parse tokens file: A JSON text must at least contain two octets!\n",
    "I, [2016-05-21T17:44:02.481918 #28892] INFO -- : SRWT Node: uk01vort003 Request: check_auth\n",
    "E, [2016-05-21T17:44:02.481959 #28892] ERROR -- : Unable to connect to node uk01vort003, no token available\n",
    "I, [2016-05-21T17:44:02.733483 #28892] INFO -- : Running: /usr/sbin/pcs status nodes corosync\n",
    "I, [2016-05-21T17:44:02.733897 #28892] INFO -- : CIB USER: hacluster, groups: \n",
    "D, [2016-05-21T17:44:02.910480 #28892] DEBUG -- : []\n",
    "D, [2016-05-21T17:44:02.910700 #28892] DEBUG -- : Duration: 0.176567846s\n",
    "I, [2016-05-21T17:44:02.910799 #28892] INFO -- : Return Value: 1\n",
    "W, [2016-05-21T17:44:02.911164 #28892] WARN -- : Cannot read config 'tokens' from '/var/lib/pcsd/tokens': No such file or directory - /var/lib/pcsd/tokens\n",
    "E, [2016-05-21T17:44:02.911296 #28892] ERROR -- : Unable to parse tokens file: A JSON text must at least contain two octets!\n",
    "I, [2016-05-21T17:44:02.912068 #28892] INFO -- : Saved config 'tokens' version 1 a71824b42061fbb2a08f42069a4285ddbb8f8040 to '/var/lib/pcsd/tokens'\n"
  ]
}

--Debug Output End--

uk01vort003: Authorized
[root@uk01vort003 pcsd]#

Whilst investigating, I came across an issue with Ruby not using IPv4 connections, so I have tried to force IPv4 by editing /usr/share/pcsd/ssl.rb but even when I force it to use IPv4 for port 2224, it still doesn't work.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1584365/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[ndsipa pomu (ndsipa-pomu)]

Okay, looks like this isn't necessarily just an Ubuntu bug. I've since had the same problems with running "pcs cluster auth" on RHEL7. It looks like my test (see above) was using 127.0.0.1 as the ip address which does work on both Ubuntu and RHEL7.

After lots of investigation, it looks like having "http_proxy" set will prevent "pcs cluster auth" from working.

[ndsipa pomu (ndsipa-pomu)]

I got this working after working around two other bugs.

"systemctl pcsd start" fails with ruby errors (/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `required`) that can be fixed by running:

gem install orderedhash

After that, there's the problem with Ruby binding to the IPv6 interface, so I edited /usr/share/pcsd/ssl.rb and changed the webrick_options:

:BindAddress => '*',
:Host => '',

After that, "systemctl restart pcsd" and then "pcs cluster auth" works.

[ndsipa pomu (ndsipa-pomu)]

Dagnammit - "pcs cluster auth" doesn't write out /var/lib/pcsd/tokens so anything I try to do with it afterwards just fails.

So, although "pcs cluster auth" returns "authorized", it doesn't actually do its job.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Valentin Vidic (vvidic)]

This could be related to Bug#1580045 pcsd does only bind to IPv6, now patched in Debian package 0.9.151-1 so you can test this.

[ndsipa pomu (ndsipa-pomu)]

Unfortunately, I couldn't get a pacemaker cluster to work on Ubuntu (I instead used ocfs to provide a clustered filesystem) so I'm no longer in a position to test this.

However, when I was tangling with it, I managed to get pcsd to bind to IPv4 but there's still the problem with it not writing out the details into /var/lib/pcsd/tokens (which did work on RHEL7).

[Mariusz B (yota-kun)]

I tested pcs 0.9.151-1 (amd64 binary from Ubuntu yakkety) on Ubuntu 16.04.
pcsd binds to IPv4 now and doesn't require orderedhash gem, but authentication still doesn't seem to work.

root@u16a:~# pcs cluster auth u16a u16b -u hacluster -p password ; echo $?
u16a: Authorized
u16b: Authorized
0

root@u16a:~# pcs cluster setup --name mycluster u16a u16b ; echo $?
Error: u16a: error checking node availability: Unable to authenticate to u16a - (HTTP error: 401), try running 'pcs cluster auth'
Error: u16b: error checking node availability: Unable to authenticate to u16b - (HTTP error: 401), try running 'pcs cluster auth'
Error: nodes availability check failed, use --force to override. WARNING: This will destroy existing cluster on the nodes.
1

[ndsipa pomu (ndsipa-pomu)]

@Mariusz - it sounds like /var/lib/pcsd/tokens still isn't getting written out by running "pcs cluster auth". Can you confirm that?

[Wojciech Giel (wkg21)]

Hello,
I can confirm that after changing ssl.rb when i authenticate nothing gets written to tokens.

I was able to workaround problem doing following steps:

1. configured corosync on each node in cluster (got 3 nodes):
totem {
    version: 2
    secauth: off
    cluster_name: openstack
    transport: udpu
}
nodelist {
    node {
        ring0_addr: 10.12.1.100
        nodeid: 1
    }

    node {
        ring0_addr: 10.12.1.101
        nodeid: 2
    }

    node {
        ring0_addr: 10.12.1.102
        nodeid: 3
    }
}
quorum {
    provider: corosync_votequorum
}
logging {
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
}

2. edited /usr/share/pcsd/ssl.rb and changed to:
  :BindAddress => '0.0.0.0',
  :Host => '0.0.0.0',

3. provided empty config for:
/var/lib/pcsd/pcs_settings.conf
{
  "format_version": 2,
  "data_version": 0,
  "clusters": [

  ],
  "permissions": {
    "local_cluster": [

    ]
  }
}

and

/var/lib/pcsd/tokens
{
  "format_version": 2,
  "data_version": 0,
  "tokens": {
  }
}

ln -s /var/lib/pcsd/ /usr/lib/
ln -s /var/log/corosync/ /var/log/cluster

4. authenticated with pcs auth ...
5. created cluster with pcs cluster setup ...
I had to use ip addresses not servers name.

so far looks like it is working fine.

[Mariusz B (yota-kun)]

Yes, I confirm. There's no "tokens" file.
Only these:

root@u16a:~# ls /var/lib/pcsd/
pcsd.cookiesecret pcsd.crt pcsd.key pcs_users.conf

[Mariusz B (yota-kun)]

Update on running pcs 0.9.151-1 on Ubuntu 16.04.

1. pacemaker and especially corosync must be stopped:

systemctl stop pacemaker corosync

2. Remove (or rename) /etc/corosync/corosync.conf
Without that "pcs cluster auth" won't create "tokens" file.

3. Modify /usr/lib/python2.7/dist-packages/pcs/cluster.py to avoid searching in /var/lib/lxcfs directory (mountpoint for fuse.lxcfs)

root@u16a:~# diff cluster.py.org cluster.py
1628c1628
< os.system("find /var/lib -name '"+name+"' -exec rm -f \{\} \;")
---
> os.system("find /var/lib -path '/var/lib/lxcfs' -prune -o -name '"+name+"' -exec rm -f \{\} \;")

Setting starting-point to "/var/lib/pacemaker/cib" is probably better solution.
Without that "pcs cluster setup" will hang.
find command will get stuck on all nodes i.e.

 7071 ? Ssl 0:02 /usr/bin/ruby -C/var/lib/pcsd -I/usr/share/pcsd -- /usr/share/pcsd/ssl.rb & > /dev/null &
 7782 ? S 0:00 \_ /usr/bin/python /usr/sbin/pcs cluster destroy
 7945 ? S 0:00 \_ sh -c find /var/lib -name 'cib-*' -exec rm -f \{\} \;
 7946 ? S 0:00 \_ find /var/lib -name cib-* -exec rm -f {} ;

root@u16a:~# strace -p `pgrep find` -s256
strace: Process 7946 attached
write(2, "'/var/lib/lxcfs/cgroup/devices/system.slice/cloud-init-local.service/devices.deny'", 82

root@u16a:~# find /var/lib -name 'cib-*' |& grep 'Permission denied' | wc -l
462

4. After all those steps pcs cluster auth, setup, start etc. should work fine.

[Domingos Novo (domingosnovo)]

Mariusz' comments are spot on to fix this issue. Actually this bug should be splitted in two parts:

- Fix token/auth generation for "pcs cluster auth" (/var/lib/pcsd/tokens is not updated unless a set of prerequisites are met)

- Cluster configuration hangs during cleanup (e.g., "pcs cluster setup --name <cluster_name> --force <node1> <node2>")

[Harry Coin (hcoin)]

It's actually a package breaker. A fresh install:
apt-get install pcs fence-agents ldirectord
fails. The file mentioned above that needs removing corosync.conf, prevents apt from a successful install. Touching that to make it empty allows the install to complete. Which, apparently, breaks the token creator which breaks all the auth.
Also, before any of it will actually run,
gem install orderedhash
must complete or pcsd won't launch.
And then, to get both ipv6 and ipv4 both bind and acceptance of hosts, /usr/share/ssl.rb needs
  :BindAddress => '*',
  :Host => '*',
  :SSLEnable => true,

Initially the * was :: which didn't connect on ipv4, preventing some internal comm from happening much less joining clusters.

So, basically, "Just Works" is not happening here.

[Rafael David Tinoco (rafaeldtinoco)]

I'm working on:

https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1580035
https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1580045
https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1640923
https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1640919

And marking this as a duplicate of:

https://bugs.launchpad.net/ubuntu/+source/pcs/+bug/1580045

Will be releasing a fix proposal soon.

[ndsipa pomu (ndsipa-pomu)]

This bug uncovers more issues than just the IP binding issue (which is easy to work-around). There's also the issue of /var/lib/pcsd/tokens not being written into and the way that pcs cluster setup hangs.

[ndsipa pomu (ndsipa-pomu)]

I've just looked at the other bugs that Rafael is working on and they cover the other issues in this bug.