ctdb service crashes on start

[Eric G]

System information:

root@san1:/var/log/ctdb# lsb_release -rd
Description: Ubuntu 14.04 LTS
Release: 14.04

root@san1:/var/log/ctdb# apt-cache policy ctdb
ctdb:
  Installed: 2.5.1+debian0-1
  Candidate: 2.5.1+debian0-1
  Version table:
 *** 2.5.1+debian0-1 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
        100 /var/lib/dpkg/status

I expect the service not to crash with a basic starting configuration. Directory /san/ctdb exists and is mounted as glusterfs volume. I have installed ctdb and use the following config files:

/etc/default/ctdb:
CTDB_RECOVERY_LOCK=/san/ctdb/lock
CTDB_NODES=/etc/ctdb/nodes
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_DEBUGLEVEL=NOTICE

/etc/ctdb/nodes:
10.0.2.21
10.0.2.22

/etc/ctdb/public_addresses:
10.0.2.31/24 eth0
10.0.2.32/24 eth0

I get the following error in /var/log/ctdb/log.ctdb:
2014/06/28 21:48:16.890594 [16056]: CTDB starting on node
2014/06/28 21:48:17.607857 [16056]: Unable to bind on ctdb socket '/var/lib/run/ctdb/ctdbd.socket'
2014/06/28 21:48:17.607950 [16056]: Cannot continue. Exiting!

This is very odd because the CTDB documentation states that in version 2.5 this file was moved to /var/run/ctdb/ctdbd.socket by default. See http://wiki.samba.org/index.php/CTDB2releaseNotes

When I add CTDB_SOCKET=/var/run/ctdb/ctdbd.socket to /etc/default/ctdb it gets a little farther and creates this file as documented by CTDB 2.5, but I still seem to have a path issue with another socket related file.

The error log shows:
2014/06/28 22:19:38.166127 [16713]: CTDB starting on node
2014/06/28 22:19:38.170044 [16714]: Starting CTDBD (Version 2.5.1) as PID: 16714
2014/06/28 22:19:38.170236 [16714]: Created PID file /var/run/ctdb/ctdbd.pid
2014/06/28 22:19:38.170298 [16714]: Set scheduler to SCHED_FIFO
2014/06/28 22:19:38.170614 [16714]: Set runstate to INIT (1)
2014/06/28 22:19:38.192313 [16714]: 00.ctdb: awk: line 2: function gensub never defined
2014/06/28 22:19:38.197628 [16714]: 00.ctdb: awk: line 2: function gensub never defined
2014/06/28 22:19:38.351683 [16714]: Unable to open /var/lib/run/ctdb/.socket_lock
2014/06/28 22:19:38.351859 [16714]: tcp/tcp_init.c:67 Failed to start listening on the CTDB socket
2014/06/28 22:19:38.351911 [16714]: CTDB daemon shutting down
2014/06/28 22:19:38.352190 [16714]: Removed PID file /var/run/ctdb/ctdbd.pid

I cannot find either 1) where the default /var/run/ctdb path might be modified from upstream or 2) where a user could set a path to correct the last error message.

[mikhail (kazan417)]

you need just run in terminal two command to solve this:
mkdir /var/lib/run
mkdir /var/lib/run/ctdb
mail me to <email address hidden> if it help to you.

[Eric G (erickg)]

Yes that did it! I simplified to:

mkdir -p /var/lib/run/ctdb

Looking forward to the bugfix for this.

On 7/27/2014 11:30 PM, mikhail wrote:
> you need just run in terminal two command to solve this:
> mkdir /var/lib/run
> mkdir /var/lib/run/ctdb
> mail me to <email address hidden> if it help to you.
>

[Rafael David Tinoco (rafaeldtinoco)]

Doing the CTDB setup I faced the same issues as:

https://bugs.launchpad.net/ubuntu/+source/ctdb/+bug/1821775
https://bugs.launchpad.net/ubuntu/+source/ctdb/+bug/1828799
https://bugs.launchpad.net/ubuntu/+source/samba/+bug/722201

I'll merge the fixes (for those and this bug) altogether.