Init fails to start CouchDB after upgrade to Vivid

[Christian Lins]

After upgrading my system to Ubuntu Vivid the CouchDB process is no longer automatically started by the init-System (systemd).

In fact a manual start of the service via

$ sudo /etc/init.d/couchdb start
[ ok ] Starting couchdb (via systemctl): couchdb.service.

does not really start the CouchDB daemon. Perhaps it crashes immediately.

The log file /var/log/couchdb/couch.log is empty.

A manual start of CouchDB via

$ sudo couchdb

works fine.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: couchdb 1.6.0-0ubuntu5
ProcVersionSignature: Ubuntu 3.19.0-16.16-generic 3.19.3
Uname: Linux 3.19.0-16-generic x86_64
ApportVersion: 2.17.2-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Thu May 21 14:38:49 2015
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-11-10 (191 days ago)
InstallationMedia: Ubuntu 14.10 "Utopic Unicorn" - Release amd64 (20141022.1)
JournalErrors: Error: command ['journalctl', '-b', '--priority', 'warning'] failed with exit code 1: No journal files were found.
PackageArchitecture: all
SourcePackage: couchdb
UpgradeStatus: Upgraded to vivid on 2015-05-13 (7 days ago)
mtime.conffile..etc.couchdb.local.ini: 2015-05-21T14:26:40.614348

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dylan McCall (dylanmccall)]

I see that the package creates a couchdb system user, and it appears the init script is correctly running Couch as that user.

So, when I do `sudo -u couchdb couchdb`, I can see /var/run/couchdb is incorrectly owned by root:

    $ sudo -u couchdb couchdb
    Apache CouchDB 1.6.0 (LogLevel=info) is starting.
    Apache CouchDB has started. Time to relax.
    [info] [<0.32.0>] Apache CouchDB has started on http://0.0.0.0:5984/
    [error] [<0.32.0>] Failed to write to URI file /var/run/couchdb/couch.uri: permission denied
    {"init terminating in do_boot",{{badmatch,{error,{eacces,{couch_app,start,[normal,["/etc/couchdb/default.ini","/etc/couchdb/local.ini"]]}}}},[{couch,start,0,[{file,"couch.erl"},{line,18}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
    init terminating in do_boot ()

However, though I can run it manually with the correct user now, changing the file ownership hasn't helped me with the system service...

[Miguel Rubio-Roy (mrubioroy)]

Removing the inclusion of /lib/lsb/init-functions in /etc/init.d/couchdb solved the problem for me.

But make sure that everything is owned by couchdb:

chown -R couchdb /var/log/couchdb
chown -R couchdb /var/run/couchdb
chown -R couchdb /var/lib/couchdb
chown -R couchdb /etc/couchdb

[Tobias Lausch (tobias-lausch-q)]

Reowning was not the problem since /etc/init.d/couchdb does this by default if any user variable is set.
note that there is no default config file /etc/default/couchdb
in this case neither stdout nor user variables are given and couchdb assumes
stdout = /couchdb.log and
stderr = /couchdb.err

I added
COUCHDB_USER="couchdb"
COUCHDB_STDERR_FILE=/var/log/couchdb/couch.err
COUCHDB_STDOUT_FILE=/var/log/couchdb/couch.log
to /etc/init.d/couchdb which could also be placed in the default config file.

now it is working but that was though.

It works with both /lib/lsb/init-functions included or excluded so i stayed with the inclusion.

[Christian Lins (cli)]

The hint of Tobias worked for me, but I had to reown the database files in /var/lib/couchdb to "couchdb" that I have created running CouchDB as root.

[Jason Gerard DeRose (jderose)]

I didn't have a chance to keep up with the systemd transition for CouchDB in Vivid, but I'm working on getting this fixed properly for Wily.

I've long had subtle problems with the upstream SysV init script (especially when it comes to getting `apt-get purge couchdb` to work correctly), and this was the biggest reason why I switched to an Upstart job in 14.04.

So I think the correct transition is to add a systemd couchdb.service unit, which I'm working on now. That way you can properly express an init dependency on couchdb, same as you can with Upstart (this is something I personally need for how we're using CouchDB internally at System76).

Not sure if this fix will be SRU-able for Vivid, but my goal is definitely to have things in good shape for Wily (and V, of course).

Sorry for the trouble folks are having with this! I dropped the ball during Vivid, my bad :(

[Jason Gerard DeRose (jderose)]

Er, I meant "and X, of course"... obviously we want couchdb in good shape for the next LTS :)

[Jason Gerard DeRose (jderose)]

Making decent progress, this is my current debian/couchdb.service:

[Unit]
Description=the system-wide CouchDB instance
After=network-online.target

[Service]
Type=simple
ExecStartPre=/bin/mkdir -p /run/couchdb
ExecStartPre=/bin/chown couchdb:couchdb /run/couchdb
ExecStart=/bin/su couchdb -c /usr/bin/couchdb
ExecStopPost=/bin/rm -rf /run/couchdb
Restart=on-failure

[Install]
WantedBy=multi-user.target

If anyone has any feedback, please let me know. Also feedback is welcome on whether starting CouchDB via systemd is the right path forward. From my past experience with the upstream SysV init script, I'm all for using systemd, but others might have use cases I'm not considering.

[Jason Gerard DeRose (jderose)]

Martin, do you have any thoughts on this? I got the impression you added the upstream SysV init script back as a quick stop-gap... but do you see any reason we shouldn't start CouchDB via debian/couchdb.service instead?

[Martin Pitt (pitti)]

Of course, if you have a proper systemd unit that's always better. But some suggestions:

> [Unit]
> Description=the system-wide CouchDB instance

Please drop the "the" and start with a capital letter for some consistency.

> After=network-online.target

Is that really necessary? I. e. couchdb doesn't get along with new network
interfaces coming up after boot? Note that this is mostly a workaround, and
won't do anything if you e. g. hotplug/enable/connect to a new network during
runtime.

If you need it, you also need a corresponding Wants=network-online.target

> [Service]
> Type=simple

That's the default, you can omit this.

> ExecStartPre=/bin/mkdir -p /run/couchdb
> ExecStartPre=/bin/chown couchdb:couchdb /run/couchdb

Replace this with RuntimeDirectory=couchdb (see man systemd.exec)

> ExecStart=/bin/su couchdb -c /usr/bin/couchdb

Don't use su, that'll go through the PAM stack etc. Use this instead:

User=couchdb
ExecStart=/usr/bin/couchdb

> ExecStopPost=/bin/rm -rf /run/couchdb

Handled automatically by RuntimeDirectory, so drop.

> Restart=on-failure
>
> [Install]
> WantedBy=multi-user.target

So, all in all:

[Unit]
Description=System-wide CouchDB instance
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=couchdb
User=couchdb
ExecStart=/usr/bin/couchdb
Restart=on-failure

[Install]
WantedBy=multi-user.target

[Martin Pitt (pitti)]

Sticking this into /lib/systemd/system/couchdb.service and "sudo systemctl enable couchdb" seems to work:

$ systemctl status -l couchdb
● couchdb.service - System-wide CouchDB instance
   Loaded: loaded (/lib/systemd/system/couchdb.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2015-07-08 07:01:30 CEST; 33s ago
 Main PID: 619 (beam)
   CGroup: /system.slice/couchdb.service
           ├─619 /usr/lib/erlang/erts-6.2/bin/beam -Bd -K true -A 4 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/couchdb -- -noshell -noinput -os_mon start_memsup false start_cpu_sup false disk_space_check_interval 1 disk_almost_full_threshold 1 -sasl errlog_type error -couch_ini /etc/couchdb/default.ini /etc/couchdb/local.ini -s couch
           └─639 sh -s disksup

Jul 08 07:01:30 autopkgtest systemd[1]: Starting System-wide CouchDB instance...
Jul 08 07:01:30 autopkgtest couchdb[619]: {error_logger,{{2015,7,8},{7,1,30}},std_error,"File operation error: eacces. Target: /lost+found/ebin. Function: read_file_info. Process: code_server."}
Jul 08 07:01:30 autopkgtest couchdb[619]: {error_logger,{{2015,7,8},{7,1,30}},std_error,"File operation error: eacces. Target: /root/ebin. Function: read_file_info. Process: code_server."}
Jul 08 07:01:30 autopkgtest couchdb[619]: =ERROR REPORT==== 8-Jul-2015::09:01:30 ===
Jul 08 07:01:30 autopkgtest couchdb[619]: File operation error: eacces. Target: /lost+found/ebin. Function: read_file_info. Process: code_server.
Jul 08 07:01:30 autopkgtest couchdb[619]: =ERROR REPORT==== 8-Jul-2015::09:01:30 ===
Jul 08 07:01:30 autopkgtest couchdb[619]: File operation error: eacces. Target: /root/ebin. Function: read_file_info. Process: code_server.
Jul 08 07:01:30 autopkgtest couchdb[619]: Apache CouchDB 1.6.0 (LogLevel=info) is starting.
Jul 08 07:01:30 autopkgtest couchdb[619]: Apache CouchDB has started. Time to relax.
Jul 08 07:01:30 autopkgtest couchdb[619]: [info] [<0.32.0>] Apache CouchDB has started on http://127.0.0.1:5984/

[Jason Gerard DeRose (jderose)]

Martin,

Thanks for your recommendations! Much cleaner now.

So do you think I just totally drop Wants=/After=network-online.target?

My rational for this is if (for whatever reason) someone wanted to bind CouchDB to a specific IP, say on a server with multiple NICs for internal and external traffic. If these IPs haven't been assigned yet, CouchDB would fail to start. Not something I personally use, but seems more correct in my mind, although bear in mind I'm not very systemd savvy yet.

I think if people are configuring CouchDB to accept external connections, most likely they'll bind to "0.0.0.0" or "::", in which case Wants=/After=network-online.target isn't needed. But I didn't want to break anyones use case here.

What do you think? What's the systemd best-practice in this case?

[Martin Pitt (pitti)]

It's ok to have Wants=/After=network-online.target, just that couchdb might start very late in the boot. In some cases, like you *only* use NetworkManager and aren't online, it will *never* happen though, and couchdb won't run at all. That's presumably not a big use case for servers with remotely accessible couchdb instances, but it's something to be aware of.

> What's the systemd best-practice in this case?

There is none, really. Wants=/After=network-online.target is a workaround for "old-school" software which doesn't handle dynamic network changes (i. e. the pre-hotplug world mindset). I. e. ideally that gets fixed in couchdb itself; as an intermediate workaround couchdb could reload itself if the network changes (similar to SSH, check /etc/network/if-up.d/openssh-server). But until that you can just add that to the unit.

[Jason Gerard DeRose (jderose)]

Martin, thanks for explaining Wants=/After=network-online.target.

I couldn't seem to get couchdb to *not* start based on what I was doing with network-manager. I tried unchecking "Enable Networking" in the indicator and rebooting, couchdb still started. I also tried rebooting when networking was enabled but no WiFi connections we configured and Ethernet was unplugged... but couchdb still started.

So Wants=/After=network-online.target at least seams pretty harmless, even if not strictly needed.

I have a merge proposal ready now:
https://code.launchpad.net/~jderose/ubuntu/wily/couchdb/fix-1457464/+merge/264199

[Martin Pitt (pitti)]

Replied on the MP. After this lands, please forward the debdiff to the Debian bug as well. Thanks!

[Jason Gerard DeRose (jderose)]

Now that things have been sorted out for Wily (pending sponsorship still), I'd like to get this fixed in Vivid also, assuming this is an SRU-able change.

Here's my merge proposal for Vivid:
https://code.launchpad.net/~jderose/ubuntu/vivid/couchdb/fix-1457464/+merge/264437

[Martin Pitt (pitti)]

Sponsored vivid update to SRU review queue.

[Launchpad Janitor (janitor)]

This bug was fixed in the package couchdb - 1.6.0-0ubuntu6

---------------
couchdb (1.6.0-0ubuntu6) wily; urgency=medium

  * Add debian/couchdb.service to start system-wide CouchDB instance via
    systemd (LP: #1457464)
  * debian/rules: call dh --with systemd
  * debian/control: Add `dh-systemd` to Build-Depends
  * Add debian/patches/support-erlang-18.diff that patches configure.ac to
    enable building against Erlang 18

 -- Jason Gerard DeRose <email address hidden> Fri, 10 Jul 2015 09:18:51 -0600

[Chris J Arges (arges)]

Hello Christian, or anyone else affected,

Accepted couchdb into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/couchdb/1.6.0-0ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Jason Gerard DeRose (jderose)]

Everything checks out for me. Testing I did:

1) Installed couchdb 1.6.0-0ubuntu5, enabled proposed, made sure there were no issue upgrading to 1.6.0-0ubuntu5.1 and that couchdb was actually running after the upgrade

2) Made sure `apt-get remove/purge couchdb` works without issue, stops couchdb properly

3) Tested a "fresh" install with proposed enabled, made sure couchdb was actually running after installation and that it starts automatically at boot

4) Tested `systemctl start/stop/restart/status couchdb`

5) Ran my normal unit tests against it for extra safety (note they really just test couchdb-bin and in theory shouldn't be effected by this change... theory matches up with reality).

So I'm removing verification-needed, adding verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package couchdb - 1.6.0-0ubuntu5.1

---------------
couchdb (1.6.0-0ubuntu5.1) vivid; urgency=medium

  * Add debian/couchdb.service to start system-wide CouchDB instance via
    systemd (LP: #1457464)
  * debian/rules: call dh --with systemd
  * debian/control: Add `dh-systemd` to Build-Depends

 -- Jason Gerard DeRose <email address hidden> Fri, 10 Jul 2015 10:33:10 -0600

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for couchdb has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.