tomcat cannot create the index directory for solr on startup

[Hasse Hagen Johansen]

After upgrading to 19.04 tomcat cannot create solr's index directory /var/lib/solr/data/index as seen by the catalina log like this:

18-May-2019 14:06:54.627 WARNING [main] org.apache.solr.core.SolrCore.initIndex [] Solr index directory '/var/lib/solr/data/index' doesn't exist. Creating new index...
18-May-2019 14:06:54.679 SEVERE [main] org.apache.solr.common.SolrException.log java.lang.RuntimeException: java.io.IOException: Cannot create directory: /var/lib/solr/data/index
 at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:403)
 at org.apache.solr.core.SolrCore.<init>(SolrCore.java:552)
 at org.apache.solr.core.CoreContainer.create(CoreContainer.java:480)
 at org.apache.solr.core.CoreContainer.load(CoreContainer.java:332)
 at org.apache.solr.core.CoreContainer.load(CoreContainer.java:216)
 at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:161)
 at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:270)
 at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:251)
 at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:102)
 at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4516)
 at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5162)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
 at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:713)
 at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
 at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695)
 at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:631)
 at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1832)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
 at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
 at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:526)
 at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:425)
 at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1577)
 at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
 at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
 at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
 at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367)
 at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:929)
 at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1377)
 at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1367)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
 at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
 at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:902)
 at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
 at org.apache.catalina.core.StandardService.startInternal(StandardService.java:423)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
 at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:928)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
 at org.apache.catalina.startup.Catalina.start(Catalina.java:638)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
 at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350)
 at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:496)
Caused by: java.io.IOException: Cannot create directory: /var/lib/solr/data/index
 at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:172)
 at org.apache.lucene.store.Lock.obtain(Lock.java:72)
 at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)
 at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:84)
 at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:398)
 ... 50 more

I have also reproduced this on a "clean" new installation in a VM

To reproduce:

sudo apt-get install solr-tomcat

and look in /var/log/tomcat9/catalina.*.log

Permissions seems ok like:

root@ubuntu-disco:/var/lib/solr/data# ls -ld /var/lib/solr/data
drwxrwx--- 2 tomcat tomcat 4096 Mar 2 22:02 /var/lib/solr/data

I also checked the /etc/solr/tomcat.policy file which indeed includes:

permission java.io.FilePermission "/var/lib/solr", "read,write,delete";
  permission java.io.FilePermission "/var/lib/solr/-", "read,write,delete";

(I guess - means recursive, but not entirely sure)

So this should also be ok and I don't see it being protected by apparmor

So I don't really know why i cannot create that directory

root@ubuntu-disco:/var/lib/solr/data# lsb_release -rd
Description: Ubuntu 19.04
Release: 19.04

root@ubuntu-disco:/var/lib/solr/data# apt-cache policy solr-tomcat
solr-tomcat:
  Installed: 3.6.2+dfsg-18
  Candidate: 3.6.2+dfsg-18
  Version table:
 *** 3.6.2+dfsg-18 500
        500 http://archive.ubuntu.com/ubuntu disco/universe amd64 Packages
        100 /var/lib/dpkg/status
root@ubuntu-disco:/var/lib/solr/data# apt-cache policy tomcat9
tomcat9:
  Installed: 9.0.16-3
  Candidate: 9.0.16-3
  Version table:
 *** 9.0.16-3 500
        500 http://archive.ubuntu.com/ubuntu disco/universe amd64 Packages
        100 /var/lib/dpkg/status
root@ubuntu-disco:/var/lib/solr/data# apt-cache policy solr-common
solr-common:
  Installed: 3.6.2+dfsg-18
  Candidate: 3.6.2+dfsg-18
  Version table:
 *** 3.6.2+dfsg-18 500
        500 http://archive.ubuntu.com/ubuntu disco/universe amd64 Packages
        100 /var/lib/dpkg/status

[Hasse Hagen Johansen (hasse-launchpad)]

I have found out that it has something to do with ProtectSystem=strict in the tomcat9 systemd unit. I will debug some more. setting ProtectSystem=full worked one time(and then something else went wrong and tomcat never got properly started)

[Hasse Hagen Johansen (hasse-launchpad)]

I found the error the systemd override snippet the solr-tomcat package provides is placed in a wrongly named directory /etc/systemd/system/tomcat9.d it should be in /etc/systemd/system/tomcat9.service.d for systemd to pick it up

so

/etc/systemd/system/tomcat9.d/solr-permissions.conf should be

/etc/systemd/system/tomcat9.service.d/solr-permissions.conf

I have provided a patch to the source package(hope it is the right way or please tell me)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "patch tomcat-solr.install" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mohonri (teancum98)]

I'm running into this exact issue, although the solution in post #2 does not resolve the issue for me.

[Hasse Hagen Johansen (hasse-launchpad)]

Mohonri:

I have tested it. For it to work you have to tell systemd to reload its units so to fix i for now doing this will solve it:

sudo systemctl stop tomcat9
sudo mv /etc/systemd/system/tomcat9.d /etc/systemd/system/tomcat9.service.d
sudo systemctl daemon-reload
sudo systemctl start tomcat9

[Mohonri (teancum98)]

Thanks for the follow up. Your instructions seem to have worked perfectly!

[Julian Lam (julian-lam)]

Yes, this patch works for me as well... or rather, simply renaming the /etc/system/systemd/tomcat.d directory.

Ubuntu Bionic 18.04.