clamav-daemon doesn't start after installation

[Eero]

[Impact]

 * Installing clamav the user expects things start to run
 * Naturally the daemon can only run after the virus-database is updated
  (conditions in systemd service)
 * But the freshclam service to update the virus-db is not starting either
   since the move to systemd (Regression to old behavior)

[Test Case]

 * (enable proposed if needed)
 * apt-get install clamav-freshclam
 * systemctl status clamav-freshclam.service
 * this should show "active" (fixed) and not just "loaded" (broken)

[Regression Potential]

 * The fix is fairly straightforward so there shouldn't be a lot. But it
   affects postinst and while not changing, it has to move some code there
   and might affect something in postinst I missed to think about.

 * It worked for adt in Y & X and for my manual tests in LXD containers
   fine

[Other Info]

 * This is (intentionally) only addressing the regression for now.
   Once fixed we think of splitting the part of auto-starting clamav-
   daemon later on once the DB is ready

Steps to reproduce:

1. Fresh 16.04 Server installation
2. "sudo apt-get install clamav-daemon clamav-freshclam"
3. ClamAV doesn't start even though it should.

$ systemctl status clamav-daemon
● clamav-daemon.service - Clam AntiVirus userspace daemon
   Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
Condition: start condition failed at Thu 2016-06-09 11:58:19 EEST; 7min ago
     Docs: man:clamd(8)
           man:clamd.conf(5)
           http://www.clamav.net/lang/en/doc/

$ systemctl status clamav-freshclam.service
● clamav-freshclam.service - ClamAV virus database updater
   Loaded: loaded (/lib/systemd/system/clamav-freshclam.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:freshclam(1)
           man:freshclam.conf(5)
           http://www.clamav.net/lang/en/doc/

Also directory /var/run/clamav/ mentioned in /etc/clamav/clamd.conf seems to be missing.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: clamav-daemon 0.99+dfsg-1ubuntu1
ProcVersionSignature: Ubuntu 4.4.0-22.40-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Thu Jun 9 12:00:59 2016
EcryptfsInUse: Yes
InstallationDate: Installed on 2016-06-09 (0 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.3)
SourcePackage: clamav
UpgradeStatus: No upgrade log present (probably fresh install)

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Indeed on a quick inspection it looks like both services are not running on install. It looks like for clamav-daemon this is intentional as freshclam has not run yet. However I'd expect the freshclam service to start. After starting it manually, it failed in my environment because no proxy was configured and my lab's firewall requires one. I used "dpkg-reconfigure" to configure a proxy, and the script then started the freshclam daemon automatically. However after it successfully ran, it still did not start clamd, noting that it failed to signal clamd in the logs because it was not running.

I'm not sure what the intended behaviour is. I was unable to check Debian's behaviour quickly because of bug 1590747. This needs following up.

Workaround: if you start clamav-freshclam manually, allow it to finish, then start clamav-daemon manually, I think everything will work from there on as normal.

[Christian Ehrhardt  (paelzer)]

Hi,
here my initial analysis of the case.

## Check 0 - Debian bug status ##
No related Debian bug found

## Check 1 - Status on Xenial, Yakkety and Sid ##
Despite slight version differences all behave the same.

On all releases when starting freshclam some of the mirrors behind db.local.clamav.net failed, but after a while it found a working one.
That took up to 11 minutes in one case depending on mirror success and download speed - so nothing we can wait on on "install".
Also for the workaround mentioned that means, you might have to wait quite some time between starting freshclam and clamav-daemon

Since clamd was not yet running all reported a warning "Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such file or directory"

## Check #2 - Was this better in the past - is it a regression? ##
On Trusty (before systemd) instead of the "condition" prints
 Setting up clamav-daemon (0.98.7+dfsg-0ubuntu0.14.04.1) ...
  * Clamav signatures not found in /var/lib/clamav
  * Please retrieve them using freshclam
  * Then run '/etc/init.d/clamav-daemon start'
But freshclam is started automatically at least

On Jessie freshclam starts automatically
=> so that is a regression
But even after it finished it still runs into the "Can't connect to clamd through /var/run/clamav/clamd.ctl" as it blocked on start due to the condition check in the service.

## Perfect world scenario (=translates to fixes needed) ##
- Freshclam starts after install (all recent releases xenial, yakkety, sid broken in that regard)
- clamav-daemon picks up the condition is now valid and starts after freshclam updated the DB

## Fix 1 - get freshclam to autostart again ##
The clamav-freshclam.service file is byte identical in jessie (working) and sid (failing).
After discussing those with pitti he suggested that the postinst might be incomplete.
Some things are not done via dh_installinit or dh_systemd_start but instead manually.
Then eventually it calls invoke-rc.d before enabling the unit.
This should likely move after ##DEBHELPER##.

To make it clear - there was no postinst change between jessie (working) and sid (fails), but invoke-rc-d might have used to start disabled things.

## Fix 2 - could clamav-daemon start once synced? ##
Re-pickup condition in systemd not built in to Condition statements.
ExecStartPre= and waiting for update not an option as it is not allowed to take long.
Discussed with systemd IRC channel - and they suggested a systemd.path unit.

I'm not deep enough into systemd yet to predict if that conflicts with the .socket file which is created by the postinst.
After reading docs that would look like (in addition to /lib/systemd/system/clamav-daemon.service):
cat /lib/systemd/system/clamav-daemon.path
[Path]
PathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
PathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}

Also we could at least change the Condition to an Assert to avoid having it fail silently AND due to the path based activation should never happen again.

I'll create and test a fix for sid, suggest it to the debian maintainer.
If successful we can pick it up on next merge and/or backport it as needed.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

There was no response by Debian yet, so I decided to carry a fix for #1 as delta for now to get rid of the regression.

THe patch is relatively straight forward which is good as we want to SRU into Xenial as well.
It passed all my build and adt tests, also it showed to fix the issue when tested in a lxd container.

I'm gonna attach a debdiff for yakkety and xenial.
I also prepare a SRU template right away, so when yakkety is fully accepted one can sponsor xenial as well without waiting for me (other than subscribing the SRU Team then).

[Christian Ehrhardt  (paelzer)]

I was too early with my statement on working adt as I was relying on that I've had it running fine last week. But usually prior to any upload I run these and today it has issues - so this might take a bit longer than expected :-/

[Christian Ehrhardt  (paelzer)]

For the sponsors:
1. please accept the xenial task I nominated
1. please review and upload the yakkety debdiff
2. if that went into yakkety without hickups please do the same for the xenial debdiff (I can check and ping here after the Y upload happened)

[Robie Basak (racb)]

Looks good, thanks! Uploaded to Yakkety. We can upload Xenial when you are ready.

[Launchpad Janitor (janitor)]

This bug was fixed in the package clamav - 0.99+dfsg-1ubuntu3

---------------
clamav (0.99+dfsg-1ubuntu3) yakkety; urgency=medium

  * Ensure freshclam service is run after package install (LP: #1590688)

 -- Christian Ehrhardt <email address hidden> Thu, 07 Jul 2016 09:22:41 +0200

[Robie Basak (racb)]

The Xenial SRU is in the SRU queue.

[Brian Murray (brian-murray)]

Hello Eero, or anyone else affected,

Accepted clamav into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/clamav/0.99+dfsg-1ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Eero (eero+launchpad)]

Freshclam starts with 0.99+dfsg-1ubuntu1.2, but clamav-daemon isn't started automatically even after the databse files have been fetched. Clamav-daemon needs to be started manually.

[Eero (eero+launchpad)]

If somebody else also needs to deploy clamav packages automatically, you can add something like this at the end of your installation scripts:

nohup $(
    command -v systemctl || exit 0
    systemctl status clamav-freshclam || systemctl restart clamav-freshclam
    while [ ! -f /var/lib/clamav/daily.cvd ]
    do
        sleep 1
    done
    sleep 5
    systemctl status clamav-daemon || systemctl restart clamav-daemon
) >/dev/null 2>&1 &

[Christian Ehrhardt  (paelzer)]

Thanks Eero for the verification of the fix I added (starting freshclam again which was a regression.
You are right that clamav-daemon doesn't start for the reasons I outlined before of the condition being not met initially and not picked up later by systemd.

I'm not sure about a sleep-loop to fix this up but I really appreciate dropping the info for anyone who wants this. I'd expect a proper solution would be something where systemd is realizing that the condition is "now true" and starts it then.

Since that part is not a regression we should discuss and agree with Debian in how to do so.
I started the discussion before on the linked Debian bug, but regarding clamav-daemon not starting the response was "this is okay."
Would you mind opening a bug there or chiming in on the linked bug again to push your argument - I also already added several suggestions there how to start it, but they need some improvement for this "realizing later" that the condition is met.

[Launchpad Janitor (janitor)]

This bug was fixed in the package clamav - 0.99+dfsg-1ubuntu1.2

---------------
clamav (0.99+dfsg-1ubuntu1.2) xenial; urgency=medium

  * Ensure freshclam service is run after package install (LP: #1590688)

 -- Christian Ehrhardt <email address hidden> Thu, 07 Jul 2016 09:22:31 +0200

[Chris J Arges (arges)]

The verification of the Stable Release Update for clamav has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.