/var/run/squid (run_dir) no longer created on start up

[Haw Loeung]

[Impact]
Squid in SMP mode, when "workers <N>" is set in squid.conf, uses /var/run/squid for several ipc sockets. This directory is not created by the packaging/service and as a result squid doesn't work in this situation.

[Test Case]
* install squid
sudo apt update && sudo apt install squid -y

* verify there is no /var/run/squid directory
$ sudo ls -la /var/run/squid
ls: cannot access '/var/run/squid': No such file or directory

* Due to bug #1815852, squid won't work correctly even after that directory is created, so let's consider the above enough to confirm this bug here.

With the updated packages, which also fix #1815852, the directory exists, and, due to the restart done by the upgrade, is populated already:
$ sudo ls -la /var/run/squid
total 0
drwxr-xr-x 2 proxy proxy 160 Feb 28 18:35 .
drwxr-xr-x 22 root root 820 Feb 28 18:35 ..
srwxr-x--- 1 proxy proxy 0 Feb 28 18:35 squid-coordinator.ipc
srwxr-x--- 1 proxy proxy 0 Feb 28 18:35 squid-kid-1.ipc
srwxr-x--- 1 proxy proxy 0 Feb 28 18:35 squid-kid-2.ipc
srwxr-x--- 1 proxy proxy 0 Feb 28 18:35 squid-kid-3.ipc
srwxr-x--- 1 proxy proxy 0 Feb 28 18:35 squid-kid-4.ipc
srwxr-x--- 1 proxy proxy 0 Feb 28 18:35 squid-kid-5.ipc

* and the proxy works:
$ http_proxy=http://localhost:3128/ wget http://www.ubuntu.com -O /dev/null
--2019-02-28 18:35:59-- http://www.ubuntu.com/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:3128... connected.
Proxy request sent, awaiting response... 302 Found
Location: https://www.ubuntu.com/ [following]
--2019-02-28 18:36:00-- https://www.ubuntu.com/
Resolving www.ubuntu.com (www.ubuntu.com)... 91.189.89.115
Connecting to www.ubuntu.com (www.ubuntu.com)|91.189.89.115|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/dev/null’

/dev/null [ <=> ] 129.75K 148KB/s in 0.9s

2019-02-28 18:36:02 (148 KB/s) - ‘/dev/null’ saved [132866]

$ echo $?
0

[Regression Potential]
The fix tells systemd to create the runtime directory /run/squid (to which /var/run is a symlink) via the tmpfiles.d mechanism, a solution used by many other services already as can be seen in "ls -la /usr/lib/tmpfiles.d". This only works for systemd systems, which is the case of cosmic and disco.
The fix is already in disco, but going through migration at the moment.

[Other Info]
Not at this time.

[Original Description]
Hi,

On squid service start up in Bionic, /var/run/squid is created by /etc/init.d/squid (create_run_dir()). This doesn't happen in Cosmic nor Disco and probably is related to the migration to systemd.

Squid needs to create the squid-kid-*.ipc UNIX sockets in /var/run/squid to function on SMP set ups (workers X).

Steps to reproduce:

  * spin up fresh cosmic or bionic instance/machine

  * install squid

  * configure 'workers 5' or enable the Rock Store cache storage type

  * reboot instance

  * try start up squid.

Logging shows this:

| 2019/02/15 04:46:58 kid1| commBind Cannot bind socket FD 28 to [::]: (2) No such file or directory

[Sebastien Bacher (seb128)]

Thank you for your bug report. THe package is in sync with Debian, would be best to report that issue to them (https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=squid)

[Christian Ehrhardt  (paelzer)]

FYI - other workers related bug 1815852 - that one is fixed since Disco and needs an SRU. Those two probably should be handled together.

[Andreas Hasenack (ahasenack)]

Thanks, confirmed.

[Andreas Hasenack (ahasenack)]

To help gauge the severity of this bug, could you tell us what doesn't work? Or is this just about the error message? This is probably some coordination between the multiple workers, I'm guessing.

[Haw Loeung (hloeung)]

Without the ability to create the UNIX sockets for IPC, squid doesn't even listen and handle requests.

[Amos Jeffries (yadi)]

Luigi has added a change in Debian 4.5-2 package that should resolve this.

[Robie Basak (racb)]

Thanks Amos!

I was curious to look, so here's the commit: https://salsa.debian.org/squid-team/squid/commit/cc917ba0780415d0334092b3e36701ead19a3094

Leaving this to Andreas as he assigned himself already.

[Haw Loeung (hloeung)]

Thanks Amos, Andreas, and Luigi!

[Andreas Hasenack (ahasenack)]

It's going through migration currently, might be a while because there is a glibc migrating:

http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#squid

[Andreas Hasenack (ahasenack)]

https://launchpad.net/~ahasenack/+archive/ubuntu/squid-smp-1815852/+packages has test packages for cosmic with both fixes (#1816006 and #1815852).

[Christian Ehrhardt  (paelzer)]

Reviewed the MP, LGTM

[Timo Aaltonen (tjaalton)]

Hello Haw, or anyone else affected,

Accepted squid into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/squid/4.1-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package squid - 4.4-1ubuntu2

---------------
squid (4.4-1ubuntu2) disco; urgency=medium

  * d/squid.tmpfile: add tmpfiles configuration to handle /var/run/squid
    at boot. Thanks to Luigi Gangitano <email address hidden> (LP: #1816006)

 -- Andreas Hasenack <email address hidden> Wed, 27 Feb 2019 08:54:45 -0300

[Haw Loeung (hloeung)]

Confirmed fixed in cosmic:

 * spun up new fresh cosmic instance

 * installed package from -proposed

 * checked existence of /run/squid

 * rebooted

 * checked existence of /run/squid

 * checked existence of /usr/lib/tmpfiles.d/squid.conf

[Łukasz Zemczak (sil2100)]

@hloeung Thanks for the verification! For the future: please also include the version number of the packages you have tested. This is required since otherwise someone might have tested some PPA-built packages instead.

[Launchpad Janitor (janitor)]

This bug was fixed in the package squid - 4.1-1ubuntu3.1

---------------
squid (4.1-1ubuntu3.1) cosmic; urgency=medium

  * d/p/smp-startup-error.patch: fix startup error in SMP mode, when workers
    is set. (LP: #1815852)
  * d/squid.tmpfile: add tmpfiles configuration to handle /var/run/squid
    at boot. Thanks to Luigi Gangitano <email address hidden> (LP: #1816006)

 -- Andreas Hasenack <email address hidden> Thu, 28 Feb 2019 13:27:11 -0300

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for squid has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.