assertion failed: store_client.cc:92: "mem"

[Thomas]

[Impact]

 * squid had a bug that showed up in some setup triggering
     store_client.cc:92: "mem" assertion
   Details are described in https://bugs.squid-cache.org/show_bug.cgi?id=4616

 * This is backporting the upstream fix to squid for Bionic which is
   rather close to the codebase the upstream fix happened.

[Test Case]

 * TODO - waiting on feedback by the reporters
 * Maybe we have to fall back to have (?multiple?) of the affected
   reporters with complex squid setups affected by the issue to verify
   this on their setup. But in that case I'd in addition ask to keep it in
   -proposed for a longer duration.

[Regression Potential]

 * The patch is rather huge myself and MP reviewers have tried to find an
   issue with the changes to createMemObject that was changed and found
   none. Never the less this is the area I'd expect issues if-any.
 * On the good side thou is that the code effectively is upstream and in
   Ubuntu >=18.10 for quit a while without being revoked since then (still
   looks similar in 20.04). There was one fixup for the primary change
   that was requested which is part of the suggested SRU.

[Other Info]

 * n/a

---

Our squid crashes many times with the follow errors in /var/log/squid/cache.log

root@proxy:~# grep -e failed /var/log/squid/cache.log
2020/01/08 06:25:55 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 06:26:31 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 06:26:37 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 06:41:04 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 06:41:41 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 06:56:10 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 06:56:48 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:11:17 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:11:54 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:26:25 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:27:01 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:31:56 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:32:01 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:32:07 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:32:11 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:32:15 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 07:32:19 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 08:17:21 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 08:26:50 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 08:27:28 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 08:32:28 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 08:34:48 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 08:41:55 kid1| assertion failed: store_client.cc:92: "mem"
2020/01/08 08:42:35 kid1| assertion failed: store_client.cc:92: "mem"

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: squid 3.5.27-1ubuntu1.4
ProcVersionSignature: Ubuntu 5.0.0-37.40~18.04.1-generic 5.0.21
Uname: Linux 5.0.0-37-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.9
Architecture: amd64
Date: Wed Jan 8 08:42:35 2020
InstallationDate: Installed on 2019-11-05 (63 days ago)
InstallationMedia: Ubuntu-Server 18.04.2 LTS "Bionic Beaver" - Release amd64 (20190210)
ProcEnviron:
 LANG=de_DE.UTF-8
 TERM=xterm-256color
 SHELL=/bin/bash
 PATH=(custom, no user)
SourcePackage: squid3
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.logrotate.d.squid: [modified]
modified.conffile..etc.squid.squid.conf: [modified]
mtime.conffile..etc.logrotate.d.squid: 2019-11-05T15:22:44.652392
mtime.conffile..etc.squid.squid.conf: 2019-11-05T15:56:05.841973

[Thomas (t.c)]

The Bug is already fixed upstream.

[Paride Legovini (paride)]

Thanks for this report and for linking it to the upstream bug.

The upstream patch is: https://github.com/squid-cache/squid/commit/76d61119ed913c9edd68aeb59ede23c10f119b34

[Paride Legovini (paride)]

I don't know if this is a "patch on plate" scenario, as the patch is for squid 4.x, while Bionic runs squid 3.5.27, and I don't know how much the codebases differ.

I also noticed that the tarballs from http://www.squid-cache.org/Versions/v3/3.5/ differ from the tarballs from https://github.com/squid-cache/squid/releases . This is true at least for version 3.5.27. The Ubuntu package is based on the tarball from squid-cache.org; the tarball generated from the SQUID_3_5_27 git tag *does* contain the fix already.

[Jacek Oleskow (jo2003)]

Hello,
I have this same Bug on several Ubuntu based Servers.
As @t.c said this Bug is fixed in the upstream project.
Is there an Update beeing planned that can fix this problem?

[Christian Ehrhardt  (paelzer)]

The fix is actually on
$ git describe --contains 76d61119ed913c9edd68aeb59ede23c10f119b34
SQUID_3_5_27~123

In Debian/Ubuntu it is since 4.0.23-1~exp8

Mark 4.x as fix released and lets take a look at a Bionic fix for it.

[Christian Ehrhardt  (paelzer)]

The change does not apply 100% cleanly to 3.5.27 as in Bionic.
But it does with fuzz 2 on the last two hunks.

I checked these and they seem fine to me.

[Christian Ehrhardt  (paelzer)]

I have build a test version in a PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3913
Please feel free to test this.

I added proposal for the changes of this fix at: https://code.launchpad.net/~paelzer/ubuntu/+source/squid3/+git/squid3/+merge/378527

@Thomas we will need some step-by-step how to trigger the bug for the SRU [1] process.
Since you hit it in your systems, did you find any way to reliably trigger this so that we can make an A/B comparison with/without fix as part of the SRU verification?

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

[Thomas (t.c)]

Hi Christian,

I installed your test package on a site, where squid crashes currently many times with the above error and will report back if it is fixed.

Thanks.

[Christian Ehrhardt  (paelzer)]

I've found an issue with the fix and added another fix to the PPA.
New version is: squid3 - 3.5.27-1ubuntu1.5~ppa2

reminder, I'm still looking for help to define the steps to reproduce.

[Christian Ehrhardt  (paelzer)]

For now - incomplete - until we hear back from PPA testing and if tests-steps can be defined.
I'm subscribed to get updates in that regard.

[Thomas (t.c)]

Hi, today I checked squids cache.log - the mem error is gone - but now we get:

root@proxy:~# grep failed /var/log/squid/cache.log | ccze -CA
2020/02/06 09:45:24 kid1| assertion failed: client_side_reply.cc:1153: "http->storeEntry()->objectLen() >= headers_sz"
2020/02/06 10:13:52 kid1| assertion failed: client_side_reply.cc:1153: "http->storeEntry()->objectLen() >= headers_sz"
2020/02/06 10:14:16 kid1| assertion failed: client_side_reply.cc:1153: "http->storeEntry()->objectLen() >= headers_sz"
2020/02/06 11:38:50 kid1| assertion failed: client_side_reply.cc:1153: "http->storeEntry()->objectLen() >= headers_sz"
2020/02/06 11:39:08 kid1| assertion failed: client_side_reply.cc:1153: "http->storeEntry()->objectLen() >= headers_sz"
2020/02/06 11:48:58 kid1| assertion failed: client_side_reply.cc:1153: "http->storeEntry()->objectLen() >= headers_sz"
2020/02/06 11:49:23 kid1| assertion failed: client_side_reply.cc:1153: "http->storeEntry()->objectLen() >= headers_sz"

[Thomas (t.c)]

Maybe this Upstream Bug:
https://bugs.squid-cache.org/show_bug.cgi?id=3369

[Thomas (t.c)]

Oh, right now I read, that you have provided a updatet package.

I will install this first and re-validate.

[Sergio Durigan Junior (sergiodj)]

Just a heads up that we're waiting to know if Christian's PPA solves the issue for Thomas.

[Lucas Kanashiro (lucaskanashiro)]

Hey @Thomas, any news about the fix proposed by Christian? Also steps to reproduce the issue.

[Paride Legovini (paride)]

I subscribed Jacek Oleskow to the bug as he reported he has several affected systems, maybe he'll be able to help with the verification.