FATAL FwdState::noteDestinationsEnd exception: opening()

[laszloj]

[ Impact ]

From the bug fix at https://github.com/squid-cache/squid/commit/1332f8d606485b5a2f57277634c2f6f7855bd9a6

"""
FwdState used to check serverConn to decide whether to open a connection
to forward the request. ... a nil serverConn pointer
does not imply that a new connection should be opened: FwdState
helper jobs may already be working on preparing an existing open
connection (e.g., sending a CONNECT request or negotiating encryption).

Bad serverConn checks in both FwdState::noteDestination() and
FwdState::noteDestinationsEnd() methods led to extra connectStart()
calls creating two conflicting concurrent helper jobs.
"""

This leads squid to random crashes such as

FATAL FwdState::noteDestinationsEnd exception: opening

[ Test plan ]

Since we have no reliable reproducer for the issue, we will need to rely on the upstream test suite introduced with the fix and on user tests in -proposed and post release reports.

[ Where problems could occur ]

The regression potentials for the changeset being applied here is listed at LP: #2013423

[ Other Info ]

This bug is being fixed as part of the squid 5.7 MRE at LP: #2013423.

[ Original message]

Squid 5.2 shipped with jammy is affected by a bug that causes random crashes. The underlying issue was fixed in version 6 and backported to 5.4.1.

Additional information:
Original PR for v6: https://github.com/squid-cache/squid/pull/877
Original bug report: https://bugs.squid-cache.org/show_bug.cgi?id=5055
5.4.1 backport: https://github.com/squid-cache/squid/commit/1332f8d606485b5a2f57277634c2f6f7855bd9a6

[Bryce Harrington (bryce)]

Hi laszloj,

The upstream bug #5055 mentions "This change has several overlapping parts. Unfortunately, merging individual parts is both difficult and likely to cause crashes." Indeed, it looks like several upstream commits would need backported, involving a non-trivial amount of code changes. Due to that, it is especially important to define a reproduction case; otherwise it's hard to know for certain that the patches are actually fixing the problem, and it will then be challenging to get this approved for SRU to jammy.

However, we may be able to proceed if you can define the bounds on the random crashes. How often do the crashes happen? E.g. daily? Every 1000 connection requests? Would you be able to script a workload that attempts a bunch of connections that triggers the crash on your hardware?

[laszloj (laszloj)]

Hi Bryce,

Thank you for the prompt response.
I wasn't able to reproduce the crashes with a script yet, however, I have been able to isolate the issue a bit more.
It seems like it only happens when ssl-bump is used - I'm using the newly introduced squid-openssl package.
I've attached a config file which is based on the following entry in Squid's FAQ and didn't cause the same crashes in squid 4.x (compiled with openssl): https://wiki.squid-cache.org/SquidFaq/WindowsUpdate

Using this config requires the squid-openssl package and the below commands must be executed before starting the service, otherwise squid gets stuck in a crash loop:
---
/usr/lib/squid/security_file_certgen -c -s /var/spool/squid/ssl_db -M 4MB && chown -R proxy:proxy /var/spool/squid/ssl_db
---

If I set the proxy config on my Windows laptop to point at the squid proxy running with the attached config on Ubuntu 22.04, squid crashes and restarts as soon as I try to open Outlook or Word. I cannot reproduce this issue with a config where ssl-bump is not used, even when using the squid-openssl package.
I tried using a Python script to simultaneously call some of the https URLs that I found in the logs, but squid didn't crash that way.
I'll try reproducing it using an app on an Ubuntu desktop VM and keep trying with the script as well.

[Paride Legovini (paride)]

If I understand the squid upstream release process [1] correctly, squid 5.5 could be considered a new upstream microrelease [2], and we could tackle this *and other* issues by releasing it as an SRU. The upstream changelog for the v5 branch [3] only points to bug fixes, as expected.

I'm marking this bug report for team discussion.

[1] https://wiki.squid-cache.org/ReleaseProcess
[2] https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases
[3] https://github.com/squid-cache/squid/blob/v5/ChangeLog

[Bryce Harrington (bryce)]

Hi laszloj,

The server team discussed this bug today at our weekly meeting and considered between cherrypicking the specific fixes for this bug, vs. applying for a 'Micro-Release Exception' (MRE) for squid itself, which if accepted would allow backporting squid 5.5 itself, as Paride described. The trade-offs are that cherrypick type SRUs generally need a well-defined test case to reliably reproduce the bug. MRE SRUs don't require individual test cases for bugs, but the process to gain an MRE tends to take more time - and potentially a lot more time - but once accepted, subsequent micro-release updates go much more smoothly.

Since for this bug, we don't yet have a defined reproducer, and since it sounds like the fix involves multiple patches, the general consensus of the team was that we should probably focus our time towards getting the MRE application for squid. However, if you are able to figure out a reliable way to reproduce the bug synthetically, we can revisit that and try to get this specific fix SRU'd.

Some additional discussion about this: https://lists.ubuntu.com/archives/ubuntu-server/2022-June/009293.html

[laszloj (laszloj)]

Hi Bryce,

Thank you for getting back to me and sorry for the belated response, I really appreciate that you are looking into this.
I did not succeed with a synthetic reproduction of the bug, so I ended up removing ssl-bump from my config as the crashes made Squid unfit for my use case, which is mainly for caching Ubuntu updates and some other OS updates/packages. This way I am able to cache Ubuntu updates (or anything served through HTTP), however, I am unable to cache anything through HTTPS. It has been running for a few weeks now and I have not observed the frequent crashes though.

Not sure if this helps, but I have found another report in the squid-users mailing list: https://<email address hidden>/msg23816.html

If I manage to come across a way to reproduce the issue, I will let you know.
Thank you.

[laszloj (laszloj)]

Hi Athos,

I can see that the status of this bug was changed to "Fix Released" - I'm just curious, does this mean that a fixed version will be released to Jammy soon? Thank you.

[Athos Ribeiro (athos-ribeiro)]

Hi laszloj,

Sorry for the lack of communication on this one. Here is what is going on:

As you mentioned in your report, the issue was fixed in squid 5.4.1 (https://github.com/squid-cache/squid/blob/master/ChangeLog#L153).

Then, I targeted this bug to jammy, and set the main target to fix released. This is to reflect the fact that the package in Ubuntu's development version (23.04) is fixed.

As for a jammy fix, there is an ongoing effort to bump squid's version to 5.7 in jammy, which we are tracking at LP: #2013423. Once this bump is approved by the SRU team, I will proceed to upload the new version and we can test it (help is always welcome here) and push to the update pocket.

[laszloj (laszloj)]

Hi Athos,

Thank you for the update, that makes sense. I am happy to help with testing if needed.

[Steve Langasek (vorlon)]

An upload of squid to jammy-proposed has been rejected from the upload queue for the following reason: "documented to break certain user configs on upgrade".

[Paul Gear (paulgear)]

Is there anything people currently affected by this bug can do to resolve it, assuming that we are happy to fix our config on upgrade? Is it available in a PPA or non-default pocket?

[Athos Ribeiro (athos-ribeiro)]

Hi Paul,

[1] has a 5.7 build for jammy which was a first attempt on providing an MRE for squid (see [2]). Note that it was rejected by the SRU team and we are currently working on a solution for the issue that led it to be rejected [3].

Since the issue lies within the kinetic upgrades, for now, I suppose the potential 5.7 update for jammy would not change much from what is available in that PPA (some documentation will definitely be changed).

Finally, note that the build in that PPA is just a test build I pushed for review purposes.

[1] https://launchpad.net/~athos-ribeiro/+archive/ubuntu/squid-5.7-mre/+packages
[2] https://wiki.ubuntu.com/SquidUpdates
[3] https://bugs.launchpad.net/ubuntu/+source/squid/+bug/2013423

[Andreas Hasenack (ahasenack)]

Hello laszloj, or anyone else affected,

Accepted squid into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/squid/5.7-0ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Simon Déziel (sdeziel)]

I've never experienced the crash/assertion with my squid.conf but I still took the -proposed package for a test drive:

The following packages will be upgraded:
   squid (5.2-1ubuntu4.4 => 5.7-0ubuntu0.22.04.1)
   squid-common (5.2-1ubuntu4.4 => 5.7-0ubuntu0.22.04.1)

...

And I'm happy to report that I saw no regression associated with the version bump. So not a real "verification-done" report but more of an I'm happy with the MRE and no regression here. Thanks!

[Athos Ribeiro (athos-ribeiro)]

Thanks, Simon.

- Build logs look sane;
- The upstream test suite and autopkgtest are passing;
- The package installs and run simple cases correctly.

As per the designed test plan, This should be good to land.

[Launchpad Janitor (janitor)]

This bug was fixed in the package squid - 5.7-0ubuntu0.22.04.1

---------------
squid (5.7-0ubuntu0.22.04.1) jammy; urgency=medium

  * New upstream version. (LP: #2013423):
    - Fix FATAL FwdState::noteDestinationsEnd exception. (LP: #1975399)
    - Fix regression that made the default value for the esi_parser
      configuration directive behave differently from its documented behavior.
      It now correctly uses libxml2 if available and falls back to libexpat
      otherwise.
    - Fix unexpected dispatch of client CA certificates to https_port clients
      when OpenSSL SSL_MODE_NO_AUTO_CHAIN mode is on.
    - Add OpenSSL 3.0 support for features that were already supported by
      squid. No new OpenSSL 3.0 feature support added at this time.
    - The configuration directive ssl_engine is no longer recognized. Since
      this option is not implemented for the OpenSSL 3 used in Ubuntu 22.04
      LTS, this is not a functional regression. Now, instead of failing with
      "FATAL: Your OpenSSL has no SSL engine support", it fails with "FATAL:
      bad configuration: Cannot use ssl_engine in Squid built with OpenSSL 3.0
      or newer".
    - For a comprehensive list of changes, please see
      http://www.squid-cache.org/Versions/v5/ChangeLog.html.
  * d/p/close-tunnel-if-to-server-conn-closes-after-client.patch: remove
    upstreamed patch.
    [ Fixed in 5.4 ]
  * d/p/0004-Change-default-Makefiles-for-debian.patch: remove upstreamed
    patch.
    [ Fixed in 5.5 ]
  * d/p/CVE-2021-46784.patch: remove upstreamed patch.
    [ Fixed in 5.6 ]
  * d/p/CVE-2022-41317.patch: drop patch to fix typo in manager ACL.
    [ Fixed in 5.7 ]
  * d/p/CVE-2022-41318.patch: drop patch to fix NTLM decoder truncated strings.
    [ Fixed in 5.7 ]
  * d/p/openssl3-*.patch: drop downstream OpenSSL 3 support patch.
    [ Fixed in 5.7 ]
  * d/p/99-ubuntu-ssl-cert-snakeoil.patch: refresh patch.

squid (5.2-1ubuntu4.4) jammy; urgency=medium

  * Make builds fail when upstream test suite fails (LP: #2004050):
    - d/p/series: do not rely on installed binaries for build time tests.
    - d/rules: halt build upon test failures.
    - d/rules: do not include additional configuration files during
      build time tests. This would lead to test failures due to missing
      paths.
    - d/t/upstream-test-suite: use installed squid binary for
      autopkgtest config file checks.

 -- Athos Ribeiro <email address hidden> Thu, 30 Mar 2023 17:06:59 -0300

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for squid has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.