MRE updates of haproxy for focal, jammy and lunar

[Bryce Harrington]

This bug tracks an update for the HAProxy package in the following Ubuntu
releases to the versions below:

 * Focal (20.04): HAProxy 2.0.33
 * Jammy (22.04): HAProxy 2.4.24

These updates are a best effort to follow the SRU policy exception defined
at https://wiki.ubuntu.com/HAProxyUpdates. Please, see the "Regression Potential" section (and its subsections) below.

[Upstream changes]

* Focal (20.04): https://www.haproxy.org/download/2.0/src/CHANGELOG (See entries for 2.0.32 and 2.0.33).
* Jammy (22.04): https://www.haproxy.org/download/2.4/src/CHANGELOG (See entries for 2.4.23 and 2.4.24).

Important bug fixes include:

- Fix a buffering issue when performing multiple large-header replacements at once, which could overwrite parts of the contents of the headers.

- Fix for CVE-2023-40225. Empty content-length headers in requests/responses are now rejected (This is already applied in Ubuntu).

- Several lower severity fixes.

[Test Plan]

Since the upstream CI piplines do not run (publicly) for HAProxy 2.4 and 2.0, we triggered those using the upstream project github workflows:

HAproxy 2.0.33 (focal): https://github.com/athos-ribeiro/haproxy-2.0/actions
HAproxy 2.4.24 (jammy): https://github.com/athos-ribeiro/haproxy-2.4/actions

The only errors in the workflows are related to gcc on Windows, and spelling related issues, which should not be blockers here.

There is also an error in the spec compliance run for the 2.4 actions. However, we can see in the actions matrix that upstream did add a "-Wno-deprecated-declarations" when openssl3 is being used for the other test runs (it seems it is just missing for this run). I patched the github actions workflow to add the missing flag and the test passes, as one can see in the 2.4 github repository above.

We ran autopkgtest for the new versions in the following PPA:
https://launchpad.net/~athos-ribeiro/+archive/ubuntu/haproxy-mre/+packages

Here are the results:

* Results:
  - haproxy/2.0.33-0ubuntu0.1
    + ✅ haproxy on focal for amd64 @ 31.10.23 23:06:37 https://autopkgtest.ubuntu.com/results/autopkgtest-focal-athos-ribeiro-haproxy-mre/focal/amd64/h/haproxy/20231031_230637_6572d@/log.gz
    + ✅ haproxy on focal for arm64 @ 31.10.23 23:03:26 https://autopkgtest.ubuntu.com/results/autopkgtest-focal-athos-ribeiro-haproxy-mre/focal/arm64/h/haproxy/20231031_230326_40487@/log.gz
    + ✅ haproxy on focal for armhf @ 31.10.23 23:02:18 https://autopkgtest.ubuntu.com/results/autopkgtest-focal-athos-ribeiro-haproxy-mre/focal/armhf/h/haproxy/20231031_230218_40487@/log.gz
    + ✅ haproxy on focal for ppc64el @ 31.10.23 22:59:11 https://autopkgtest.ubuntu.com/results/autopkgtest-focal-athos-ribeiro-haproxy-mre/focal/ppc64el/h/haproxy/20231031_225911_40487@/log.gz
    + ✅ haproxy on focal for s390x @ 31.10.23 23:00:08 https://autopkgtest.ubuntu.com/results/autopkgtest-focal-athos-ribeiro-haproxy-mre/focal/s390x/h/haproxy/20231031_230008_40487@/log.gz

  - haproxy/2.4.24-0ubuntu0.22.04.1
    + ✅ haproxy on jammy for amd64 @ 31.10.23 23:06:26 https://autopkgtest.ubuntu.com/results/autopkgtest-jammy-athos-ribeiro-haproxy-mre/jammy/amd64/h/haproxy/20231031_230626_670ea@/log.gz
    + ✅ haproxy on jammy for arm64 @ 31.10.23 23:28:44 https://autopkgtest.ubuntu.com/results/autopkgtest-jammy-athos-ribeiro-haproxy-mre/jammy/arm64/h/haproxy/20231031_232844_8d106@/log.gz
    + ✅ haproxy on jammy for armhf @ 31.10.23 22:56:34 https://autopkgtest.ubuntu.com/results/autopkgtest-jammy-athos-ribeiro-haproxy-mre/jammy/armhf/h/haproxy/20231031_225634_45bb9@/log.gz
    + ✅ haproxy on jammy for ppc64el @ 31.10.23 23:00:08 https://autopkgtest.ubuntu.com/results/autopkgtest-jammy-athos-ribeiro-haproxy-mre/jammy/ppc64el/h/haproxy/20231031_230008_806ab@/log.gz
    + ✅ haproxy on jammy for s390x @ 31.10.23 22:55:01 https://autopkgtest.ubuntu.com/results/autopkgtest-jammy-athos-ribeiro-haproxy-mre/jammy/s390x/h/haproxy/20231031_225501_806ab@/log.gz

[Regression Potential]

HAProxy itself does not have many reverse dependencies, however, any upgrade is
a risk to introduce some breakage to other packages. Whenever a test failure is
detected, we will be on top of it and make sure it doesn't affect existing
users.

[Regression Potential - Changes Analysis (CA)]

There is a significant number of low regression risk (as per upstream classification) functional changes.

Moreover, some (fewer) bug fixes have a possible major regression risk (again, as per upstream classification).

The functional changes mentioned above were included because they are, in majority, needed by other entries which are bug fixes, i.e., these are functional changes needed to fix specific bugs.

[Regression Potential - CA - Upstream changes classification criteria]

https://github.com/haproxy/haproxy/blob/master/CONTRIBUTING#L632
describes the upstream guidelines for tagging the entries in the upstream changelog based
on their purpose, importance, severity, etc.

Below, I summarize the relevant bits of such guidelines.

Patches "fixing a bug must have the 'BUG' tag", e.g., "BUG/MAJOR: description"

"When the patch cannot be categorized, [...] only use a risk or complexity
information [...]. This is commonly the case for new features". For
instance, "MINOR: description"

For MINOR tags, the patch "is safe enough to be backported to stable
branches".

Patches tagged MEDIUM "may cause unexpected regressions of low importance
[...], the patch is safe but touches working areas".

Patches tagged MAJOR carry a "major risk of hidden regression".

There is also a CRITICAL tag but no changes are tagged with it in the new
candidate versions.

[Regression Potential - CA - Impact]

For the next Focal MRE, we would upgrade HAPRoxy from 2.0.31 to 2.0.33. Among
the changes, there are 2 bug fixes tagged as BUG/MAJOR and 9 uncategorized changes
(potentially functional) tagged as MINOR.

For the next Jammy MRE, we would upgrade HAPRoxy from 2.4.22 to 2.4.24. Among
the changes, there are 2 bug fixes tagged as BUG/MAJOR (the same ones being fixed
in Focal), and 1 MEDIUM and 22 MINOR uncategorized changes (potentially functional).

For the next Lunar MRE, we would upgrade HAPRoxy from 2.6.9 to 2.6.15. This is
the largest change set for all the possible HAProxy MREs we may want to
propose. Among the changes, there are 8 bug fixes tagged as BUG/MAJOR, 3 MAJOR, 8
MEDIUM and 65 MINOR uncategorized changes (potentially functional).

[Regression Potential - CA - Assessment]

Below we discuss the changes with the greater regression potential (and the most relevant uncategorized ones, which may contain functional changes)

Focal 20.04:

- BUG/MAJOR: http-ana: Get a fresh trash buffer for each header value replacement
- BUG/MAJOR: http: reject any empty content-length header value

It seems that, for both of them, the MAJOR tag was chosen due to the severity
of the bugs and not due to the regression potential. The first, changes the way
replace-header operations are buffered. The latter changes the content-length
header parser to reject requests/responses with empty values for that header.

The remaining uncategorized Focal changes are all MINOR, and are either adding
new internal functions used by other bug fixes, other internal changes where
regressions are not expected, or are discussed below.

- MINOR: h2: pass accept-invalid-http-request down the request parser
 While this change is only forwarding a new parameter, it is used for a bug fix
 which makes HAProxy stop accepting some invalid characters as part of the URI
 component. According to upstream, "such requests appear at a rate of roughly 1
 per million and only come from attacks or poorly written web crawlers
 incorrectly following links found on various pages". After this change set, the
 former behavior can still be turned on with the accept-invalid-http-request
 option.

- MINOR: checks: make sure spread-checks is used also at boot time
 Make randomly spread health check timings to also be used at boot time.
 This seems to be a small enhancement and not a bug fix.

Jammy 22.04:

It carries the same changes listed for Focal, and:

- MEDIUM: proto_ux: properly suspend named UNIX listeners
  Before this patch, suspending a listener using named UNIX sockets would
  result in a no-op and recv events on the socket could still be processed.
  This changes the behavior to stop listening on those sockets when the
  listener is suspended.
- MINOR: startup: HAPROXY_STARTUP_VERSION contains the version used to start
  This is also an improvement instead of a fix. This sets the
  HAPROXY_STARTUP_VERSION environment variable, which contains the version used
  to start the master in a master-worker mode.

Lunar 23.04:

For Lunar, there are too many changes classified in the categories mentioned above, which drastically increases the regression risk.

Since Mantic is already on 2.6.15, Lunar being an interim series, and given we
are approaching the Mantic release date, we are proposing to skip the Lunar MRE entirely.

While no version comparison issues would arise due to skipping Lunar, a non-LTS
upgrade path (Jammy->Lunar) would introduce regressions in the package, which
would be fixed with another upgrade (Lunar->Mantic). As per the item 1 at https://wiki.ubuntu.com/StableReleaseUpdates#Newer_Releases, while not ideal, having the fixes in Mantic should cover this case (the fixes are already present in Mantic).

[Appendix A - Upstream potentially breaking changes list]

Below you will find the list of changes I extracted from the full changelogs of
the candidate MRE versions. I filtered the changelogs with the following command:

$ cat 2.0_changelog 2.4_changelog 2.6_changelog | grep -E '^[^ ]|(- )?(MAJOR|CRIT)|- (MINOR|MEDIUM)'

This selected only the unclassified (not bug fixing) changes and the bug fixing
changes classified as BUG/MAJOR and BUG/CRITICAL.

Focal - potentially upgrading from 2.0.31 to 2.0.33
ChangeLog :
===========
2023/08/19 : 2.0.33
 - BUG/MAJOR: http-ana: Get a fresh trash buffer for each header value replacement
 - BUG/MAJOR: http: reject any empty content-length header value
 - MINOR: ist: add new function ist_find_range() to find a character range
 - MINOR: ist: Add istend() function to return a pointer to the end of the string
 - MINOR: http: add new function http_path_has_forbidden_char()
 - MINOR: h2: pass accept-invalid-http-request down the request parser
2023/06/12 : 2.0.32
 - MINOR: proxy/pool: prevent unnecessary calls to pool_gc()
 - MINOR: checks: make sure spread-checks is used also at boot time
 - MINOR: clock: measure the total boot time
 - MINOR: spoe: Don't stop disabled proxies
 - MINOR: server: explicitly commit state change in srv_update_status()

Jammy - potentially upgrading from 2.4.22 to 2.4.24
ChangeLog :
===========
2023/08/19 : 2.4.24
 - MINOR: proto_uxst: add resume method
 - MINOR: listener/api: add lli hint to listener functions
 - MINOR: listener: add relax_listener() function
 - MINOR: listener: workaround for closing a tiny race between resume_listener() and stopping
 - MINOR: listener: make sure we don't pause/resume bypassed listeners
 - MINOR: listener: pause_listener() becomes suspend_listener()
 - MEDIUM: proto_ux: properly suspend named UNIX listeners
 - MINOR: proto_ux: ability to dump ABNS names in error messages
 - MINOR: lua: Add a function to get a reference on a table in the stack
 - MINOR: hlua: add simple hlua reference handling API
 - MINOR: hlua: simplify lua locking
 - MINOR: sink/api: pass explicit maxlen parameter to sink_write()
 - BUG/MAJOR: http-ana: Get a fresh trash buffer for each header value replacement
 - BUG/MAJOR: http: reject any empty content-length header value
 - MINOR: ist: add new function ist_find_range() to find a character range
 - MINOR: http: add new function http_path_has_forbidden_char()
 - MINOR: h2: pass accept-invalid-http-request down the request parser
2023/06/09 : 2.4.23
 - MINOR: startup: HAPROXY_STARTUP_VERSION contains the version used to start
 - MINOR: proxy/pool: prevent unnecessary calls to pool_gc()
 - MINOR: proxy: check if p is NULL in free_proxy()
 - MINOR: checks: make sure spread-checks is used also at boot time
 - MINOR: clock: measure the total boot time
 - MINOR: spoe: Don't stop disabled proxies
 - MINOR: server: explicitly commit state change in srv_update_status()
 - MINOR: proxy: add http_free_redirect_rule() function

Lunar - potentially upgrading from 2.6.9 to 2.6.15
ChangeLog :
===========
2023/08/09 : 2.6.15
 - MINOR: compression/slz: add support for a pure flush of pending bytes
 - MINOR: quic: Move QUIC encryption level structure definition
 - MINOR: quic: Move packet number space related functions
 - MINOR: quic: Reduce the maximum length of TLS secrets
 - MINOR: sink/api: pass explicit maxlen parameter to sink_write()
 - MINOR: quic: Make ->set_encryption_secrets() be callable two times
 - MINOR: quic: Useless call to SSL_CTX_set_quic_method()
 - BUG/MAJOR: http-ana: Get a fresh trash buffer for each header value replacement
 - BUG/MAJOR: h3: reject header values containing invalid chars
 - BUG/MAJOR: http: reject any empty content-length header value
 - MINOR: ist: add new function ist_find_range() to find a character range
 - MINOR: http: add new function http_path_has_forbidden_char()
 - MINOR: h2: pass accept-invalid-http-request down the request parser
2023/06/09 : 2.6.14
 - MINOR: ssl: ssl_sock_load_cert_chain() display error strings
 - MINOR: quic: use real sending rate measurement
 - MINOR: spoe: Don't stop disabled proxies
 - MINOR: http-rules: Add missing actions in http-after-response ruleset
 - MINOR: proxy: add http_free_redirect_rule() function
 - MINOR: htx: add function to set EOM reliably
 - MINOR: checks: make sure spread-checks is used also at boot time
 - MINOR: clock: measure the total boot time
 - MINOR: mux-quic: uninline qc_attach_sc()
2023/05/02 : 2.6.13
 - MINOR: proxy/pool: prevent unnecessary calls to pool_gc()
 - MINOR: quic: Add missing traces in cubic algorithm implementation
 - BUG/MAJOR: quic: Congestion algorithms states shared between the connection
 - MINOR: server: add SRV_F_DELETED flag
 - MINOR: http-ana: Add a HTTP_MSGF flag to state the Expect header was checked
 - MINOR: quic: Trace fix in quic_pto_pktns() (handshaske status)
 - MINOR: quic: Modify qc_try_rm_hp() traces
 - MINOR: quic: Dump more information at proto level when building packets
 - MINOR: quic: Add a trace for packet with an ACK frame
 - MINOR: quic: Add connection flags to traces
 - MINOR: quic: Remove a useless test about probing in qc_prep_pkts()
 - MINOR: quic: Do not allocate too much ack ranges
 - MINOR: proto_uxst: add resume method
 - MINOR: listener/api: add lli hint to listener functions
 - MINOR: listener: add relax_listener() function
 - MINOR: listener: workaround for closing a tiny race between resume_listener() and stopping
 - MINOR: listener: make sure we don't pause/resume bypassed listeners
 - MINOR: listener: pause_listener() becomes suspend_listener()
 - MEDIUM: proto_ux: properly suspend named UNIX listeners
 - MINOR: proto_ux: ability to dump ABNS names in error messages
 - MINOR: hlua: add simple hlua reference handling API
 - MINOR: hlua: simplify lua locking
 - MINOR: quic: Add traces to qc_kill_conn()
 - MINOR: quic: Add trace to debug idle timer task issues
 - MINOR: quic: Add <pto_count> to the traces
 - MINOR: quic: Display the packet number space flags in traces
 - MINOR: server: explicitly commit state change in srv_update_status()
 - MINOR: quic: Move traces at proto level
 - MINOR: mux-quic: do not set buffer for empty STREAM frame
 - MINOR: mux-quic: do not allocate Tx buf for empty STREAM frame
2023/03/28 : 2.6.12
 - BUG/MAJOR: poller: drop FD's tgid when masks don't match
2023/03/17 : 2.6.11
 - MINOR: buffer: add br_single() to check if a buffer ring has more than one buf
 - MINOR: h2: add h2_phdr_to_ist() to make ISTs from pseudo headers
 - MEDIUM: mux-h2/trace: add tracing support for headers
 - MINOR: trace: add a TRACE_ENABLED() macro to determine if a trace is active
 - MINOR: trace: add a trace_no_cb() dummy callback for when to use no callback
 - MINOR: trace: add the long awaited TRACE_PRINTF()
 - BUG/MAJOR: qpack: fix possible read out of bounds in static table
2023/03/10 : 2.6.10
 - MINOR: startup: HAPROXY_STARTUP_VERSION contains the version used to start
 - MINOR: fd/cli: report the polling mask in "show fd"
 - MINOR: mux-h2/traces: do not log h2s pointer for dummy streams
 - MINOR: mux-h2/traces: add a missing TRACE_LEAVE() in h2s_frt_handle_headers()
 - MINOR: ssl: rename confusing ssl_bind_kws
 - MEDIUM: epoll: don't synchronously delete migrated FDs
 - MEDIUM: poller: program the update in fd_update_events() for a migrated FD
 - MAJOR: fd: remove pending updates upon real close
 - MINOR: fd: delete unused updates on close()
 - MEDIUM: fd: add the tgid to the fd and pass it to fd_insert()
 - MINOR: cli/fd: show fd's tgid and refcount in "show fd"
 - MINOR: fd: add functions to manipulate the FD's tgid
 - MINOR: fd: add fd_get_running() to atomically return the running mask
 - MAJOR: fd: grab the tgid before manipulating running
 - MINOR: fd: make fd_clr_running() return the previous value instead
 - MEDIUM: fd: make fd_insert/fd_delete atomically update fd.tgid
 - MEDIUM: fd: quit fd_update_events() when FD is closed
 - MAJOR: poller: only touch/inspect the update_mask under tgid protection
 - MEDIUM: fd: support broadcasting updates for foreign groups in updt_fd_polling
 - BUG/MAJOR: fd/thread: fix race between updates and closing FD
 - BUG/MAJOR: fd/threads: close a race on closing connections after takeover
 - MINOR: h3/hq-interop: handle no data in decode_qcs() with FIN set
 - MINOR: quic: adjust request reject when MUX is already freed
 - MINOR: quic: Move code to wakeup the timer task to avoid anti-amplication deadlock

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Timo Aaltonen (tjaalton)]

Hello Bryce, or anyone else affected,

Accepted haproxy into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/2.4.24-0ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Bryce, or anyone else affected,

Accepted haproxy into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/2.0.33-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Athos Ribeiro (athos-ribeiro)]

The packages in the proposed pocket seem to work fine. There are no regressions in autopkgtest for these packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package haproxy - 2.0.33-0ubuntu0.1

---------------
haproxy (2.0.33-0ubuntu0.1) focal; urgency=medium

  * New upstream release (LP: #2028418)
    - Major and critical bug fixes according to the upstream changelog:
      + BUG/MAJOR: http-ana: Get a fresh trash buffer for each header value
        replacement
      + BUG/MAJOR: http: reject any empty content-length header value
    - For further information, refer to the upstream changelog at
      https://www.haproxy.org/download/2.0/src/CHANGELOG and to the upstream
      release announcements at
      https://<email address hidden>/msg43668.html
      (2.0.32), and
      https://<email address hidden>/msg43904.html (2.0.33)
    - Remove patches applied by upstream in debian/patches:
      + CVE-2023-40225-1.patch
      + CVE-2023-40225-2.patch

 -- Athos Ribeiro <email address hidden> Tue, 31 Oct 2023 16:00:44 -0300

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for haproxy has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package haproxy - 2.4.24-0ubuntu0.22.04.1

---------------
haproxy (2.4.24-0ubuntu0.22.04.1) jammy; urgency=medium

  * New upstream release (LP: #2028418)
    - Major and critical bug fixes according to the upstream changelog:
      + BUG/MAJOR: http-ana: Get a fresh trash buffer for each header value
        replacement
      + BUG/MAJOR: http: reject any empty content-length header value
    - For further information, refer to the upstream changelog at
      https://www.haproxy.org/download/2.4/src/CHANGELOG and to the upstream
      release announcements at
      https://<email address hidden>/msg43664.html
      (2.4.23), and
      https://<email address hidden>/msg43901.html (2.4.24)
    - Remove patches applied by upstream in debian/patches:
      + CVE-2023-40225-1.patch
      + CVE-2023-40225-2.patch

 -- Athos Ribeiro <email address hidden> Tue, 31 Oct 2023 11:16:29 -0300