mod_proxy_hcheck does not detect AJP/CPING support

[Sistemi CeSIA]

[Impact]

The Apache JServe Protocol (AJP) proxies inbound requests to an application server, such as health checks via CPING/CPONG. mod_proxy_hcheck added support for AJP/CPING in 22.04, however the following error is encountered when enabling it, which prevents the new feature from operating:
   BalancerMember Health check method CPING not (yet) implemented
This is caused by a incorrect check for AJP support in hc_post_config() that occurs too late, after the configuration syntax has already been marked invalid by the time the "hcmethod=CPING" token is found. The fix is to move the check from hc_post_config() to hc_pre_config().

[Test Plan]

Make a container for testing:

$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test

Type in:

# apt update && apt dist-upgrade -y
# apt install -y apache2

# cat > /etc/apache2/sites-available/httpd-hcheck-ajp.conf << __EOF__

<VirtualHost *:80>
    ServerAdmin <email address hidden>
    DocumentRoot "/var/www/html"
    ServerName myapp.example.com
    ErrorLog "${APACHE_LOG_DIR}/myapp.example.com-error_log"
    CustomLog "${APACHE_LOG_DIR}/myapp.example.com-access_log" common
    <Proxy balancer://myapp>
       BalancerMember ajp://127.0.0.1:8009/myapp route=app-route timeout=300 ping=3
 connectiontimeout=3 hcmethod=CPING
     </Proxy>
 ProxyPass /myapp balancer://myapp stickysession=JSESSIONID
     </VirtualHost>
__EOF__

# a2enmod proxy
# a2enmod proxy_ajp
# a2enmod proxy_hcheck
# a2ensite httpd-hcheck-ajp
# apachectl -t

Example of failed output:
# apachectl -t

AH00526: Syntax error on line 10 of /etc/apache2/sites-enabled/httpd-hcheck-ajp.conf:
BalancerMember Health check method CPING not (yet) implemented
Action ‘-t’ failed.

The Apache error log may have more information.

Example of successful output:
# apachectl -t
Syntax OK

[Where problems could occur]

The patch itself modifies the code of mod_proxy_hcheck, so any new bugs involving that module would be suspect. The patch changes configuration code, so issues cropping up that seem related to module configuration could be suspect. Finally, since the patch modifies C code, issues typical of C code (segfaults, memory leaks, …) would be possible, however since this moves a chunk of code unmodified this seems unlikely.

--------------------------original bug report-----------------------------

We were very excited to see mod_proxy_hcheck support for AJP/CPING land in Ubuntu 22.04, however trying to enable it results in the following error:

> BalancerMember Health check method CPING not (yet) implemented

This unfortunately renders AJP-based health checks, which is a new feature introduced in version 2.4.49 (and thus only available in Jammy), impossible to use.

Luckily we were able to find the code responsible for the misbehavior and propose a fix which has now been accepted upstream.

Upstream bug: https://bz.apache.org/bugzilla/show_bug.cgi?id=66300
Upstream fix: https://svn.apache.org/viewvc?view=revision&revision=1905608

We would like to see this fix backported in Jammy.

[Alessandro-cavalier7 (alessandro-cavalier7)]

We were very excited to see mod_proxy_hcheck support for AJP/CPING (r1887415) land in Ubuntu 22.04; however trying to enable it yields the following error:

   BalancerMember Health check method CPING not (yet) implemented

This was surprising for us since from a cursory look at the code this functionality seems implemented.

A couple of hours of debugging later we concluded that the order of execution of the code added in r1887415 seems to be incorrect: the code checking for AJP availability is executed in hc_post_config() which is unfortunately too late since the configuration syntax has already been marked invalid when the "hcmethod=CPING" token was found.

To fix this we simply moved the code checking for AJP availability in hc_pre_config(). We are running a patched version of http-2.4.52 on a couple of hosts running Ubuntu 22.04 and it seems to be working without issues so far.

[Alessandro-cavalier7 (alessandro-cavalier7)]

Created attachment 38404
mod_proxy_hcheck: move AJP availability detection in hc_pre_config()

[Alessandro-cavalier7 (alessandro-cavalier7)]

Attaching MCVE showing the issue and fix, use "docker buildx build -f <FILENAME> ." to run.

[Alessandro-cavalier7 (alessandro-cavalier7)]

Created attachment 38405
Dockerfile.bug

[Alessandro-cavalier7 (alessandro-cavalier7)]

Created attachment 38406
Dockerfile.fix

[V-jiz-h (v-jiz-h)]

Thanks for the report and the patch. It looks good. I'll do some more testing and apply to trunk and propose for back port to 2.4.x

[Paride Legovini (paride)]

Hello and thanks for this bug report. I did not try to setup a reproducer for this, but the bug has been triaged and fixed upstream, so there's no doubt it's valid.

According to the upstream CHANGES file the fix will be released in apache2 2.4.55, currently not in Lunar and not in Debian, so fixing this in Jammy/Kinetic will require patching the Lunar package first.

I think the upstream patch [1] makes this good SRU material.

[1] https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_hcheck.c?r1=1905608&r2=1905607&pathrev=1905608

[Bryce Harrington (bryce)]

The SRU process requires an easy to follow test case be documented with the bug, to allow it to be easily validated. Is that something you could assist us with in writing? You can see an example of what we're looking for from the [Test Case] in this bug report:

  https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1988224

If we can get a good [Test Case] defined for this bug report, it should hopefully help us expedite getting the fix out to folks.

[Sistemi CeSIA (sistemicesia)]

The linked bugzilla issue has 2 files attached with a complete and self-container example of the issue and fix, see

https://bz.apache.org/bugzilla/show_bug.cgi?id=66300#c2
https://bz.apache.org/bugzilla/show_bug.cgi?id=66300#c3
https://bz.apache.org/bugzilla/show_bug.cgi?id=66300#c4

Posting the one showing the issue for posterity:

--- BEGIN DOCKERFILE ---
# syntax=docker/dockerfile:1.3-labs
FROM ubuntu:22.04
RUN apt update && \
    apt install -y make autoconf libtool-bin gcc libaprutil1-dev subversion libpcre3-dev vim patch
RUN svn checkout http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x httpd-2.4.x && \
    cd httpd-2.4.x/ && \
    svn co http://svn.apache.org/repos/asf/apr/apr/trunk srclib/apr
RUN cd httpd-2.4.x/ && \
    ./buildconf && \
    ./configure --prefix=/opt/httpd-2.4.x && \
    make -j`nproc` && \
    make install
RUN /opt/httpd-2.4.x/bin/apachectl -t
COPY <<EOF /opt/httpd-2.4.x/conf/extra/httpd-hcheck-ajp.conf

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_hcheck_module modules/mod_proxy_hcheck.so

<VirtualHost *:80>
    ServerAdmin <email address hidden>
    DocumentRoot "/opt/httpd-2.4.x/docs/myapp.example.com"
    ServerName myapp.example.com
    ErrorLog "logs/myapp.example.com-error_log"
    CustomLog "logs/myapp.example.com-access_log" common

    <Proxy balancer://myapp>
        BalancerMember ajp://127.0.0.1:8009/myapp route=app-route timeout=300 ping=3 connectiontimeout=3 hcmethod=CPING
    </Proxy>

    ProxyPass /myapp balancer://myapp stickysession=JSESSIONID

</VirtualHost>
EOF
RUN echo 'Include conf/extra/httpd-hcheck-ajp.conf' >> /opt/httpd-2.4.x/conf/httpd.conf
RUN /opt/httpd-2.4.x/bin/apachectl -t
--- END DOCKERFILE ---

I can post here a modified version using ubuntu's packages instead of compiling apache2 from source if needed.

[Bryce Harrington (bryce)]

> I can post here a modified version using ubuntu's packages instead of compiling apache2 from source if needed.

Thanks, yes that would help, so we know of specifically what needs modified from stock. It looks like the main thing here is the config for the proxy balancer for the app?

I'm also curious about the inclusion of apr from upstream svn (and libaprutil1-dev which is presumably a build dep). Does the issue still crop up even with apr omitted?

[Sistemi CeSIA (sistemicesia)]

> It looks like the main thing here is the config for the proxy balancer for the app?

Yes, and specifically enabling the AJP health check (hcmethod=CPING) instead of the usual HTTP/TCP health checks (hcmethod=GET and hcmethod=TCP rispectively)

> I'm also curious about the inclusion of apr from upstream svn (and libaprutil1-dev which is presumably a build dep). Does the issue still crop up even with apr omitted?

Building from upstream svn (both httpd and apr) was just a mean to show apache2 maintainers that the issue can be reproduced with the latest bits; the issue is still present with apr installed from binary packages.

[Test Case]
# apt update && apt dist-upgrade -y
# apt install -y apache2
# cat > /etc/apache2/sites-available/httpd-hcheck-ajp.conf << __EOF__
<VirtualHost *:80>
    ServerAdmin <email address hidden>
    DocumentRoot "/var/www/html"
    ServerName myapp.example.com
    ErrorLog "logs/myapp.example.com-error_log"
    CustomLog "logs/myapp.example.com-access_log" common

    <Proxy balancer://myapp>
        BalancerMember ajp://127.0.0.1:8009/myapp route=app-route timeout=300 ping=3 connectiontimeout=3 hcmethod=CPING
    </Proxy>

    ProxyPass /myapp balancer://myapp stickysession=JSESSIONID

</VirtualHost>
__EOF__
# a2enmod proxy
# a2enmod proxy_ajp
# a2enmod proxy_hcheck
# a2ensite httpd-hcheck-ajp
# apachectl -t

Example failed output:

root@ubuntu:~# apachectl -t
AH00526: Syntax error on line 10 of /etc/apache2/sites-enabled/httpd-hcheck-ajp.conf:
BalancerMember Health check method CPING not (yet) implemented
Action '-t' failed.
The Apache error log may have more information.

Example of successful (expected) output:

root@ubuntu:~# apachectl -t
Syntax OK

[tititou (christophe-jaillet)]

Fix in trunk in r1904516
Backported in 2.4.x in r1906498

This is part of 2.4.55

[tititou (christophe-jaillet)]

Oops, backport in 2.4.x is r1905608

[Michał Małoszewski (michal-maloszewski99)]

It is in Lunar:
apache2 | 2.4.55-1ubuntu1 | lunar | source, amd64, arm64, armhf, i386, ppc64el, riscv64, s390x

Set Fix Released in Lunar.

[Michał Małoszewski (michal-maloszewski99)]

Taking care of that and preparing patches for Jammy and Kinetic.

[Steve Langasek (vorlon)]

An upload of apache2 to kinetic-proposed has been rejected from the upload queue for the following reason: "there is already an apache2 2.4.54-2ubuntu1.2 in kinetic-security, this needs rebased".

[Steve Langasek (vorlon)]

An upload of apache2 to jammy-proposed has been rejected from the upload queue for the following reason: "there is already an apache2 2.4.52-1ubuntu4.4 in jammy-security, this needs rebased".

[Michał Małoszewski (michal-maloszewski99)]

Thanks, rebased.

[Timo Aaltonen (tjaalton)]

Hello Sistemi, or anyone else affected,

Accepted apache2 into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.54-2ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Sistemi, or anyone else affected,

Accepted apache2 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apache2/2.4.52-1ubuntu4.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apache2 (2.4.52-1ubuntu4.5) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.48.2-0ubuntu1 (ppc64el)
libsoup3/3.0.7-0ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apache2 (2.4.54-2ubuntu1.3) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.50.2-2 (ppc64el)
libsoup3/3.2.0-1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#apache2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Christian Ehrhardt  (paelzer)]

The fail in gvfs/1.50.2-2 (ppc64el) is a common flaky case, it had hit systemd, glib2.0 and others before.

Example:
AssertionError: Items in the first set but not the second:
'hello_gvfs_tests.2.txt'

Furthermore the breakage isn't related to the changes we applied. Retries should fix this, scheduling them ...

The fail in libsoup3/3.2.0-1 (armhf)
This is another flaky case, having hit systemd before two times a few weeks ago.

Example:

# Apache command: '/usr/sbin/apache2' '-d' '/usr/libexec/installed-tests/libsoup-3.0' '-f' 'httpd.conf' '-c' 'ErrorLog /tmp/test-tmp-libsoup-3.0_connection-test.test-H7US21/error.log' '-c' 'PidFile /tmp/test-tmp-libsoup-3.0_connection-test.test-H7US21/httpd.pid' '-k' 'start'
(98)Address already in use: AH00072: make_sock: could not bind to address 127.0.0.1:47526
no listening sockets available, shutting down
...
FAIL: libsoup-3.0/connection-test.test (Child process exited with code 1)

I'll be retrying both

[Christian Ehrhardt  (paelzer)]

libsoup-3.0 resolved on retry

gvfs resolved on retry as well

[Christian Ehrhardt  (paelzer)]

Same for jammy now

[Michał Małoszewski (michal-maloszewski99)]

Jammy SRU Verification:

Fix works, 2.4.52-1ubuntu4.5 fixes the bug.

I've created the jammy container using steps from the [Test Plan] section listed above in the Bug Description and inside that container:

I have installed apache2 using:

$ apt install apache2

Then I typed in:

$ apt policy apache2

The output:

apache2:
  Installed: 2.4.52-1ubuntu4.4
  Candidate: 2.4.52-1ubuntu4.4
  Version table:
 *** 2.4.52-1ubuntu4.4 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.52-1ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Then I repeated steps from [Test Plan] section.

I've noticed that nothing has changed there, so the problem still existed, because as we could see in the output, the package version was not the one where the fix is.

I have enabled proposed.

Then I've upgraded apache2 using:

$ apt install apache2=2.4.52-1ubuntu4.5

Later I've typed in:

$ apt policy apache2
to check if installed version is changed and we see that we have new version installed (with fix)

apache2:
  Installed: 2.4.52-1ubuntu4.5
  Candidate: 2.4.52-1ubuntu4.5
  Version table:
 *** 2.4.52-1ubuntu4.5 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.52-1ubuntu4.4 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     2.4.52-1ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Finally when I repeated steps from the [Test Plan] the problem did not exist, "Syntax OK" message. So the fix works.

Important NOTE: If a syntax error pops up in the terminal, please open /etc/apache2/sites-enabled/httpd-hcheck-ajp.conf and remember to keep:
BalancerMember ajp://127.0.0.1:8009/myapp route=app-route timeout=300 ping=3 connectiontimeout=3 hcmethod=CPING
in one line.

[Michał Małoszewski (michal-maloszewski99)]

Kinetic SRU Verification:

Fix works, 2.4.54-2ubuntu1.3 fixes the bug.

I've created the kinetic container using steps from the [Test Plan] section listed above in the Bug Description and inside that container:

I have installed apache2 using:

$ apt install apache2

Then I typed in:

$ apt policy apache2

The output:

apache2:
  Installed: 2.4.54-2ubuntu1.2
  Candidate: 2.4.54-2ubuntu1.2
  Version table:
 *** 2.4.54-2ubuntu1.2 500
        500 http://archive.ubuntu.com/ubuntu kinetic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu kinetic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.54-2ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu kinetic/main amd64 Packages

Then I repeated steps from [Test Plan] section.

I've noticed that nothing has changed there, so the problem still existed, because as we could see in the output, the package version was not the one where the fix is.

I have enabled proposed.

Then I've upgraded apache2 using:

$ apt install apache2=2.4.54-2ubuntu1.3

Later I've typed in:

$ apt policy apache2
to check if installed version is changed and we see that we have new version installed (with fix)

apache2:
  apache2:
  Installed: 2.4.54-2ubuntu1.3
  Candidate: 2.4.54-2ubuntu1.3
  Version table:
 *** 2.4.54-2ubuntu1.3 500
        500 http://security.ubuntu.com/ubuntu kinetic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.54-2ubuntu1.2 500
        500 http://archive.ubuntu.com/ubuntu kinetic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu kinetic-security/main amd64 Packages
     2.4.54-2ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu kinetic/main amd64 Packages

Finally when I repeated steps from the [Test Plan] the problem did not exist, "Syntax OK" message. So the fix works.

Important NOTE: If a syntax error pops up in the terminal, please open /etc/apache2/sites-enabled/httpd-hcheck-ajp.conf and remember to keep:
BalancerMember ajp://127.0.0.1:8009/myapp route=app-route timeout=300 ping=3 connectiontimeout=3 hcmethod=CPING
in one line.

[Andreas Hasenack (ahasenack)]

I noticed that the jammy upload has a stray hunk in the patch[1], at the end:

--- /dev/null
+++ b/mod_proxy_hcheck_jammy_fix_to_detect_support.patch
@@ -0,0 +1 @@
+

That will create an empty mod_proxy_hcheck_jammy_fix_to_detect_support.patch file at the root of the source code[2]:

$ dquilt push -a
...
Applying patch mod_proxy_hcheck_jammy_fix_to_detect_support.patch
patching file modules/proxy/mod_proxy_hcheck.c
patching file mod_proxy_hcheck_jammy_fix_to_detect_support.patch

Now at patch mod_proxy_hcheck_jammy_fix_to_detect_support.patch

$ ls -la
...
-rw-rw-r-- 1 andreas andreas 53749 abr 27 10:08 Makefile.win
-rw-rw-r-- 1 andreas andreas 1 abr 27 10:40 mod_proxy_hcheck_jammy_fix_to_detect_support.patch
drwxrwxr-x 29 andreas andreas 4096 abr 27 10:08 modules
...

Another small issue is that the date in the DEP3 header of the jammy patch is incorrect:

$ grep Last-Update debian/patches/mod_proxy_hcheck_jammy_fix_to_detect_support.patch
Last-Update: 2022-03-01

None of these affect the build or the final result in the binary package, but it's something that should have been flagged and fixed prior to the upload.

Since it does not affect the build, and the package is in proposed already, I'll release it, but please double check next time.

1. https://git.launchpad.net/ubuntu/+source/apache2/tree/debian/patches/mod_proxy_hcheck_jammy_fix_to_detect_support.patch?h=applied/ubuntu/jammy-devel
2. https://git.launchpad.net/ubuntu/+source/apache2/tree/mod_proxy_hcheck_jammy_fix_to_detect_support.patch?h=applied/ubuntu/jammy-devel

[Andreas Hasenack (ahasenack)]

I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions, or they were fixed.

There is no SRU freeze ongoing at the moment.

There is no halted phasing on the previous update.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.54-2ubuntu1.3

---------------
apache2 (2.4.54-2ubuntu1.3) kinetic; urgency=medium

  * d/p/mod_proxy_hcheck_kinetic_fix_to_detect_support.patch: Fix issue
    where enabling mod_proxy_hcheck results in error (LP: #1998311)

 -- Michal Maloszewski <email address hidden> Thu, 02 Mar 2023 00:01:26 +0100

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for apache2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apache2 - 2.4.52-1ubuntu4.5

---------------
apache2 (2.4.52-1ubuntu4.5) jammy; urgency=medium

  * d/p/mod_proxy_hcheck_jammy_fix_to_detect_support.patch: Fix issue
    where enabling mod_proxy_hcheck results in error (LP: #1998311)

 -- Michal Maloszewski <email address hidden> Wed, 01 Mar 2023 23:43:55 +0100