mod_proxy_hcheck does not detect AJP/CPING support

[Sistemi CeSIA]

[Impact]

The Apache JServe Protocol (AJP) proxies inbound requests to an application server, such as health checks via CPING/CPONG. mod_proxy_hcheck added support for AJP/CPING in 22.04, however the following error is encountered when enabling it, which prevents the new feature from operating:
   BalancerMember Health check method CPING not (yet) implemented
This is caused by a incorrect check for AJP support in hc_post_config() that occurs too late, after the configuration syntax has already been marked invalid by the time the "hcmethod=CPING" token is found. The fix is to move the check from hc_post_config() to hc_pre_config().

[Test Plan]

Make a container for testing:

$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test

Type in:

# apt update && apt dist-upgrade -y
# apt install -y apache2

# cat > /etc/apache2/sites-available/httpd-hcheck-ajp.conf << __EOF__

<VirtualHost *:80>
    ServerAdmin <email address hidden>
    DocumentRoot “/var/www/html”
    ServerName myapp.example.com
    ErrorLog “${APACHE_LOG_DIR}/myapp.example.com-error_log"
    CustomLog "${APACHE_LOG_DIR}/myapp.example.com-access_log" common
    <Proxy balancer://myapp>
       BalancerMember ajp://127.0.0.1:8009/myapp route=app-route timeout=300 ping=3
 connectiontimeout=3 hcmethod=CPING
     </Proxy>
 ProxyPass /myapp balancer://myapp stickysession=JSESSIONID
     </VirtualHost>
__EOF__

# a2enmod proxy
# a2enmod proxy_ajp
# a2enmod proxy_hcheck
# a2ensite httpd-hcheck-ajp
# apachectl -t

Example of failed output:
# apachectl -t

AH00526: Syntax error on line 10 of /etc/apache2/sites-enabled/httpd-hcheck-ajp.conf:
BalancerMember Health check method CPING not (yet) implemented
Action ‘-t’ failed.

The Apache error log may have more information.

Example of successful output:
# apachectl -t
Syntax OK

[Where problems could occur]

The patch itself modifies the code of mod_proxy_hcheck, so any new bugs involving that module would be suspect. The patch changes configuration code, so issues cropping up that seem related to module configuration could be suspect. Finally, since the patch modifies C code, issues typical of C code (segfaults, memory leaks, …) would be possible, however since this moves a chunk of code unmodified this seems unlikely.

--------------------------original bug report-----------------------------

We were very excited to see mod_proxy_hcheck support for AJP/CPING land in Ubuntu 22.04, however trying to enable it results in the following error:

> BalancerMember Health check method CPING not (yet) implemented

This unfortunately renders AJP-based health checks, which is a new feature introduced in version 2.4.49 (and thus only available in Jammy), impossible to use.

Luckily we were able to find the code responsible for the misbehavior and propose a fix which has now been accepted upstream.

Upstream bug: https://bz.apache.org/bugzilla/show_bug.cgi?id=66300
Upstream fix: https://svn.apache.org/viewvc?view=revision&revision=1905608

We would like to see this fix backported in Jammy.

[Alessandro-cavalier7 (alessandro-cavalier7)]

We were very excited to see mod_proxy_hcheck support for AJP/CPING (r1887415) land in Ubuntu 22.04; however trying to enable it yields the following error:

   BalancerMember Health check method CPING not (yet) implemented

This was surprising for us since from a cursory look at the code this functionality seems implemented.

A couple of hours of debugging later we concluded that the order of execution of the code added in r1887415 seems to be incorrect: the code checking for AJP availability is executed in hc_post_config() which is unfortunately too late since the configuration syntax has already been marked invalid when the "hcmethod=CPING" token was found.

To fix this we simply moved the code checking for AJP availability in hc_pre_config(). We are running a patched version of http-2.4.52 on a couple of hosts running Ubuntu 22.04 and it seems to be working without issues so far.

[Alessandro-cavalier7 (alessandro-cavalier7)]

Created attachment 38404
mod_proxy_hcheck: move AJP availability detection in hc_pre_config()

[Alessandro-cavalier7 (alessandro-cavalier7)]

Attaching MCVE showing the issue and fix, use "docker buildx build -f <FILENAME> ." to run.

[Alessandro-cavalier7 (alessandro-cavalier7)]

Created attachment 38405
Dockerfile.bug

[Alessandro-cavalier7 (alessandro-cavalier7)]

Created attachment 38406
Dockerfile.fix

[V-jiz-h (v-jiz-h)]

Thanks for the report and the patch. It looks good. I'll do some more testing and apply to trunk and propose for back port to 2.4.x

[Paride Legovini (paride)]

Hello and thanks for this bug report. I did not try to setup a reproducer for this, but the bug has been triaged and fixed upstream, so there's no doubt it's valid.

According to the upstream CHANGES file the fix will be released in apache2 2.4.55, currently not in Lunar and not in Debian, so fixing this in Jammy/Kinetic will require patching the Lunar package first.

I think the upstream patch [1] makes this good SRU material.

[1] https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/proxy/mod_proxy_hcheck.c?r1=1905608&r2=1905607&pathrev=1905608

[Bryce Harrington (bryce)]

The SRU process requires an easy to follow test case be documented with the bug, to allow it to be easily validated. Is that something you could assist us with in writing? You can see an example of what we're looking for from the [Test Case] in this bug report:

  https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1988224

If we can get a good [Test Case] defined for this bug report, it should hopefully help us expedite getting the fix out to folks.

[Sistemi CeSIA (sistemicesia)]

The linked bugzilla issue has 2 files attached with a complete and self-container example of the issue and fix, see

https://bz.apache.org/bugzilla/show_bug.cgi?id=66300#c2
https://bz.apache.org/bugzilla/show_bug.cgi?id=66300#c3
https://bz.apache.org/bugzilla/show_bug.cgi?id=66300#c4

Posting the one showing the issue for posterity:

--- BEGIN DOCKERFILE ---
# syntax=docker/dockerfile:1.3-labs
FROM ubuntu:22.04
RUN apt update && \
    apt install -y make autoconf libtool-bin gcc libaprutil1-dev subversion libpcre3-dev vim patch
RUN svn checkout http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x httpd-2.4.x && \
    cd httpd-2.4.x/ && \
    svn co http://svn.apache.org/repos/asf/apr/apr/trunk srclib/apr
RUN cd httpd-2.4.x/ && \
    ./buildconf && \
    ./configure --prefix=/opt/httpd-2.4.x && \
    make -j`nproc` && \
    make install
RUN /opt/httpd-2.4.x/bin/apachectl -t
COPY <<EOF /opt/httpd-2.4.x/conf/extra/httpd-hcheck-ajp.conf

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_hcheck_module modules/mod_proxy_hcheck.so

<VirtualHost *:80>
    ServerAdmin <email address hidden>
    DocumentRoot "/opt/httpd-2.4.x/docs/myapp.example.com"
    ServerName myapp.example.com
    ErrorLog "logs/myapp.example.com-error_log"
    CustomLog "logs/myapp.example.com-access_log" common

    <Proxy balancer://myapp>
        BalancerMember ajp://127.0.0.1:8009/myapp route=app-route timeout=300 ping=3 connectiontimeout=3 hcmethod=CPING
    </Proxy>

    ProxyPass /myapp balancer://myapp stickysession=JSESSIONID

</VirtualHost>
EOF
RUN echo 'Include conf/extra/httpd-hcheck-ajp.conf' >> /opt/httpd-2.4.x/conf/httpd.conf
RUN /opt/httpd-2.4.x/bin/apachectl -t
--- END DOCKERFILE ---

I can post here a modified version using ubuntu's packages instead of compiling apache2 from source if needed.

[Bryce Harrington (bryce)]

> I can post here a modified version using ubuntu's packages instead of compiling apache2 from source if needed.

Thanks, yes that would help, so we know of specifically what needs modified from stock. It looks like the main thing here is the config for the proxy balancer for the app?

I'm also curious about the inclusion of apr from upstream svn (and libaprutil1-dev which is presumably a build dep). Does the issue still crop up even with apr omitted?

[Sistemi CeSIA (sistemicesia)]

> It looks like the main thing here is the config for the proxy balancer for the app?

Yes, and specifically enabling the AJP health check (hcmethod=CPING) instead of the usual HTTP/TCP health checks (hcmethod=GET and hcmethod=TCP rispectively)

> I'm also curious about the inclusion of apr from upstream svn (and libaprutil1-dev which is presumably a build dep). Does the issue still crop up even with apr omitted?

Building from upstream svn (both httpd and apr) was just a mean to show apache2 maintainers that the issue can be reproduced with the latest bits; the issue is still present with apr installed from binary packages.

[Test Case]
# apt update && apt dist-upgrade -y
# apt install -y apache2
# cat > /etc/apache2/sites-available/httpd-hcheck-ajp.conf << __EOF__
<VirtualHost *:80>
    ServerAdmin <email address hidden>
    DocumentRoot "/var/www/html"
    ServerName myapp.example.com
    ErrorLog "logs/myapp.example.com-error_log"
    CustomLog "logs/myapp.example.com-access_log" common

    <Proxy balancer://myapp>
        BalancerMember ajp://127.0.0.1:8009/myapp route=app-route timeout=300 ping=3 connectiontimeout=3 hcmethod=CPING
    </Proxy>

    ProxyPass /myapp balancer://myapp stickysession=JSESSIONID

</VirtualHost>
__EOF__
# a2enmod proxy
# a2enmod proxy_ajp
# a2enmod proxy_hcheck
# a2ensite httpd-hcheck-ajp
# apachectl -t

Example failed output:

root@ubuntu:~# apachectl -t
AH00526: Syntax error on line 10 of /etc/apache2/sites-enabled/httpd-hcheck-ajp.conf:
BalancerMember Health check method CPING not (yet) implemented
Action '-t' failed.
The Apache error log may have more information.

Example of successful (expected) output:

root@ubuntu:~# apachectl -t
Syntax OK

[Michał Małoszewski (michal-maloszewski99)]

It is in Lunar:
apache2 | 2.4.55-1ubuntu1 | lunar | source, amd64, arm64, armhf, i386, ppc64el, riscv64, s390x

Set Fix Released in Lunar.

[Michał Małoszewski (michal-maloszewski99)]

Taking care of that and preparing patches for Jammy and Kinetic.

[Steve Langasek (vorlon)]

An upload of apache2 to kinetic-proposed has been rejected from the upload queue for the following reason: "there is already an apache2 2.4.54-2ubuntu1.2 in kinetic-security, this needs rebased".

[Steve Langasek (vorlon)]

An upload of apache2 to jammy-proposed has been rejected from the upload queue for the following reason: "there is already an apache2 2.4.52-1ubuntu4.4 in jammy-security, this needs rebased".

[Michał Małoszewski (michal-maloszewski99)]

Thanks, rebased.