Apache won't start, in version 2.4.10 (trusty-backports) update

[nilandenterprise]

on Mon Dec 10 08:18:32 2018 apache2 was updated to (trusty-backports) 2.4.10-1ubuntu1.1~ubuntu14.04.2 and now apache will not start.

the error is:apache2: Syntax error on line 140 of /etc/apache2/apache2.conf: Syntax error on line 2 of /etc/apache2/mods-enabled/authz_svn.load: Cannot load /usr/lib/apache2/modules/mod_authz_svn.so into server: /usr/lib/apache2/modules/mod_authz_svn.so: undefined symbol: ap_hook_force_authn
Action 'configtest' failed.

The log of the update is:
Commit Log for Mon Dec 10 08:18:32 2018
Upgraded the following packages:
apache2 to 2.4.10-1ubuntu1.1~ubuntu14.04.2
apache2-bin (2.4.7-1ubuntu4.20) to 2.4.10-1ubuntu1.1~ubuntu14.04.2
apache2-data (2.4.7-1ubuntu4.20) to 2.4.10-1ubuntu1.1~ubuntu14.04.2
apache2-doc (2.4.7-1ubuntu4.20) to 2.4.10-1ubuntu1.1~ubuntu14.04.2
apache2-mpm-prefork (2.4.7-1ubuntu4.20) to 2.4.10-1ubuntu1.1~ubuntu14.04.2
apache2-suexec-custom (2.4.7-1ubuntu4.20) to 2.4.10-1ubuntu1.1~ubuntu14.04.2
apache2-utils (2.4.7-1ubuntu4.20) to 2.4.10-1ubuntu1.1~ubuntu14.04.2

Clearing the tick for trusty-backports does not change the update and force version to 2.4.7-1ubuntu4.20 causes broken packages.
i.e. apache2-bin (2.4.7-1ubuntu4.20) breaks apache2 (2.4.7-1ubuntu4.20)
Marking only apache2 (2.4.7-1ubuntu4.20) force version does not revert version?
sudo apt-get install apache2=2.4.7-1ubuntu4.20 does not work

***********
Also /usr/lib/apache2/modules/httpd.exp seems to now be an older file (Aug 31 2016) while the previous file was Apr 18 2018, but loading this file does not work either?
the differences are:
- (line 39) ap_regcomp_get_default_cflags
- (line 40) ap_regcomp_set_default_cflags
- (line 41) ap_regcomp_default_cflag_by_name
+ (line 43) ap_regname
- (line 280) ap_get_basic_auth_components
- (line 368) ap_hook_force_authn
- (line 369) ap_hook_get_force_authn
- (line 370) ap_run_force_authn
- (line 380) ap_some_authn_required
- (line 414) ap_scan_http_field_content
- (line 415) ap_scan_http_token
- (line 416) ap_scan_vchar_obstext
- (line 484) ap_cstr_casecmp
- (line 485) ap_cstr_casecmpn
+ (line 518) ap_hook_suspend_connection
+ (line 519) ap_hook_get_suspend_connection
+ (line 520) ap_run_suspend_connection
+ (line 521) ap_hook_resume_connection
+ (line 522) ap_hook_get_resume_connection
+ (line 523) ap_run_resume_connection
************

So how do I get my server to work again?lsb_release -rd
Description: Ubuntu 14.04.5 LTS
Release: 14.04
apt-cache policy apache2
apache2:
  Installed: 2.4.10-1ubuntu1.1~ubuntu14.04.2
  Candidate: 2.4.10-1ubuntu1.1~ubuntu14.04.2
  Version table:
 *** 2.4.10-1ubuntu1.1~ubuntu14.04.2 0
        100 /var/lib/dpkg/status
     2.4.7-1ubuntu4.20 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     2.4.7-1ubuntu4 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[Christian Ehrhardt  (paelzer)]

Hi,
the recent update was only a security fix [1] that seems totally unrelated.

The conf file in your error /etc/apache2/mods-enabled/authz_svn.load is actually from
libapache2-mod-svn out of src:subversion.
All it does is loading /usr/lib/apache2/modules/mod_authz_svn.so and that fails in your case with:
  Cannot load /usr/lib/apache2/modules/mod_authz_svn.so into server:
  /usr/lib/apache2/modules/mod_authz_svn.so: undefined symbol: ap_hook_force_authn

First I tried the non-backport versions of
apache2 2.4.7-1ubuntu4.20
libapache2-mod-svn 1.8.8-1ubuntu3.3
They work fine (I ensured with a2enmod authz_svn that it is loaded.

Maybe the backports-apache is not binary compatible with the plugins built for the actual apache2 in the archive?
I upgraded to the version in trusty-backports and can confirm the issue:
apache2: Syntax error on line 140 of /etc/apache2/apache2.conf: Syntax error on line 2 of /etc/apache2/mods-enabled/authz_svn.load: Cannot load /usr/lib/apache2/modules/mod_authz_svn.so into server: /usr/lib/apache2/modules/mod_authz_svn.so: undefined symbol: ap_hook_force_authn
Action 'configtest' failed.

Usually the resolution would be to rebuild subversion against the newer apache, but while it might help -backports this would break the "actual" apache in main.

I found that the version in the main archive is fixed or better modified to do all that.
See [2] for that change

Since then the apache2 in main has that new API and deprecated the old inseucre one.
Any later rebuild to subversion will have made it pick up that.
That would have been [4] shortly after.
The apache2 in backports most likely would need that change as well to get backports and main archive matching again.

I can say that the patch would somewhat apply to the version in backports, but have not enugh subject matter expertise to be sure.
$ patch --dry-run -p1 < /tmp/CVE-2015-3185.patch
checking file include/http_request.h
Hunk #2 succeeded at 541 with fuzz 1.
Hunk #3 succeeded at 596 (offset 2 lines).
checking file server/request.c

I'm afraid the apache in trusty-backports is broken (as you reported - thanks for the report BTW), not by the last upload but by a version incompatibility.
There could be more plugins that won't load if they got rebuild and use the new API/ABI.
Due to the nature of the change that mostly will be auth plugins.

Someone would need to prep an upload for that in Backports [4] for that.
Sorry I currently don't have the cycles to do so, but maybe the analysis helps backporters to do it more easily.

For the time being, I installed all other libapache2-mod-auth* and it seems only the subversion plugin is affected for now. So if you don't rely on that, maybe just remove that for now?

[1]: https://git.launchpad.net/ubuntu/+source/apache2/commit/?id=21979d8ee350ab3df0d24558229be4ce19300cf7
[2]: https://git.launchpad.net/ubuntu/+source/apache2/commit/?id=efd270510e6ed37564d375c950b5365fc7929c3e
[3]: https://launchpad.net/ubuntu/+source/subversion/1.8.8-1ubuntu3.1
[4]: https://help.ubuntu.com/community/UbuntuBackports

[Christian Ehrhardt  (paelzer)]

prio low due to "only" affecting -backports

[nilandenterprise (nilands55s)]

I was not able to downgrade (force version) in synaptic apache2, though selected it did nothing. So I had to remove:
apache2
apache2-mpm-prefork
libapache2-mod-php5
virtualmin-lamp-stack

I could not reinstall as I got these errors:
apache2:
  Depends: apache2-bin (=2.4.7-1ubuntu4.20) but 2.4.10-1ubuntu1.1~ubuntu14.04.2 is to be installed
  Depends: apache2-data (=2.4.7-1ubuntu4.20) but 2.4.10-1ubuntu1.1~ubuntu14.04.2 is to be installed
downgrade (force version)

I was able to downgrade (force version), apache2-bin, and apache2-data. Then I was able to reinstall, apache2, apache2-mpm-prefork, libapache2-mod-php5, virtualmin-lamp-stack.
Apache2 started right up after undoing the previous dpkg config done by the backport Apache2 2.4.10-1ubuntu1.1~ubuntu14.04.2 version during the reinstall.