Apache update to 2.4.18-2ubuntu3.10 stopped service but did not start it back

[Cicatrice]

Hello,

The latest release of Apache 2 package for Xenial has stopped Apache service but did not restart it after update.
This affected several of our servers during unattended update.

$ apt-cache policy apache2
apache2:
  Installed: 2.4.18-2ubuntu3.10
  Candidate: 2.4.18-2ubuntu3.10
  Version table:
 *** 2.4.18-2ubuntu3.10 500
        500 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.4.18-2ubuntu3 500
        500 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

$ lsb_release -rd
Description: Ubuntu 16.04.5 LTS
Release: 16.04

== /var/log/apache2/error.log ==
[Fri Apr 05 06:25:04.417533 2019] [mpm_event:notice] [pid 3154:tid 140296233985920] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Fri Apr 05 06:25:04.417543 2019] [core:notice] [pid 3154:tid 140296233985920] AH00094: Command line: '/usr/sbin/apache2'
[Fri Apr 05 06:33:34.765284 2019] [mpm_event:notice] [pid 3154:tid 140296233985920] AH00491: caught SIGTERM, shutting down
[Fri Apr 05 10:09:54.670424 2019] [mpm_event:notice] [pid 22945:tid 140627168151424] AH00489: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Fri Apr 05 10:09:54.670498 2019] [core:notice] [pid 22945:tid 140627168151424] AH00094: Command line: '/usr/sbin/apache2'

== /var/log/apache2/dpkg.log ==
2019-04-05 06:33:34 upgrade apache2:amd64 2.4.18-2ubuntu3.9 2.4.18-2ubuntu3.10
2019-04-05 06:33:34 status half-configured apache2:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:35 status unpacked apache2:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:35 status half-installed apache2:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:35 status triggers-pending man-db:amd64 2.7.5-1
2019-04-05 06:33:35 status triggers-pending ufw:all 0.35-0ubuntu2
2019-04-05 06:33:35 status triggers-pending systemd:amd64 229-4ubuntu21.16
2019-04-05 06:33:35 status triggers-pending ureadahead:amd64 0.100.0-19
2019-04-05 06:33:35 status half-installed apache2:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:35 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:35 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:35 upgrade apache2-bin:amd64 2.4.18-2ubuntu3.9 2.4.18-2ubuntu3.10
2019-04-05 06:33:35 status half-configured apache2-bin:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:35 status unpacked apache2-bin:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:35 status half-installed apache2-bin:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status half-installed apache2-bin:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status unpacked apache2-bin:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status unpacked apache2-bin:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 upgrade apache2-utils:amd64 2.4.18-2ubuntu3.9 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status half-configured apache2-utils:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status unpacked apache2-utils:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status half-installed apache2-utils:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status half-installed apache2-utils:amd64 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status unpacked apache2-utils:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status unpacked apache2-utils:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 upgrade apache2-data:all 2.4.18-2ubuntu3.9 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status half-configured apache2-data:all 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status unpacked apache2-data:all 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status half-installed apache2-data:all 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status half-installed apache2-data:all 2.4.18-2ubuntu3.9
2019-04-05 06:33:36 status unpacked apache2-data:all 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status unpacked apache2-data:all 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 trigproc man-db:amd64 2.7.5-1 <none>
2019-04-05 06:33:36 status half-configured man-db:amd64 2.7.5-1
2019-04-05 06:33:36 status installed man-db:amd64 2.7.5-1
2019-04-05 06:33:36 trigproc ufw:all 0.35-0ubuntu2 <none>
2019-04-05 06:33:36 status half-configured ufw:all 0.35-0ubuntu2
2019-04-05 06:33:36 status installed ufw:all 0.35-0ubuntu2
2019-04-05 06:33:36 trigproc systemd:amd64 229-4ubuntu21.16 <none>
2019-04-05 06:33:36 status half-configured systemd:amd64 229-4ubuntu21.16
2019-04-05 06:33:36 status installed systemd:amd64 229-4ubuntu21.16
2019-04-05 06:33:36 trigproc ureadahead:amd64 0.100.0-19 <none>
2019-04-05 06:33:36 status half-configured ureadahead:amd64 0.100.0-19
2019-04-05 06:33:36 status installed ureadahead:amd64 0.100.0-19
2019-04-05 06:33:36 startup packages configure
2019-04-05 06:33:36 configure apache2-bin:amd64 2.4.18-2ubuntu3.10 <none>
2019-04-05 06:33:36 status unpacked apache2-bin:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status half-configured apache2-bin:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status installed apache2-bin:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 configure apache2-utils:amd64 2.4.18-2ubuntu3.10 <none>
2019-04-05 06:33:36 status unpacked apache2-utils:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status half-configured apache2-utils:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status installed apache2-utils:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 configure apache2-data:all 2.4.18-2ubuntu3.10 <none>
2019-04-05 06:33:36 status unpacked apache2-data:all 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status half-configured apache2-data:all 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status installed apache2-data:all 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 configure apache2:amd64 2.4.18-2ubuntu3.10 <none>
2019-04-05 06:33:36 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
[... more than 100 times the same lines ...]
2019-04-05 06:33:37 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:37 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:37 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:37 status half-configured apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:37 status installed apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:37 startup packages configure

[Marc Deslauriers (mdeslaur)]

That's odd. Could you please also paste the result of "grep apache /var/log/syslog"? Thanks.

[Cicatrice (cicatrice)]

Here is the result :

# grep apache /var/log/syslog
# grep -i apache /var/log/syslog.1
Apr 5 06:25:03 0d47a8f6bd573173d systemd[1]: Reloading LSB: Apache2 web server.
Apr 5 06:25:03 0d47a8f6bd573173d apache2[26744]: * Reloading Apache httpd web server apache2
Apr 5 06:25:03 0d47a8f6bd573173d apache2[26744]: *
Apr 5 06:25:03 0d47a8f6bd573173d systemd[1]: Reloaded LSB: Apache2 web server.

[Cicatrice (cicatrice)]

Sorry, wrong machine; here is the correct file:
# grep -i apache /var/log/syslog
Apr 5 06:33:34 0616675e8cc176244 systemd[1]: Stopping LSB: Apache2 web server...
Apr 5 06:33:34 0616675e8cc176244 apache2[20777]: * Stopping Apache httpd web server apache2
Apr 5 06:33:35 0616675e8cc176244 apache2[20777]: *
Apr 5 06:33:35 0616675e8cc176244 systemd[1]: Stopped LSB: Apache2 web server.
Apr 5 10:09:54 0616675e8cc176244 systemd[1]: Starting LSB: Apache2 web server...
Apr 5 10:09:54 0616675e8cc176244 apache2[22929]: * Starting Apache httpd web server apache2
Apr 5 10:09:54 0616675e8cc176244 apache2[22929]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
Apr 5 10:09:55 0616675e8cc176244 apache2[22929]: *
Apr 5 10:09:55 0616675e8cc176244 systemd[1]: Started LSB: Apache2 web server.

[Cicatrice (cicatrice)]

# journalctl -u apache2
-- Logs begin at Fri 2019-03-29 14:29:08 UTC, end at Fri 2019-04-05 12:34:50 UTC. --
[...]
Apr 04 06:25:04 0616675e8cc176244 systemd[1]: Reloading LSB: Apache2 web server.
Apr 04 06:25:04 0616675e8cc176244 apache2[15672]: * Reloading Apache httpd web server apache2
Apr 04 06:25:04 0616675e8cc176244 apache2[15672]: *
Apr 04 06:25:04 0616675e8cc176244 systemd[1]: Reloaded LSB: Apache2 web server.
Apr 05 06:25:04 0616675e8cc176244 systemd[1]: Reloading LSB: Apache2 web server.
Apr 05 06:25:04 0616675e8cc176244 apache2[20161]: * Reloading Apache httpd web server apache2
Apr 05 06:25:04 0616675e8cc176244 apache2[20161]: *
Apr 05 06:25:04 0616675e8cc176244 systemd[1]: Reloaded LSB: Apache2 web server.
Apr 05 06:33:34 0616675e8cc176244 systemd[1]: Stopping LSB: Apache2 web server...
Apr 05 06:33:34 0616675e8cc176244 apache2[20777]: * Stopping Apache httpd web server apache2
Apr 05 06:33:35 0616675e8cc176244 apache2[20777]: *
Apr 05 06:33:35 0616675e8cc176244 systemd[1]: Stopped LSB: Apache2 web server.
Apr 05 10:09:54 0616675e8cc176244 systemd[1]: Starting LSB: Apache2 web server...
Apr 05 10:09:54 0616675e8cc176244 apache2[22929]: * Starting Apache httpd web server apache2
Apr 05 10:09:54 0616675e8cc176244 apache2[22929]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
Apr 05 10:09:55 0616675e8cc176244 apache2[22929]: *
Apr 05 10:09:55 0616675e8cc176244 systemd[1]: Started LSB: Apache2 web server.

[Christian Ehrhardt  (paelzer)]

Hi,
thanks for the log
This seems interesting at first
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message

this seemed to me like:
1. had a valid server config
2. start the server/service
3. modify the configuration (broken ServerName)
4. upgrade triggers restart of the server which picks up the new config
   Due to that it fails to start on the upgrade.

But I think that is a red herring as this is not an error that makes it fail to start.
Especially as it then says "started" afterwards.

Can you check if the error of not being restarted is reproducible be re-installing the package like:
  $ sudo apt install --reinstall apache2
If it is reproducible - just to be sure - please check and fix your config in regard to the hostname. Retry it again.

If with a a fixed hostname you still have the issue that the command to re-install apache does not restart it then you could please in another shell open up journal like
 $ journalctl -f

And then re-install it again checking status before and after.
  $ sudo systemctl status apache2
  $ sudo apt install --reinstall apache2
  $ sudo systemctl status apache2

Please report the output of both consoles then here, as with the logs as is I can't find the issue that you are facing.

[Christian Ehrhardt  (paelzer)]

FYI hints on the config issue:
https://unix.stackexchange.com/questions/154811/apache-2-4-wont-reload-any-problem-with-my-configuration/
https://unix.stackexchange.com/questions/155150/where-in-apache-2-do-you-set-the-servername-directive-globally

[Cicatrice (cicatrice)]

Hello,

we tried the following command:
$ sudo apt install --reinstall apache2

Several times on affected machines, and we never had the problem again (even without fixing the Apache configuration).

The issue seems to happend only for the update from 2.4.18-2ubuntu3.9 to 2.4.18-2ubuntu3.10

[Robie Basak (racb)]

I tried to reproduce this by installing 2.4.18-2ubuntu3.9 in a container and then upgrading up to 2.4.18-2ubuntu3.10, but it restarted after upgrade correctly.

Your report of:

2019-04-05 06:33:36 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
2019-04-05 06:33:36 status unpacked apache2:amd64 2.4.18-2ubuntu3.10
[... more than 100 times the same lines ...]
2019-04-05 06:33:37 status unpacked apache2:amd64 2.4.18-2ubuntu3.10

...seems suspicious to me.

However unfortunately since I cannot reproduce I don't think this bug can make any progress. I also don't see any other reports of this problem, which suggests to me that it's caused by something specific to your configuration.

We do appreciate the report though - in aggregate these are helpful to find major problems.

For now, this bug will have to remain Incomplete since there's nothing that Ubuntu developers can do to identify a bug at this stage. If you can find steps to reproduce the problem, then please provide them and change the bug status back to New. Or if others affected also find this report then please also let us know.

[Launchpad Janitor (janitor)]

[Expired for apache2 (Ubuntu) because there has been no activity for 60 days.]