after upgrade 19.04 to 19.10, apache serves php code

[Christopher Barrington-Leigh]

Apache2 has been working as a server on localhost (as well as on an fixed IP when my laptop is at work) for years/many version.
I just upgraded to 19.10 from 19.04, and without receiving any warning during the upgrade, my index.php was rendered as plain text on the server! This was a security breach.

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: apache2 (not installed)
ProcVersionSignature: Ubuntu 5.3.0-19.20-generic 5.3.1
Uname: Linux 5.3.0-19-generic x86_64
Apache2ConfdDirListing: False
Apache2Modules: Error: command ['pkexec', '/usr/sbin/apachectl', '-D DUMP_MODULES'] failed with exit code 127: Error accessing /usr/sbin/apachectl: No such file or directory
ApportVersion: 2.20.11-0ubuntu8.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Fri Nov 1 09:52:04 2019
EcryptfsInUse: Yes
InstallationDate: Installed on 2019-01-24 (280 days ago)
InstallationMedia: Ubuntu 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.3)
SourcePackage: apache2
UpgradeStatus: Upgraded to eoan on 2019-10-31 (0 days ago)
error.log: Error: [Errno 13] Permission denied: '/var/log/apache2/error.log'
modified.conffile..etc.apache2.apache2.conf: [modified]
modified.conffile..etc.apache2.mods-available.dir.conf: [modified]
modified.conffile..etc.apache2.sites-available.000-default.conf: [modified]
mtime.conffile..etc.apache2.apache2.conf: 2019-01-27T12:48:55.101659
mtime.conffile..etc.apache2.mods-available.dir.conf: 2019-11-01T09:30:55.155540
mtime.conffile..etc.apache2.sites-available.000-default.conf: 2019-01-27T12:49:32.812651

[Christopher Barrington-Leigh (cpbl)]

It seems that something like the following my close the security hole:

 `sudo a2enmod php7.3`

 `systemctl restart apache2`

In which case, this should have been taken care of during the upgrade? Or when php5 was deleted??

[Andreas Hasenack (ahasenack)]

Thanks for filing this bug in Ubuntu.

Could you please share the following logs:
- /var/log/dist-upgrade/*
- /var/log/dpkg.log
- /var/log/apache2/error.log

I'm trying to determine exactly from which version of each package did you upgrade, and if there were any errors during that process.

You also said you were using php5? Ubuntu Disco (19.04) has php 7.2. Xenial has php 7.0, so from which ubuntu release were you coming from?

[Christopher Barrington-Leigh (cpbl)]

Can I safely just zip up the entire dist-upgrade? It has some non- *.log files and an entire subdirectory.
 And Can I easily share these privately somehow? I'd rather not post the apache error log broadly.

[Christian Ehrhardt  (paelzer)]

Hi Christopher,
at least the answer for sharing privately would most likely be "send a mail".
You'll see mail addresses for me, Andreas and anyone else if you clock on their username above the comments here on launchpad.

The path should only contain things triggered by the upgrade, not sure what other content you have there in sub-directories. It is the usual answer [1] to the question where to look for details of the upgrade.

[1]: https://askubuntu.com/questions/130969/can-i-look-at-the-upgrade-log-after-a-distribution-upgrade

[Christopher Barrington-Leigh (cpbl)]

This is kind of useless, but I do not now know why I mentioned php5. I may have seen mention of it but I'm not sure. I hope that you can see everything more definitively in the logs. It looks to me like it was php7.2 that is mostly recently mentioned in the logs, but you can read them better than I: I have sent emails to paelzer and ahasenack with the files Andreas asked for.
Thanks

[Christian Ehrhardt  (paelzer)]

I saw in your dist-upgrade/apt-term.log file (first column are line numbers):
7245 Setting up libapache2-mod-php7.3 (7.3.11-0ubuntu0.19.10.1) ...
7247 Creating config file /etc/php/7.3/apache2/php.ini with new version
7248 libapache2-mod-php7.3: php7.2 module already enabled, not enabling PHP 7.3
and later
7860 Removing libapache2-mod-php7.2 (7.2.24-0ubuntu0.19.04.1) ...
7861 Module php7.2 disabled.
7862 apache2_invoke prerm: Disable module php7.2

It might be an ordering issue needing 7.2 removed before 7.3 is installed to avoid this?

[Christian Ehrhardt  (paelzer)]

I have myself created a 19.04 system with:
- libapache2-mod-php + libapache2-mod-php7.2
  - as expected on configure does this automatically
  - This is from /var/lib/dpkg/info/libapache2-mod-php7.2.postinst
     php_invoke enmod 7.2 apache2 $mod
- Created the most trivial PHP (index.php with phpinfo in /var/www/html/index.php
  - Upgraded from 19.04 to 19.10
  - Confirmed to hit the described isse

# ll /etc/apache2/mods-available/*php* /etc/apache2/mods-enabled/*php*
ls: cannot access '/etc/apache2/mods-enabled/*php*': No such file or directory
-rw-r--r-- 1 root root 855 Oct 24 11:38 /etc/apache2/mods-available/php7.3.conf
-rw-r--r-- 1 root root 102 Oct 24 11:38 /etc/apache2/mods-available/php7.3.load

And the server now serving the PHP code.

[Christian Ehrhardt  (paelzer)]

I'd expect the two versions of libapache2-mod-php7 to have a conflicts statement to enforce ordering.
Checking the same on 7.0 (xenial) -> 7.2 (bionic)

I'd expect this bug would have come up before if it was broken there as well, so that should server as a good comparison.

There it is the same:

$ grep 'mod-php' /var/log/dist-upgrade/apt-term.log
Selecting previously unselected package libapache2-mod-php7.2.
Preparing to unpack .../libapache2-mod-php7.2_7.2.24-0ubuntu0.18.04.1_amd64.deb ...
Unpacking libapache2-mod-php7.2 (7.2.24-0ubuntu0.18.04.1) ...
Preparing to unpack .../libapache2-mod-php_1%3a7.2+60ubuntu1_all.deb ...
Unpacking libapache2-mod-php (1:7.2+60ubuntu1) over (1:7.0+35ubuntu6.1) ...
Setting up libapache2-mod-php7.2 (7.2.24-0ubuntu0.18.04.1) ...
>> libapache2-mod-php7.2: php7.0 module already enabled, not enabling PHP 7.2
Setting up libapache2-mod-php (1:7.2+60ubuntu1) ...
Removing libapache2-mod-php7.0 (7.0.33-0ubuntu0.16.04.7) ...
Module php7.0 disabled.^M
apache2_invoke prerm: Disable module php7.0^M
Purging configuration files for libapache2-mod-php7.0 (7.0.33-0ubuntu0.16.04.7) ...

And the server is serving the PHP code now ...

It seems to affect all major version bumps (unless I'm missing a detail and doing things wrong)

[Christian Ehrhardt  (paelzer)]

After seeing that it even affects 16.04 -> 18.04 I was wondering and looked for prior reports.
I found [1][2] and many similar, but no bug report.

[1]: https://www.digitalocean.com/community/tutorials/how-to-upgrade-to-ubuntu-18-04
[2]: https://ubuntuforums.org/showthread.php?t=2393823

[Christian Ehrhardt  (paelzer)]

Just like the manual a2enmod the following resolves the situation as well as now the old mod-php7.2 is "out-of-the-way".

$ apt install --reinstall libapache2-mod-php7.3

[Christian Ehrhardt  (paelzer)]

Hmm [1][2] which I'd consider to be needed here leave me somewhat confused.
I'm leaving my comfort zone here, IIRC breaks should always be versioned and versioning the dependency here makes no sense.
OTOH a deconfigure would be enough so conflicts seems kind of hard for this.
And finally both provide virtual "libapache2-mod-php" which would again be more like "Conflicts"

For a trial run before discussing with more people I gave it a try with the harder style (Conflicts) to see if it would enforce ordering correctly.

PPA [3] holds a test build with conflicts, once built later on I'll give it a test if it works.
If it does I'd want to discuss it with the Team if I'm overlooking something and maybe pass it to Bryce for coordinating PHP uploads and considerations for PHP in general.

MP [4] holds the changes I have pushed to that PPA so that we can discuss about it.

[1]: https://www.debian.org/doc/debian-policy/ch-relationships.html#packages-which-break-other-packages-breaks
[2]: https://www.debian.org/doc/debian-policy/ch-relationships.html#conflicting-binary-packages-conflicts
[3]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1850933-mod-php-ordering-on-install
[4]: https://code.launchpad.net/~paelzer/ubuntu/+source/php7.3/+git/php7.3/+merge/375381

[Christian Ehrhardt  (paelzer)]

Testing on the PPA confirmed that this would change the ordering but not trigger the disable of mod-php7.2 as I expected.
Here the ordering from the log:
$ grep "mod-php7" apt.log
Removing libapache2-mod-php7.2 (7.2.24-0ubuntu0.19.04.1) ...
Package apache2 is not configured yet. Will defer actions by package libapache2-mod-php7.2.
Selecting previously unselected package libapache2-mod-php7.3.
Preparing to unpack .../0-libapache2-mod-php7.3_7.3.11-0ubuntu0.19.10.2~ppa3_amd64.deb ...
Unpacking libapache2-mod-php7.3 (7.3.11-0ubuntu0.19.10.2~ppa3) ...
Setting up libapache2-mod-php7.3 (7.3.11-0ubuntu0.19.10.2~ppa3) ...
libapache2-mod-php7.3: php7.2 module already enabled, not enabling PHP 7.3

It was removed first, but not deconfigured I guess.

It is even worse, as php7.2 is still configured (in /etc) but no more installed (the .so file).
Therefore the apache2 restart fails with a proper message about it.
One might say this is nice as it is the safer approach but there must be something better.

I confirmed my next assumption, only a "purge" action would disable the 7.2 module.

# Automatically added by dh_apache2/UNDECLARED
if [ "$1" = "purge" ] ; then
        if php_enable; then
                if [ -e /usr/share/apache2/apache2-maintscript-helper ] ; then
                        . /usr/share/apache2/apache2-maintscript-helper

                        for conf in php7.2 ; do
                                apache2_invoke dismod $conf || exit 1
                        done
                fi
        fi
fi
# End automatically added section

This should affect Debian upgrades e,g, oldstable 7.0 -> stable 7.3 as well.
We might even want to discuss it there if we'd want to go a joint path or if there are drawbacks.

It seems the solution to this isn't too complex, maybe just another postinst snippet, but the impact suggests that there on't be an SRU for it. OTOH we should clearly get a proper hold of this before 20.04 is finished.

But for now let me raise it with the team first.

[Bryce Harrington (bryce)]

Offhand, wondering if we can take advantage of the fact that Ubuntu provides one single php version per release. IOW, when a new php version is installed, configurations should be pointing to that version; this should work both ways - upgrading from php7.3 to php7.4, as well as downgrading from php7.3 to php7.2.

Like Christian said, this may not be SRUable but would want fixed for 20.04 and forward.

Debian differs from Ubuntu here in that they try to support multiple php versions co-installed. So, it's unlikely this fix would be upstreamable.

[Andreas Hasenack (ahasenack)]

Still in our queue.

[Simon Déziel (sdeziel)]

This is not 19.04->19.10 specific as no later than yesterday it affected one of my client. I've reported the 16.04->18.04 bug against php-defaults as it's the provider of mod_php, see LP: #1865218

[Christian Ehrhardt  (paelzer)]

The MP at [1] still is suggested as a solution to this issue going forward.
As Bryce coordinates the php uploads I'll leave that to him thou.

At least the overall migration is done, so maybe this can be a little fix-follow up now.

[1]: https://code.launchpad.net/~paelzer/ubuntu/+source/php7.3/+git/php7.3/+merge/375381

[Bryce Harrington (bryce)]

Fix has been landed for focal's php7.4:

https://code.launchpad.net/~bryce/ubuntu/+source/php7.4/+git/php7.4/+merge/381283