[FFe] [SRU] build mellon with --enable-diagnostics to ease up SSO debugging

[Dmitrii Shcherbakov]

FFE Section for disco
---------------------
[Rationale]
This change to mod_auth_mellon adds a very useful capability for enabling diagnostics output from the module:
https://github.com/Uninett/mod_auth_mellon/commit/e8579f6387d9841ce619d836110050fb18117753

It is available as of v0.14.0 (present in Cosmic):

git --no-pager tag --contains=e8579f6387d9841ce619d836110050fb18117753
v0.14.0
v0.14.1

This is generally useful for field engineering and operations teams and other users as SAML exchanges are difficult to debug.

[Build Verification]
https://paste.ubuntu.com/p/2kt3BsxJKn/

[Installation]
https://paste.ubuntu.com/p/VcfcgyPHqH/

"MellonDiagnosticsEnable Off" is the default setting and it results in am_diag_open_log returning 1 which does NOT result in an error returned from am_diag_log_init. Also installed a package and verified that setting this to off explicitly or implicitly (default) does not result in errors on startup or page access.

https://git.launchpad.net/ubuntu/+source/libapache2-mod-auth-mellon/tree/auth_mellon_diagnostics.c?h=ubuntu/disco&id=49c8ccfedca2db17d76348573e6daa862e104f6d#n311

int am_diag_log_init(apr_pool_t *pc, apr_pool_t *p, apr_pool_t *pt, server_rec *s)
{
    for ( ; s ; s = s->next) {
        if (!am_diag_open_log(s, p)) {
            return HTTP_INTERNAL_SERVER_ERROR;
        }
    }

// ...
static int am_diag_open_log(server_rec *s, apr_pool_t *p) {
// ...
    if (!(diag_cfg->flags & AM_DIAG_FLAG_ENABLED)) {
        ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s,
                     "mellon diagnostics disabled for %s", server_desc);
        return 1;
// ...

[Upgrades]
No impact

SRU section for cosmic and bionic
---------------------------------
[Impact]
See FFE Rationale above.

[Test Case]
To test

Add the following to /etc/apache2/conf-available/mellon.conf

MellonDiagnosticsFile /var/log/apache2/mellon_diagnostics.log
MellonDiagnosticsEnable On

a2enconf mellon
systemctl reload apache2

After browsing to a location that is mod_auth_mellon enabled (see the keystone-saml-mellon charm) logging from the mellon module including environment variables in the SAML messages will be found in /var/log/apache2/mellon_diagnostics.log.

Regression testing can be done using the keystone-saml-mellon charm's functional tests.
https://github.com/openstack-charmers/charm-keystone-saml-mellon
At the time of this writing the functional tests are not fully automated and still require some manual configuration:
https://github.com/openstack-charmers/charm-keystone-saml-mellon/blob/master/src/README.md#configuration

[Regression Potential]
As mentioned above in the FFE section, "MellonDiagnosticsEnable Off" can be set in the apache configuration to disable diagnostics. This is also the default setting, so regression potential is certainly limited by this.

In particular the cosmic regression potential is much lower than the bionic potential since there is much less involved.

For bionic please see [Discussion] below.

[Discussion]
** cosmic SRU **
For the cosmic SRU this will be a fairly straight forward and trivial update to the package to run configure with "--enable-diagnostics". Cosmic is already at version 0.14.0 which has the diagnostics support.

** bionic SRU **
For the bionic SRU, things are more complicated as bionic is at version 0.13.1 which does not include diagnostics support. What I'd like to do is to update the bionic package to 0.14.0. I know this is not business as usual but I think the regression potential is minimized by updating to 0.14.0 rather than risking any missed code when cherry-picking various patches.

For some analysis regarding updating bionic to 0.14.0, I've analyzed the delta between 0.13.1 and 0.14.0 and I'm seeing mostly bug fixes and 2 new features (1 for diagnostics support, and 1 for MellonSignatureMethod support). Here's the full commit summary between 0.13.1 and 0.14.0:

/tmp/mod_auth_mellon$ git remote -v
origin https://github.com/UNINETT/mod_auth_mellon (fetch)
origin https://github.com/UNINETT/mod_auth_mellon (push)
/tmp/mod_auth_mellon$ git log --no-merges --date-order --pretty=oneline --format=" - [%h] %s" v0.13.1..v0.14.0
 - [29d2872] Bump version to 0.14.0.
 - [21f78ab] Add release notes for version 0.14.0.
 - [262768a] NEWS: Add consistent whitespace between releases.
 - [7bb98cf] Fix config.h.in missing in .tar.gz.
 - [aee068f] Fix typos in the user guide
 - [8abbcf9] Update User Guide on error responses and ADFS issues
 - [9b17e5c] Add MellonSignatureMethod to control signature algorithm
 - [582f283] Log SAML status response information
 - [524d558] convert README to README.md
 - [0851045] Fix consistency, grammar, and usage in user guide
 - [70e8abc] Give clear error if building with diagnostics support on old Apache
 - [15fcbf7] Fix build error on Apache 2.2.
 - [fe8b978] Add example for dual auth support.
 - [f865919] Add clarification on using info vs auth
 - [5927b5c] Fix Mellon user guide typos
 - [86eb344] Fix conditional build of auth_mellon_diagnostics.c
 - [89a3c81] Add NameID discussion to User Guide
 - [93faba4] Update log msg for Invalid Destination and Invalid Audience to show both the expected and received values.
 - [de853e1] Add user_guide to distribution, use AC_DEFINE instead of CFLAGS
 - [8d49ab6] Replace ap_log_rerror with AM_LOG_RERROR
 - [e8579f6] Add diagnostic logging
 - [6d2ee84] Track file information
 - [ee97812] Add Mellon User Guide
 - [daa5d1e] If no IdP's are defined explicitly log that fact
 - [119cbdd] modify cache functions to take request_rec parameter instead of server_rec
 - [c291232] Make MellonUser case-insensitive.
 - [2c2e19d] Fix incorrect error check for many `lasso_*`-functions.
 - [5c5ed1d] Fix segmentation fault with POST field without a value.
 - [4c924d9] Fix some log message typos

Of the commits above, those required for diagnostics support include:
 - [582f283] Log SAML status response information
 - [70e8abc] Give clear error if building with diagnostics support on old Apache
 - [86eb344] Fix conditional build of auth_mellon_diagnostics.c
 - [8d49ab6] Replace ap_log_rerror with AM_LOG_RERROR
 - [e8579f6] Add diagnostic logging
 - [6d2ee84] Track file information

And the MellonSignatureMethod support is provided in a single commit:
 - [9b17e5c] Add MellonSignatureMethod to control signature algorithm

MellonSignatureMethod is also controlled by a config option in the apache configuration.

Following is the full commit message for 9b17e5c:
https://github.com/Uninett/mod_auth_mellon/commit/9b17e5c1078a9be90de1e9d03079b34ca4056e96

One upgrade consideration coming from this patch is that it changes the default sha algorithm used by Mellon from rsa-sha1 to rsa-sha256. And as the commit says, this was done because SHA1 is no longer considered safe, SHA256 is now the current recommendation. This would likely be a good update anyway.

[Corey Bryant (corey.bryant)]

@Dmitri, thanks for the patch. I've uploaded this to the disco unapproved queue: https://launchpad.net/ubuntu/disco/+queue?queue_state=1&queue_text=

[Launchpad Janitor (janitor)]

This bug was fixed in the package libapache2-mod-auth-mellon - 0.14.2-1ubuntu1

---------------
libapache2-mod-auth-mellon (0.14.2-1ubuntu1) disco; urgency=medium

  * Build the module with --enable-diagnostics (LP: #1820279)

 -- Dmitrii Shcherbakov <email address hidden> Fri, 22 Mar 2019 22:27:25 +0300

[Corey Bryant (corey.bryant)]

A new version of libapache2-mod-auth-mellon has been uploaded to the cosmic unapproved queue where it is awaiting SRU team review. https://launchpad.net/ubuntu/cosmic/+queue?queue_state=1&queue_text=

[Corey Bryant (corey.bryant)]

A new version of libapache2-mod-auth-mellon has been uploaded to the bionic unapproved queue where it is awaiting SRU team review. https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=

[Brian Murray (brian-murray)]

Hello Dmitrii, or anyone else affected,

Accepted libapache2-mod-auth-mellon into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libapache2-mod-auth-mellon/0.14.0-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Sorry, this is not acceptable for SRU in its current form to bionic. This is a 5kloc diff to the code which has as its justification the enablement of a single debugging feature.

A cherry-pick of the commits specific to diagnostics support may be acceptable under the SRU policy.

[Steve Langasek (vorlon)]

An upload of libapache2-mod-auth-mellon to bionic-proposed has been rejected from the upload queue for the following reason: "https://bugs.launchpad.net/ubuntu/+source/libapache2-mod-auth-mellon/+bug/1820279/comments/6".

[Corey Bryant (corey.bryant)]

@Steve, No problem, thanks for the review.

[Corey Bryant (corey.bryant)]

I'm switching bionic to won't fix for now as it will be non-trivial to get this SRU'd for bionic. If there is high demand for this please state your case in this bug and we can evaluate the efforts required.

[Corey Bryant (corey.bryant)]

Hello Dmitrii, or anyone else affected,

Accepted libapache2-mod-auth-mellon into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Dmitrii, or anyone else affected,

Accepted libapache2-mod-auth-mellon into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hi Dmitrii, if you get a chance, please can you test that this is fixed in -proposed and tag this bug accordingly (ie. verification-done-*, etc)? Thanks!

[Dmitrii Shcherbakov (dmitriis)]

Tested on rocky-proposed:

apt policy libapache2-mod-auth-mellon
libapache2-mod-auth-mellon:
  Installed: 0.14.0-1ubuntu0.2~cloud0
  Candidate: 0.14.0-1ubuntu0.2~cloud0
  Version table:
 *** 0.14.0-1ubuntu0.2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64 Packages
        100 /var/lib/dpkg/status

# grep MellonDiag /etc/apache2/mods-enabled/auth_mellon.conf
MellonDiagnosticsFile /var/log/apache2/mellon_diagnostics.log
MellonDiagnosticsEnable On

sudo tail -f /var/log/apache2/mellon_diagnostics.log

---------------------------------- New Request ---------------------------------
GET - /v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
log_id: (null)
server: scheme=(null) hostname=keystone.maas port=0
pid: 1232172, tid: 140619597444864
unparsed_uri: /v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
uri: /v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
path_info: (null)
filename: proxy:http://localhost:4980/v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
query args: (null)
Request Headers:
  Host: keystone.maas:5000
  User-Agent: openstacksdk/0.26.0 keystoneauth1/3.13.1 python-requests/2.18.4 CPython/3.6.7
  Accept-Encoding: gzip, deflate
  Accept: */*,application/vnd.paos+xml
  Connection: keep-alive
  PAOS: ver="urn:liberty:paos:2003-08";"urn:oasis:names:tc:SAML:2.0:profiles:SSO:ecp"
Mellon Directory Configuration for URL: /v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
  MellonEnable (enable): auth

[Dmitrii Shcherbakov (dmitriis)]

Tested with stein-proposed:

 apt policy apt policy libapache2-mod-auth-mellon

apt:
  Installed: 1.6.11
  Candidate: 1.6.11
  Version table:
 *** 1.6.11 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status

# grep MellonDiag /etc/apache2/mods-enabled/auth_mellon.conf
MellonDiagnosticsFile /var/log/apache2/mellon_diagnostics.log
MellonDiagnosticsEnable On

sudo tail -f /var/log/apache2/mellon_diagnostics.log
---------------------------------- New Request ---------------------------------
GET - /v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
log_id: (null)
server: scheme=(null) hostname=keystone.maas port=0
pid: 1668880, tid: 140134808790784
unparsed_uri: /v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
uri: /v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
path_info: (null)
filename: proxy:http://localhost:4980/v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
query args: (null)
Request Headers:
  Host: keystone.maas:5000
  User-Agent: openstacksdk/0.26.0 keystoneauth1/3.13.1 python-requests/2.18.4 CPython/3.6.7
  Accept-Encoding: gzip, deflate
  Accept: */*,application/vnd.paos+xml
  Connection: keep-alive
  PAOS: ver="urn:liberty:paos:2003-08";"urn:oasis:names:tc:SAML:2.0:profiles:SSO:ecp"
Mellon Directory Configuration for URL: /v3/OS-FEDERATION/identity_providers/samltestid/protocols/saml2/auth
# ...

[Dmitrii Shcherbakov (dmitriis)]

Also tested with cosmic-proposed with the same test case as in #14 with a positive outcome.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libapache2-mod-auth-mellon - 0.14.0-1ubuntu0.2

---------------
libapache2-mod-auth-mellon (0.14.0-1ubuntu0.2) cosmic; urgency=medium

  * d/rules: Build the module with --enable-diagnostics (LP: #1820279)

 -- Corey Bryant <email address hidden> Fri, 29 Mar 2019 16:08:39 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for libapache2-mod-auth-mellon has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

The verification of the Stable Release Update for libapache2-mod-auth-mellon has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package libapache2-mod-auth-mellon - 0.14.0-1ubuntu0.2~cloud0
---------------

 libapache2-mod-auth-mellon (0.14.0-1ubuntu0.2~cloud0) bionic-rocky; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 libapache2-mod-auth-mellon (0.14.0-1ubuntu0.2) cosmic; urgency=medium
 .
   * d/rules: Build the module with --enable-diagnostics (LP: #1820279)
 .
 libapache2-mod-auth-mellon (0.14.0-1ubuntu0.1) cosmic-security; urgency=medium
 .
   * SECURITY UPDATE: Open redirect in logout
     - debian/patches/CVE-2019-3877.patch: fix in
       auth_mellon_util.c.
     - CVE-2019-3877
   * SECURITY UPDATE: Authentication bypass
     - debian/patches/CVE-2019-3878.patch: fix in
       mod_auth_mellon.c.
     - CVE-2019-3878
 .
 libapache2-mod-auth-mellon (0.14.0-1) unstable; urgency=medium
 .
   * New upstream release.