Activity log for bug #1906627

Date Who What changed Old value New value Message
2020-12-03 10:09:24 Rolf Fujino bug added bug
2020-12-03 14:55:10 Cristiano Cumer bug added subscriber Cristiano Cumer
2020-12-03 14:55:20 Launchpad Janitor adcli (Ubuntu): status New Confirmed
2020-12-03 22:22:57 Matthew Ruffell bug added subscriber Matthew Ruffell
2020-12-03 22:23:13 Matthew Ruffell nominated for series Ubuntu Bionic
2020-12-03 22:23:13 Matthew Ruffell bug task added adcli (Ubuntu Bionic)
2020-12-03 22:34:24 Matthew Ruffell tags regression-update
2020-12-03 23:49:25 Matthew Ruffell adcli (Ubuntu): status Confirmed Fix Released
2020-12-03 23:49:29 Matthew Ruffell adcli (Ubuntu Bionic): status New In Progress
2020-12-03 23:49:32 Matthew Ruffell adcli (Ubuntu Bionic): importance Undecided High
2020-12-03 23:49:37 Matthew Ruffell adcli (Ubuntu Bionic): assignee Matthew Ruffell (mruffell)
2020-12-04 01:47:20 Jason Alavaliant bug added subscriber Jason Alavaliant
2020-12-04 04:32:37 Matthew Ruffell bug watch added https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917129
2020-12-04 04:35:45 Matthew Ruffell bug task added cyrus-sasl2 (Ubuntu)
2020-12-04 23:44:23 Matthew Ruffell attachment added Debdiff for adcli on Bionic https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/+attachment/5441133/+files/lp1906627_adcli_bionic.debdiff
2020-12-05 00:21:29 Launchpad Janitor cyrus-sasl2 (Ubuntu): status New Confirmed
2020-12-05 00:21:29 Launchpad Janitor cyrus-sasl2 (Ubuntu Bionic): status New Confirmed
2020-12-05 00:32:01 Matthew Ruffell cyrus-sasl2 (Ubuntu Bionic): status Confirmed In Progress
2020-12-05 00:32:05 Matthew Ruffell cyrus-sasl2 (Ubuntu Bionic): importance Undecided Medium
2020-12-05 00:32:07 Matthew Ruffell cyrus-sasl2 (Ubuntu Bionic): assignee Matthew Ruffell (mruffell)
2020-12-05 00:33:54 Ubuntu Foundations Team Bug Bot tags regression-update patch regression-update
2020-12-05 00:34:00 Ubuntu Foundations Team Bug Bot bug added subscriber Ubuntu Sponsors Team
2020-12-05 01:07:37 Matthew Ruffell cyrus-sasl2 (Ubuntu): status Confirmed Fix Released
2020-12-05 19:48:30 Eric Desrochers bug added subscriber Eric Desrochers
2020-12-06 23:51:50 Matthew Ruffell summary adcli fails, can't contact LDAP server GSS-SPNEGO implementation in cyrus-sasl2 is incompatible with Active Directory, causing recent adcli regression
2020-12-06 23:53:35 Matthew Ruffell description Package: adcli Version: 0.8.2-1ubuntu1 Release: Ubuntu 18.04 LTS When trying to join the domain with this new version of adcli, it gets to the point of 'Using GSS-SPNEGO for SASL bind' and then it will not do anything for 10 minutes. It will then fail, complaining it can't reach the LDAP server. Logs: Dec 03 01:39:50 example001.domain.com realmd[6419]: * Authenticated as user: domain-join-account@DOMAIN.COM Dec 03 01:39:50 example001.domain.com adcli[6459]: GSSAPI client step 1 Dec 03 01:39:50 example001.domain.com realmd[6419]: * Authenticated as user: domain-join-account@DOMAIN.COM Dec 03 01:39:50 example001.domain.com adcli[6459]: GSSAPI client step 1 Dec 03 01:39:50 example001.domain.com realmd[6419]: * Using GSS-SPNEGO for SASL bind Dec 03 01:39:50 example001.domain.com realmd[6419]: * Using GSS-SPNEGO for SASL bind Dec 03 01:39:50 example001.domain.com adcli[6459]: GSSAPI client step 1 Dec 03 01:55:27 example001.domain.com realmd[6419]: ! Couldn't lookup domain short name: Can't contact LDAP server Dec 03 01:55:27 example001.domain.com realmd[6419]: ! Couldn't lookup domain short name: Can't contact LDAP server Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using fully qualified name: example001.domain.com Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using fully qualified name: example001.domain.com Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using domain name: domain.com Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using domain name: domain.com Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using computer account name: EXAMPLE001 Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using computer account name: EXAMPLE001 Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using domain realm: domain.com Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using domain realm: domain.com Dec 03 01:55:27 example001.domain.com realmd[6419]: * Calculated computer account name from fqdn: EXAMPLE001 Dec 03 01:55:27 example001.domain.com realmd[6419]: * Calculated computer account name from fqdn: EXAMPLE001 Dec 03 01:55:27 example001.domain.com realmd[6419]: * With user principal: host/example001.domain.com@DOMAIN.COM Dec 03 01:55:27 example001.domain.com realmd[6419]: * With user principal: host/example001.domain.com@DOMAIN.COM Dec 03 01:55:27 example001.domain.com realmd[6419]: * Generated 120 character computer password Dec 03 01:55:27 example001.domain.com realmd[6419]: * Generated 120 character computer password Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using keytab: FILE:/etc/krb5.keytab Dec 03 01:55:27 example001.domain.com realmd[6419]: * Using keytab: FILE:/etc/krb5.keytab Dec 03 01:55:27 example001.domain.com realmd[6419]: ! Couldn't lookup computer account: EXAMPLE001$: Can't contact LDAP server Dec 03 01:55:27 example001.domain.com realmd[6419]: ! Couldn't lookup computer account: EXAMPLE001$: Can't contact LDAP server Dec 03 01:55:27 example001.domain.com realmd[6419]: adcli: joining domain domain.com failed: Couldn't lookup computer account: EXAMPLE001$: Can't contact LDAP server Dec 03 01:55:27 example001.domain.com realmd[6419]: adcli: joining domain domain.com failed: Couldn't lookup computer account: EXAMPLE001$: Can't contact LDAP server Dec 03 01:55:27 example001.domain.com realmd[6419]: process exited: 6459 Dec 03 01:55:27 example001.domain.com realmd[6419]: ! Failed to join the domain Dec 03 01:55:27 example001.domain.com realmd[6419]: ! Failed to join the domain On the network level, adcli gets to the point of send an ldap query to the domain controller and the domain controller returns an ack tcp packet, but then there is no more traffic between the domain controller and the server except for ntp packets until it fails. The domain controller traffic also shows that it is receiving the ldap query packet from the server but it never sends a reply and there is no log in directory services regarding the query. We also couldn't find anything in procmon regarding this query either. Workaround/Fix: Downgrading the adcli package back to version 0.8.2-1 fixes the issues and domain join works properly again. [Impact] A recent release of adcli 0.8.2-1ubuntu1 to bionic-updates caused a regression for some users when attempting to join a Active Directory realm. adcli introduced a default behaviour change, moving from GSS-API to GSS-SPNEGO as the default channel encryption algorithm. adcli uses the GSS-SPNEGO implementation from libsasl2-modules-gssapi-mit, a part of cyrus-sasl2. The implementation seems to have some compatibility issues with particular configurations of Active Directory on recent Windows Server systems. Particularly, adcli sends a ldap query to the domain controller, which responds with a tcp ack, but never returns a ldap response. The connection just hangs at this point and no more traffic is sent. You can see it on the packet trace below: https://paste.ubuntu.com/p/WRnnRMGBPm/ On Focal, where the implementation of GSS-SPNEGO is working, we see a full exchange, and adcli works as expected: https://paste.ubuntu.com/p/8668pJrr2m/ The fix is to not assume use of confidentiality and integrity modes, and instead use the flags negotiated by GSS-API during the initial handshake, as required by Microsoft's implementation. [Testcase] You will need to set up a Windows Server 2019 system, install and configure Active Directory and enable LDAP extensions and configure LDAPS and import the AD SSL certificate to the Ubuntu client. Create some users in Active Directory. On the Ubuntu client, set up /etc/hosts with the hostname of the Windows Server machine, if your system isn't configured for AD DNS. From there, install adcli 0.8.2-1 from -release. $ sudo apt install adcli Set up a packet trace with tcpdump: $ sudo tcpdump -i any port '(389 or 3268 or 636 or 3269)' Next, join the AD realm using the normal GSS-API: # adcli join --verbose -U Administrator --domain WIN-SB6JAS7PH22.testing.local --domain-controller WIN-SB6JAS7PH22.testing.local --domain-realm TESTING.LOCAL You will be prompted for Administrator's passowrd. The output should look like the below: https://paste.ubuntu.com/p/NWHGQn746D/ Next, enable -proposed, and install adcli 0.8.2-1ubuntu1 which caused the regression. Repeat the above steps. Now you should see the connection hang. https://paste.ubuntu.com/p/WRnnRMGBPm/ Finally, install the fixed cyrus-sasl2 package, which is available from the below ppa: https://launchpad.net/~mruffell/+archive/ubuntu/lp1906627-test $ sudo add-apt-repository ppa:mruffell/lp1906627-test $ sudo apt-get update $ sudo apt install libsasl2-2 libsasl2-modules libsasl2-modules-db libsasl2-modules-gssapi-mit Repeat the steps. GSS-SPNEGO should be working as intended, and you should get output like below: https://paste.ubuntu.com/p/W5cJNGvCsx/ [Where problems could occur] Since we are changing the implementation of GSS-SPNEGO, and cyrus-sasl2 is the library which provides it, we can potentially break any package which depends on libsasl2-modules-gssapi-mit for GSS-SPNEGO. $ apt rdepends libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-mit Reverse Depends: |Suggests: ldap-utils Depends: adcli Conflicts: libsasl2-modules-gssapi-heimdal |Suggests: libsasl2-modules Conflicts: libsasl2-modules-gssapi-heimdal |Recommends: sssd-krb5-common |Suggests: slapd |Suggests: libsasl2-modules |Suggests: ldap-utils |Depends: msktutil Conflicts: libsasl2-modules-gssapi-heimdal |Depends: libapache2-mod-webauthldap Depends: freeipa-server Depends: freeipa-client Depends: adcli Depends: 389-ds-base |Recommends: sssd-krb5-common |Suggests: slapd |Suggests: libsasl2-modules While this SRU makes cyrus-sasl2 work with Microsoft implementations of GSS-SPNEGO, which will be the more common usecase, it may change the behaviour when connecting to a MIT krb5 server with the GSS-SPNEGO protocol, as krb5 assumes use of confidentiality and integrity modes. This shouldn't be a problem as the krb5 implementation signals its intentions by setting the correct flags during handshake, which these patches to cyrus-sasl2 should now parse correctly. [Other Info] The below two commits are needed. The first fixes the problem, the second fixes some unused parameter warnings. commit 816e529043de08f3f9dcc4097380de39478b0b16 Author: Simo Sorce <simo@redhat.com> Date: Thu Feb 16 15:25:56 2017 -0500 Subject: Fix GSS-SPNEGO mechanism's incompatible behavior https://github.com/cyrusimap/cyrus-sasl/commit/816e529043de08f3f9dcc4097380de39478b0b16 commit ed2ad48f242fe16e846a9db552a04fca1a5da45f Author: Simo Sorce <simo@redhat.com> Date: Tue Apr 11 18:31:46 2017 -0400 Subject: Drop unused parameter from gssapi_spnego_ssf() https://github.com/cyrusimap/cyrus-sasl/commit/ed2ad48f242fe16e846a9db552a04fca1a5da45f
2020-12-07 00:20:04 Matthew Ruffell attachment added cyrus-sasl2 debdiff for Bionic https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/+attachment/5441530/+files/lp1906627_cyrus_sasl2_bionic.debdiff
2020-12-07 02:20:25 Matthew Ruffell tags patch regression-update patch regression-update sts-sponsor
2020-12-07 16:04:58 Eric Desrochers bug added subscriber STS Sponsors
2020-12-08 03:44:29 Matthew Ruffell attachment added debdiff for adcli on Bionic https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/+attachment/5441872/+files/lp1906627_adcli_option_one.debdiff
2020-12-08 03:46:21 Matthew Ruffell attachment added debdiff for adcli on Bionic option two https://bugs.launchpad.net/ubuntu/+source/adcli/+bug/1906627/+attachment/5441873/+files/lp1906627_adcli_option_two.debdiff
2020-12-08 14:45:33 Ryan Chase bug added subscriber Ryan Chase
2020-12-08 15:16:10 Jimothy bug added subscriber James
2020-12-09 15:26:31 Robie Basak removed subscriber Ubuntu Sponsors Team
2020-12-09 15:26:41 Robie Basak bug added subscriber Łukasz Zemczak
2020-12-10 17:20:45 Eric Desrochers removed subscriber STS Sponsors
2020-12-10 17:25:09 Łukasz Zemczak description [Impact] A recent release of adcli 0.8.2-1ubuntu1 to bionic-updates caused a regression for some users when attempting to join a Active Directory realm. adcli introduced a default behaviour change, moving from GSS-API to GSS-SPNEGO as the default channel encryption algorithm. adcli uses the GSS-SPNEGO implementation from libsasl2-modules-gssapi-mit, a part of cyrus-sasl2. The implementation seems to have some compatibility issues with particular configurations of Active Directory on recent Windows Server systems. Particularly, adcli sends a ldap query to the domain controller, which responds with a tcp ack, but never returns a ldap response. The connection just hangs at this point and no more traffic is sent. You can see it on the packet trace below: https://paste.ubuntu.com/p/WRnnRMGBPm/ On Focal, where the implementation of GSS-SPNEGO is working, we see a full exchange, and adcli works as expected: https://paste.ubuntu.com/p/8668pJrr2m/ The fix is to not assume use of confidentiality and integrity modes, and instead use the flags negotiated by GSS-API during the initial handshake, as required by Microsoft's implementation. [Testcase] You will need to set up a Windows Server 2019 system, install and configure Active Directory and enable LDAP extensions and configure LDAPS and import the AD SSL certificate to the Ubuntu client. Create some users in Active Directory. On the Ubuntu client, set up /etc/hosts with the hostname of the Windows Server machine, if your system isn't configured for AD DNS. From there, install adcli 0.8.2-1 from -release. $ sudo apt install adcli Set up a packet trace with tcpdump: $ sudo tcpdump -i any port '(389 or 3268 or 636 or 3269)' Next, join the AD realm using the normal GSS-API: # adcli join --verbose -U Administrator --domain WIN-SB6JAS7PH22.testing.local --domain-controller WIN-SB6JAS7PH22.testing.local --domain-realm TESTING.LOCAL You will be prompted for Administrator's passowrd. The output should look like the below: https://paste.ubuntu.com/p/NWHGQn746D/ Next, enable -proposed, and install adcli 0.8.2-1ubuntu1 which caused the regression. Repeat the above steps. Now you should see the connection hang. https://paste.ubuntu.com/p/WRnnRMGBPm/ Finally, install the fixed cyrus-sasl2 package, which is available from the below ppa: https://launchpad.net/~mruffell/+archive/ubuntu/lp1906627-test $ sudo add-apt-repository ppa:mruffell/lp1906627-test $ sudo apt-get update $ sudo apt install libsasl2-2 libsasl2-modules libsasl2-modules-db libsasl2-modules-gssapi-mit Repeat the steps. GSS-SPNEGO should be working as intended, and you should get output like below: https://paste.ubuntu.com/p/W5cJNGvCsx/ [Where problems could occur] Since we are changing the implementation of GSS-SPNEGO, and cyrus-sasl2 is the library which provides it, we can potentially break any package which depends on libsasl2-modules-gssapi-mit for GSS-SPNEGO. $ apt rdepends libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-mit Reverse Depends: |Suggests: ldap-utils Depends: adcli Conflicts: libsasl2-modules-gssapi-heimdal |Suggests: libsasl2-modules Conflicts: libsasl2-modules-gssapi-heimdal |Recommends: sssd-krb5-common |Suggests: slapd |Suggests: libsasl2-modules |Suggests: ldap-utils |Depends: msktutil Conflicts: libsasl2-modules-gssapi-heimdal |Depends: libapache2-mod-webauthldap Depends: freeipa-server Depends: freeipa-client Depends: adcli Depends: 389-ds-base |Recommends: sssd-krb5-common |Suggests: slapd |Suggests: libsasl2-modules While this SRU makes cyrus-sasl2 work with Microsoft implementations of GSS-SPNEGO, which will be the more common usecase, it may change the behaviour when connecting to a MIT krb5 server with the GSS-SPNEGO protocol, as krb5 assumes use of confidentiality and integrity modes. This shouldn't be a problem as the krb5 implementation signals its intentions by setting the correct flags during handshake, which these patches to cyrus-sasl2 should now parse correctly. [Other Info] The below two commits are needed. The first fixes the problem, the second fixes some unused parameter warnings. commit 816e529043de08f3f9dcc4097380de39478b0b16 Author: Simo Sorce <simo@redhat.com> Date: Thu Feb 16 15:25:56 2017 -0500 Subject: Fix GSS-SPNEGO mechanism's incompatible behavior https://github.com/cyrusimap/cyrus-sasl/commit/816e529043de08f3f9dcc4097380de39478b0b16 commit ed2ad48f242fe16e846a9db552a04fca1a5da45f Author: Simo Sorce <simo@redhat.com> Date: Tue Apr 11 18:31:46 2017 -0400 Subject: Drop unused parameter from gssapi_spnego_ssf() https://github.com/cyrusimap/cyrus-sasl/commit/ed2ad48f242fe16e846a9db552a04fca1a5da45f [Impact] A recent release of adcli 0.8.2-1ubuntu1 to bionic-updates caused a regression for some users when attempting to join a Active Directory realm. adcli introduced a default behaviour change, moving from GSS-API to GSS-SPNEGO as the default channel encryption algorithm. adcli uses the GSS-SPNEGO implementation from libsasl2-modules-gssapi-mit, a part of cyrus-sasl2. The implementation seems to have some compatibility issues with particular configurations of Active Directory on recent Windows Server systems. Particularly, adcli sends a ldap query to the domain controller, which responds with a tcp ack, but never returns a ldap response. The connection just hangs at this point and no more traffic is sent. You can see it on the packet trace below: https://paste.ubuntu.com/p/WRnnRMGBPm/ On Focal, where the implementation of GSS-SPNEGO is working, we see a full exchange, and adcli works as expected: https://paste.ubuntu.com/p/8668pJrr2m/ The fix is to not assume use of confidentiality and integrity modes, and instead use the flags negotiated by GSS-API during the initial handshake, as required by Microsoft's implementation. [Testcase] You will need to set up a Windows Server 2019 system, install and configure Active Directory and enable LDAP extensions and configure LDAPS and import the AD SSL certificate to the Ubuntu client. Create some users in Active Directory. On the Ubuntu client, set up /etc/hosts with the hostname of the Windows Server machine, if your system isn't configured for AD DNS. From there, install adcli 0.8.2-1 from -release. $ sudo apt install adcli Set up a packet trace with tcpdump: $ sudo tcpdump -i any port '(389 or 3268 or 636 or 3269)' Next, join the AD realm using the normal GSS-API: # adcli join --verbose -U Administrator --domain WIN-SB6JAS7PH22.testing.local --domain-controller WIN-SB6JAS7PH22.testing.local --domain-realm TESTING.LOCAL You will be prompted for Administrator's passowrd. The output should look like the below: https://paste.ubuntu.com/p/NWHGQn746D/ Next, enable -proposed, and install adcli 0.8.2-1ubuntu1 which caused the regression. Repeat the above steps. Now you should see the connection hang. https://paste.ubuntu.com/p/WRnnRMGBPm/ Finally, install the fixed cyrus-sasl2 package from -proposed https://launchpad.net/~mruffell/+archive/ubuntu/lp1906627-test $ sudo apt-get update $ sudo apt install libsasl2-2 libsasl2-modules libsasl2-modules-db libsasl2-modules-gssapi-mit Repeat the steps. GSS-SPNEGO should be working as intended, and you should get output like below: https://paste.ubuntu.com/p/W5cJNGvCsx/ [Where problems could occur] Since we are changing the implementation of GSS-SPNEGO, and cyrus-sasl2 is the library which provides it, we can potentially break any package which depends on libsasl2-modules-gssapi-mit for GSS-SPNEGO. $ apt rdepends libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-mit Reverse Depends:  |Suggests: ldap-utils   Depends: adcli   Conflicts: libsasl2-modules-gssapi-heimdal  |Suggests: libsasl2-modules   Conflicts: libsasl2-modules-gssapi-heimdal  |Recommends: sssd-krb5-common  |Suggests: slapd  |Suggests: libsasl2-modules  |Suggests: ldap-utils  |Depends: msktutil   Conflicts: libsasl2-modules-gssapi-heimdal  |Depends: libapache2-mod-webauthldap   Depends: freeipa-server   Depends: freeipa-client   Depends: adcli   Depends: 389-ds-base  |Recommends: sssd-krb5-common  |Suggests: slapd  |Suggests: libsasl2-modules While this SRU makes cyrus-sasl2 work with Microsoft implementations of GSS-SPNEGO, which will be the more common usecase, it may change the behaviour when connecting to a MIT krb5 server with the GSS-SPNEGO protocol, as krb5 assumes use of confidentiality and integrity modes. This shouldn't be a problem as the krb5 implementation signals its intentions by setting the correct flags during handshake, which these patches to cyrus-sasl2 should now parse correctly. [Other Info] The below two commits are needed. The first fixes the problem, the second fixes some unused parameter warnings. commit 816e529043de08f3f9dcc4097380de39478b0b16 Author: Simo Sorce <simo@redhat.com> Date: Thu Feb 16 15:25:56 2017 -0500 Subject: Fix GSS-SPNEGO mechanism's incompatible behavior https://github.com/cyrusimap/cyrus-sasl/commit/816e529043de08f3f9dcc4097380de39478b0b16 commit ed2ad48f242fe16e846a9db552a04fca1a5da45f Author: Simo Sorce <simo@redhat.com> Date: Tue Apr 11 18:31:46 2017 -0400 Subject: Drop unused parameter from gssapi_spnego_ssf() https://github.com/cyrusimap/cyrus-sasl/commit/ed2ad48f242fe16e846a9db552a04fca1a5da45f
2020-12-10 17:27:47 Łukasz Zemczak cyrus-sasl2 (Ubuntu Bionic): status In Progress Fix Committed
2020-12-10 17:27:50 Łukasz Zemczak bug added subscriber Ubuntu Stable Release Updates Team
2020-12-10 17:27:52 Łukasz Zemczak bug added subscriber SRU Verification
2020-12-10 17:28:00 Łukasz Zemczak tags patch regression-update sts-sponsor patch regression-update sts-sponsor verification-needed verification-needed-bionic
2020-12-10 23:50:18 Łukasz Zemczak adcli (Ubuntu Bionic): status In Progress Fix Committed
2020-12-11 07:43:06 Oscar Franzen bug added subscriber Oscar Franzen
2020-12-16 03:52:21 Matthew Ruffell tags patch regression-update sts-sponsor verification-needed verification-needed-bionic patch sts-sponsor verification-done-bionic
2021-01-07 11:09:04 Launchpad Janitor adcli (Ubuntu Bionic): status Fix Committed Fix Released
2021-01-07 11:09:10 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2021-01-07 11:19:14 Launchpad Janitor cyrus-sasl2 (Ubuntu Bionic): status Fix Committed Fix Released
2021-02-06 22:43:34 Mathew Hodson tags patch sts-sponsor verification-done-bionic patch regression-update sts-sponsor verification-done-bionic