Properly convert DNS names with '-' characters to valid dbus object paths

[Matthew Ruffell]

[Impact]

It is common that domain names contain the '-' character, as in "test-example.com", and adsys versions 0.9.2 and below cannot parse these correctly, leading to the error:

ERRORgithub.com/ubuntu/adsys/cmd/adsysd/main.go:50 main.run() Error from server: error while updating policy: can't get policies for "test-example.com": failed to retrieve offline state from SSSD: dbus: invalid message: invalid path name

when attempting to run adsys on a system attached to "test-example.com" Active Directory.

Currently, 0.9.2 only changes '.' into '_2e', and this would change all special characters to use their hexadecimal representations, notably '-' becomes '_2d'.

There is plans from Foundations + Desktop to SRU 0.12.0 back to at least Jammy, documented in bug 2020682 which depends on golang 1.20 to be included in the jammy archive, documented in bug 2020658. However, this fixup is required with high priority while the 0.12.0 release is being prepared, and the SRU will hopefully bridge a few weeks between SRU release to release of 0.12.0.

[Testcase]

Start a Windows Server VM, 2022 will be fine, and create an Active Directory with the domain "test-example.com".

Launch a Focal, or Jammy, or Kinetic VM, and use SSSD to join the domain.

Try to enable adsys:

$ sudo apt install adsys
$ adsysctl update
ERRORgithub.com/ubuntu/adsys/cmd/adsysd/main.go:50 main.run() Error from server: error while updating policy: can't get policies for "test-example.com": failed to retrieve offline state from SSSD: dbus: invalid message: invalid path name

There are test packages in the below ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf360012-test

If you install the test package and retry to join the domain, it will succeed.

[Where problems could occur]

We are changing how domain names are being parsed and converted to valid dbus object path names. Domain names can only contain [0-9], [A-Z], [a-z], [.], and [-], so by adding '-' to being processed to its hexadecimal representation of '_2d', there should be limited scope of regressions.

However, if a regression were to occur, then users may not be able to use adsys to apply group policy restrictions, and could run into issues accessing files, shares and networks.

As mentioned in the impact section, this will be a temporary fix to 0.9.2 while 0.12.0 is being prepared to be released into the archive, which contains the full fix and testsuite coverage. This SRU should hopefully be short lived.

[Other Info]

The upstream merge request is:

https://github.com/ubuntu/adsys/pull/498

This was fixed in 0.10.0 by the commit:

commit 5752ba87347d7813dd56bc6a9ec6369ec56e5dc4
Author: Didier Roche <email address hidden>
Date: Tue Nov 15 11:10:51 2022 +0100
Subject: Fix special characters in domain conversion to dbus object path
Link: https://github.com/ubuntu/adsys/commit/5752ba87347d7813dd56bc6a9ec6369ec56e5dc4

Now, there were some additional commits that added testsuite coverage:

commit cd79b3f81441a3d9ab50f11bc8c3b5c7bf722540
Author: Didier Roche <email address hidden>
Date: Tue Nov 15 11:13:03 2022 +0100
Subject: Refresh golden file now that we properly handle the path.
Link: https://github.com/ubuntu/adsys/commit/cd79b3f81441a3d9ab50f11bc8c3b5c7bf722540

commit 4571e39cd724a973270a586d2b18f653f0007de9
Author: Didier Roche <email address hidden>
Date: Tue Nov 15 11:14:35 2022 +0100
Subject: Use a better case to assert on ServerURL() failure being ignored.
Link: https://github.com/ubuntu/adsys/commit/4571e39cd724a973270a586d2b18f653f0007de9

commit fdca6e462c26e1cbecdb8386f43515c1947d423d
Author: Didier Roche <email address hidden>
Date: Tue Nov 15 11:16:21 2022 +0100
Subject: Add a separate case for special characters in domain name.
Link: https://github.com/ubuntu/adsys/commit/fdca6e462c26e1cbecdb8386f43515c1947d423d

These commits are not compatible with 0.9.2 due to testsuite harnesses and frameworks and test data files not being added until 0.10.0, and adding such commits is numerous, and contains too many changes for a SRU. Regrettably, the testsuite commits must be omitted.

[Matthew Ruffell (mruffell)]

Attached is a debdiff for Kinetic. Note the package uses a debian/source/format of "3.0 (native)", and I did not use quilt to prepare the patch.

[Matthew Ruffell (mruffell)]

Attached is a debdiff for Jammy. Note the package uses a debian/source/format of "3.0 (native)", and I did not use quilt to prepare the patch.

[Matthew Ruffell (mruffell)]

Attached is a debdiff for Focal. Note the package uses a debian/source/format of "3.0 (native)", and I did not use quilt to prepare the patch.

[Heitor Alves de Siqueira (halves)]

Hi Matthew,

thank you for the debdiffs! Things look good overall, I'd just like to add attribution for the upstream patches (for Didier Roche). This is something one can easily add when uploading, so don't worry about supplying a new debdiff just for this change.

We've discussed this offline, but I'm going to mention it here as well, so we can track it later:
There's an upload for adsys 0.12 sitting in the Jammy queue, and we'd have to either trample over that or wait for it to release in order to get the fix for this bug. There's some discussion going on over at the ubuntu-release mailing lists about granting adsys an SRU exception [0].
I'm going to reach out to the ones in charge of LP bug 2020682 to see if we can release this bug fix first, as the SRU exception discussion might take some time, still.

[0] https://lists.ubuntu.com/archives/ubuntu-release/2023-June/005650.html

[Heitor Alves de Siqueira (halves)]

I've discussed this internally, and seems like the Desktop team is ok with us pushing this bug fix as an SRU before the new 0.12 release for adsys.

Matthew, I did a minor modification to your changelog entry to add attribution for the upstream patches, but otherwise your debdiffs seem good. Sponsored for F/J/K, thank you!

[Chris Halse Rogers (raof)]

Hello Matthew, or anyone else affected,

Accepted adsys into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/adsys/0.9.2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Matthew, or anyone else affected,

Accepted adsys into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/adsys/0.9.2~22.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Matthew, or anyone else affected,

Accepted adsys into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/adsys/0.9.2~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Fabio Augusto Miranda Martins (fabio.martins)]

I've tested Focal and it works well with adsys from -proposed:

https://pastebin.ubuntu.com/p/6RBcBpZq2T/

And also does Jammy:

https://pastebin.ubuntu.com/p/pqvgSkkmcQ/

I'm having issues with Kinetic. When I install adsys, I'm no longer able to login to the instance, I'm getting the following errors when trying to login:

https://pastebin.ubuntu.com/p/hh9SHB3ZXx/

It works if I purge the adsys package.

[Heitor Alves de Siqueira (halves)]

I see riscv64 on Focal currently fails to build, but looking at the previous adsys versions there that seems to be the case for some time now. Looking at the build logs, it's unlikely to be related to this patch.

I'm a little concerned about the report on Kinetic breaking login with this new version, as that seems like a severe regression. Would dropping this for Kinetic altogether be an option? If I'm reading the Releases page correctly, it's going to be EOL by the end of the month and future releases should already include this fix (Lunar has adsys-0.11, and Mantic has adsys-0.12).

[Robie Basak (racb)]

I think it's OK to skip Kinetic if you don't want to proceed with it.

However, what are the implications of that severe regression for the QA process for other releases? Are there QA gaps that could also affect a release into Jammy, for example? Is it worth investigating the Kinetic failure further - not necessarily for the Kinetic release, but to consider if there are any implications that might cause a regression in Jammy?

[Robie Basak (racb)]

Releasing this to Jammy is blocked on understanding what happened with Kinetic. Not necessarily for the Kinetic release, but I think we should understand what happened first in case it has implications for Jammy.

[Heitor Alves de Siqueira (halves)]

Agreed, I think it's reasonable to track down why this seemed to break Kinetic entirely before releasing to the other stable series. I can see from the logs that adsys seems to be failing due to an internal LDAP error (NT_STATUS_INVALID_PARAMETER), so it might be worth looking into that a bit deeper.

[Łukasz Zemczak (sil2100)]

The version of adsys in the proposed pocket of Kinetic that was purported to fix this bug report has been removed because one or more bugs that were to be fixed by the upload have failed verification and been in this state for more than 10 days.

[Matthew Ruffell (mruffell)]

Hi everyone,

The term "regression" is a slight overstatement for Kinetic, as login fails with the version in -updates as well, due to it not supporting the "-" in the domain name.

With 0.9.2 in kinetic -updates:

ERRORgithub.com/ubuntu/adsys/cmd/adsysd/main.go:50 main.run() Error from server: error while updating policy: can't get policies for "ec2amaz-hg2r0q8.fabio-rg.com": failed to retrieve offline state from SSSD: dbus: invalid message: invalid path name

and then login also fails. So, the package in -proposed does not break anything further, but it doesn't fix everything either.

Now, here is what I found with the package in -proposed:

Jul 20 05:21:35 ip-172-31-55-219 adsysctl[2564546]: level=error msg="Error from server: error while updating policy: can't get policies for \"ip-172-31-55-219\": failed to retrieve the list of GPO (exited with 1): exit status 1
Failed to bind - LDAP client internal error: NT_STATUS_INVALID_PARAMETER
Failed to connect to 'ldap://ec2amaz-hg2r0q8.fabio-rg.com' with backend 'ldap': LDAP client internal error: NT_STATUS_INVALID_PARAMETER
Failed to open session: (1, 'LDAP client internal error: NT_STATUS_INVALID_PARAMETER')
"

> Failed to connect to 'ldap://ec2amaz-hg2r0q8.fabio-rg.com'
> ith backend 'ldap': LDAP client internal error: NT_STATUS_INVALID_PARAMETER

So this is a ldap based error, but we know this already.

I then tried to use other ldap based tools, like ldbsearch:

$ sudo apt install ldb-tools samba-dsdb-modules
$ cd /tmp
$ ll
krb5cc_1930801111_Xu8aKP
$ sudo ldbsearch -H ldap://ec2amaz-hg2r0q8.fabio-rg.com --use-krb5-ccache=/tmp/krb5cc_1930801111_Xu8aKP --debug-stdout --debuglevel 20
...
resolve_lmhosts: Attempting lmhosts lookup for name ec2amaz-hg2r0q8.fabio-rg.com<0x20>
startlmhosts: Can't open lmhosts file /etc/samba/lmhosts. Error was No such file or directory
Starting GENSEC mechanism spnego
Starting GENSEC submechanism gssapi_krb5
cli_credentials(WORKGROUP/root) without realm, cannot use kerberos for this connection ldap/ec2amaz-hg2r0q8.fabio-rg.com
Failed to start GENSEC client mech gssapi_krb5: NT_STATUS_INVALID_PARAMETER
gensec_spnego_create_negTokenInit_step: Failed to setup SPNEGO negTokenInit request
gensec_update_send: spnego[0x5556c039ff10]: subreq: 0x5556c03a0450
gensec_update_done: spnego[0x5556c039ff10]: NT_STATUS_INVALID_PARAMETER tevent_req[0x5556c03a0450/../../auth/gensec/spnego.c:1631]: state[3] error[-7963671676338569203 (0x917B5ACDC000000D)] state[struct gensec_spnego_update_state (0x5556c03a0610)] timer[(nil)] finish[../../auth/gensec/spnego.c:1947]
Failed to bind - LDAP client internal error: NT_STATUS_INVALID_PARAMETER
Failed to connect to 'ldap://ec2amaz-hg2r0q8.fabio-rg.com' with backend 'ldap': LDAP client internal error: NT_STATUS_INVALID_PARAMETER
Failed to connect to ldap://ec2amaz-hg2r0q8.fabio-rg.com - LDAP client internal error: NT_STATUS_INVALID_PARAMETER

Now, if we do this on Jammy, it works fine.

I was trying out different parameters to ldbsearch on kinetic, as you can see the user is incorrect:

WORKGROUP/root

The clue came from:

cannot use kerberos for this connection ldap/ec2amaz-hg2r0q8.fabio-rg.com

I started looking at kerberos credential cache...

[Robie Basak (racb)]

Thank you for the detailed analysis! So to make sure I understand: we think adsys 0.9.2 was broken in Kinetic all along due to some problem at the Kerberos end. You isolated this issue to outside adsys using ldbsearch. This issue with ldbsearch does not occur with Jammy. So you're confident that the problem isn't something that is being introduced by adsys. Is my understanding correct?

If so then there might be implications for a standing SRU exception for adsys but not for this particular bugfix SRU in Jammy, so if you can confirm please then I think this fix can be released to Jammy and to Focal.

[Fabio Augusto Miranda Martins (fabio.martins)]

I've tested this in Lunar and Mantic.

TL;DR: In Lunar I hit the exact same issue as in Kinetic. After installing adsys, I'm no longer able to login using a domain user. On Mantic, it works fine and I'm able to login even after installing adsys. On the other hand, both Lunar and Mantic have the same issue with the kerberos keytab / credential cache missing the service principal for <email address hidden>.

Details:

- Lunar: https://pastebin.ubuntu.com/p/mg7xXzM79N/
- Mantic: https://pastebin.ubuntu.com/p/bfGhnCdkjJ/

[Heitor Alves de Siqueira (halves)]

Hi Robie,

your understanding that things were broken in Kinetic all along is correct, and Fabio's tests demonstrated that this also seems to be broken in newer releases. We're confident that the problem isn't related to this particular SRU, but it does seem like either adsys or SSSD is the trigger for the Kerberos credential cache issue.

I've opened bug 2029489 for us to investigate this further. Considering that login issue is not related to the adsys DNS fix we're targeting here, could we re-consider the 'regression-proposed' status for F/J and release this SRU? Testing on the earlier releases indicates the reported issue has been fixed, and we haven't seen any other regressions besides Kinetic (which was already broken before Matthew's patches).

Cheers,
Heitor

[Launchpad Janitor (janitor)]

This bug was fixed in the package adsys - 0.9.2~20.04.2

---------------
adsys (0.9.2~20.04.2) focal; urgency=medium

  [ Didier Roche ]
  [ Matthew Ruffell ]
  * Fix processing of domain names to correctly parse '-' characters
    when creating valid dbus object paths, enabling domains with
    '-' to work, e.g. "test-example.com". (LP: #2020834)
    - internal/ad/ad.go

 -- Matthew Ruffell <email address hidden> Fri, 26 May 2023 15:57:39 +1200

[Utkarsh Gupta (utkarsh)]

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.