SSSD internal DNS resolver is broken when using Cisco Anyconnect VPN client (inotify issue)

[Franck Iaropoli]

Hi everyone,
I am facing an issue with SSSD internal DNS resolver (I was able to reproduce this issue with SSSD versions 2.2.3 coming with Ubuntu 20.04 and version 1.16.1 coming with Ubuntu 18.04) when I am using Cisco Anyconnect VPN client.
SSSD is not detecting the new DNS servers setting up by Cisco Anyconnect client and keeps using previous ones.
DNS is managed with systemd-resolved and the /etc/resolv.conf file is a symlink to /run/systemd/resolve/resolv.conf file (I am not using the internal DNS stub resolver).
When Cisco Anyconnect client connects to VPN the /etc/resolv.conf symlink is renamed to /etc/resolv.conf.vpnbackup and a regular file /etc/resolve.conf is created with DNS servers to use while on VPN.
When Cisco Anyconnect client disconnects from VPN the /etc/resolv.conf.vpnbackup is moved back to /etc/resolv.conf.
I have checked this with inotifywait (I only kept interesting parts):

/etc/ MOVED_FROM resolv.conf
/etc/ MOVED_TO resolv.conf.vpnbackup
/etc/ CREATE resolv.conf
/etc/ OPEN resolv.conf
/etc/ ATTRIB resolv.conf
/etc/ MODIFY resolv.conf
/etc/ CLOSE_WRITE,CLOSE resolv.conf
...
/etc/ MOVED_FROM resolv.conf.vpnbackup
/etc/ MOVED_TO resolv.conf

I can workaround this issue by changing the way SSSD detects DNS changes (stop using inotify and poll the /etc/resolv.conf file every 5 seconds) with option try_inotify set to false but I guess this can impact performance (even though I don't think this should be a big impact).

The SSSD team told me that my issue is the same issue as this one https://github.com/SSSD/sssd/issues/1031
Newer SSSD versions have already been fixed.

Will it be possible to backport: patches
- https://github.com/SSSD/sssd/commit/0c5711f9bae1cb46d4cd3fbe5d86d8688087be13 to version 2.2.3 (Ubuntu 20.04)
- https://github.com/SSSD/sssd/commit/758b99590a8e1f69b4487fad9cf343525797e05f to version 1.16.1 (Ubuntu 18.04)

Thanks for your help :)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Paride Legovini (paride)]

Hello Franck, thanks for this bug report and for pointing to the upstream patches. I prepared a PPA with Focal's sssd rebuilt with [1] applied (slightly adapted):

  https://launchpad.net/~paride/+archive/ubuntu/sssd-lp1958391

Would you be able to verify that it actually fixes this bug?

If it works we'll be able to begin the Stable Release Update process [2], which will require a more formal verification process, but first let's quickly check where we are. Thanks!

[1] https://github.com/SSSD/sssd/commit/0c5711f9bae1cb46d4cd3fbe5d86d8688087be13
[2] https://wiki.ubuntu.com/StableReleaseUpdates

[Franck Iaropoli (franck-iaropoli-arm)]

Hi Paride,

Thanks for your help :)

I have setup the PPA and installed your updated packages:

~$ apt list --installed sssd*
Listing... Done
sssd-ad-common/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-ad/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-common/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-dbus/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed]
sssd-ipa/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-krb5-common/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-krb5/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-ldap/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-proxy/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-tools/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed]
sssd/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]

I connected to VPN and SSSD still seems unable to detect resolv.conf change :

~# ls -al /etc/resolv.conf*
-rw-r--r-- 1 root root 115 Jan 20 13:06 /etc/resolv.conf
lrwxrwxrwx 1 root root 34 Dec 8 17:30 /etc/resolv.conf.vpnbackup -> ../run/systemd/resolve/resolv.conf

# tail -f /var/log/sssd/sssd.log | egrep -i resolv

(Thu Jan 20 13:10:37 2022) [sssd] [process_dir_event] (0x4000): inotify name: .#resolv.confazayDh
(Thu Jan 20 13:10:37 2022) [sssd] [process_dir_event] (0x0400): Not interested in .#resolv.confazayDh
(Thu Jan 20 13:10:37 2022) [sssd] [process_dir_event] (0x4000): inotify name: .#stub-resolv.confoBQ85k
(Thu Jan 20 13:10:37 2022) [sssd] [process_dir_event] (0x0400): Not interested in .#stub-resolv.confoBQ85k

[Franck Iaropoli (franck-iaropoli-arm)]

Hi Paride,
I re-opened SSSD bug report (https://github.com/SSSD/sssd/issues/5959) and provide more details.
It seems to be a different issue that is not yet fixed.

[Paride Legovini (paride)]

Thanks for the follow-up, let's see what upstream has to say.

In that bug report you asked about how to try a newer sssd on Focal [1]. By chance I have a PPA ready with sssd 2.6.1-1ubuntu4 from Jammy compiled on Focal with no changes:

  https://launchpad.net/~paride/+archive/ubuntu/sssd-lp1958392

This may help verifying if the issue is still there.

[1] https://github.com/SSSD/sssd/issues/5959#issuecomment-1016222688

[Franck Iaropoli (franck-iaropoli-arm)]

Hey Paride, sorry for the late reply (crazy day :D).
I am having difficulties to use your PPA:

E: The repository 'http://ppa.launchpad.net/paride/sssd-lp1958392/ubuntu focal Release' does not have a Release file.

Is this because focal is missing from the list of supported Ubuntu versions ?

I asked the SSSD team if there was anything else I could do to help them but they seem to have all information they need https://github.com/SSSD/sssd/issues/5959#issuecomment-1020237062

Still happy to test your ppa though :)

[Paride Legovini (paride)]

The PPA should work now, however given the upstream triage I think we don't really expect the Jammy version to fix this bug, as no upstream version fixes it.

I'm marking this report as Triaged, but at this point we'll wait for upstream to deliver a fix, which hopefully we'll be able to cherry-pick and release as an Ubuntu update.

Again, thanks for reporting this bug, testing, and for the upstream bug report.

[Franck Iaropoli (franck-iaropoli-arm)]

Hi Paride,
Yes I am still experiencing the same issue with sssd packages coming from your PPA. Thanks for making them available though :)
Let's wait for a fix from the SSSD team. I will keep you posted.