Spotlight search function broken with macOS Ventura and later client

[Dan Smith]

[ Impact ]

This bug causes users to receive no feedback (results or errors) when trying to search a Samba share from macOS Ventura or later with the Finder/Spotlight (OS-native) mechanism. Before Ventura, searching was possible because of Samba's elasticsearch integration. With the affected versions (which we expect will include all future versions from here on out) a user performs a search and just never gets results, nor any positive feedback that something has failed.

The patch to Samba accounts for a difference in the query parameter arrangement in newer macOS versions. Before the patch, Samba would not properly interpret a search query from the newer versions because it was nested in a more complicated structure than before.

This is fixed upstream in https://gitlab.com/samba-team/samba/-/merge_requests/2915 via this bug https://bugzilla.samba.org/show_bug.cgi?id=15299

[ Test Plan ]

To reproduce:

1. Browse to a Samba share
2. Type a search string in the box at the top of the Finder window
3. The default destination is "this mac", so change it to the option provided to search the current network share
4. Observe no results, errors, or other feedback

To confirm the fix:

Same as above, but in step 4, results should be provided.

[ Where problems could occur ]

This feature is macOS specific, so if macOS changes the structure again, then this will need to be fixed again.

[Christian Ehrhardt  (paelzer)]

Thank you for the report, indeed a patch on a plate.

I think this will need a MacOS user with ventura to help us verifying - otherwise the try to fix is in vain.

Once a packager would make this available, would you be able to verify this as part of the SRU [1]?
I'm mark incomplete until you or someone else confirmed this (as it is only actionable then).

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

[Dan Smith (dansmith)]

Yep, sure can.

[Athos Ribeiro (athos-ribeiro)]

Thanks, Dan.

Would you also be able to describe a minimal test case you would go through to verify this bug so we can complete the paperwork described in https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template (and to allow others to verify it as well)?

It could be a simple step-by-step describing what the outcomes are in a buggy scenario and what you would expect to see in a fixed version of the package.

Once you do, please, move the status of this bug back to "New".

[Dan Smith (dansmith)]

Not sure if you're looking for the whole template from me or just a pre/post test case, so use whatever of this you think is worthwhile:

[ Impact ]

This bug causes users to receive no feedback (results or errors) when trying to search a Samba share from macOS Ventura or later with the Finder/Spotlight (OS-native) mechanism. Before Ventura, searching was possible because of Samba's elasticsearch integration. With the affected versions (which we expect will include all future versions from here on out) a user performs a search and just never gets results, nor any positive feedback that something has failed.

The patch to Samba accounts for a difference in the query parameter arrangement in newer macOS versions. Before the patch, Samba would not properly interpret a search query from the newer versions because it was nested in a more complicated structure than before.

[ Test Plan ]

To reproduce:

1. Browse to a Samba share
2. Type a search string in the box at the top of the Finder window
3. The default destination is "this mac", so change it to the option provided to search the current network share
4. Observe no results, errors, or other feedback

To confirm the fix:

Same as above, but in step 4, results should be provided.

[ Where problems could occur ]

The patch is very small and limited to the code that handles searches from (AFAIK) macOS only. I'm not a Samba expert, but I think the blast radius is very small in this case, and since searching from the supported versions of macOS currently doesn't work, it is probably a "can't make it worse" sort of situation.

[ Other Info ]

This is a feature that used to work fine before macOS Ventura, so users of 20.04 and 22.04 like myself had this capability working before the *client* upgrade broke it. I think it's just a bug in Samba's reverse-engineering of the queries they saw at the time, not fully implementing the query DSL semantics such that a completely-legit query in a later version just wasn't handled by Samba.

[Athos Ribeiro (athos-ribeiro)]

> Not sure if you're looking for the whole template from me or just a pre/post test case, so use whatever of this you think is worthwhile:

This is more than I was asking for. Thank you for writing that! These templates usually go in the bug Description, so I am updating it with your SRU paperwork.

Since you've done all the work here so far, would you like to prepare a debdiff (or a merge proposal) for this change? I'd happily guide you through the process (in case you are not familiar with it) and upload the package (sponsor) for you.

Otherwise, Paride already tagged this to the server backlog, so someone on the server team should pick this up soon.

Thanks again :)

[Dan Smith (dansmith)]

I think I'll defer to Paride if that's okay. If it ends up not getting done in a reasonable timeframe, I'll circle back and do that.

Thanks!

[Mitchell Dzurick (mitchdz)]

Hi Dan,

Could you please test this package out and let us know if it works for you?

https://launchpad.net/~mitchdz/+archive/ubuntu/samba-spotlight-fix-lp2046994

[Dan Smith (dansmith)]

Yep, looks good. I added the PPA and upgraded my locally-patched packages to the manifest shown below and nuked all existing connections to make sure they got reestablished to the new binary. My Ventura machine can (still) successfully search the SMB share after the upgrade.

Thanks very much!

$ dpkg -l | grep samba
ii python3-samba 2:4.15.13+dfsg-0ubuntu1.6~jammy2 amd64 Python 3 bindings for Samba
ii samba 2:4.15.13+dfsg-0ubuntu1.6~jammy2 amd64 SMB/CIFS file, print, and login server for Unix
ii samba-common 2:4.15.13+dfsg-0ubuntu1.6~jammy2 all common files used by both the Samba server and client
ii samba-common-bin 2:4.15.13+dfsg-0ubuntu1.6~jammy2 amd64 Samba common files used by both the server and the client
ii samba-dev:amd64 2:4.15.13+dfsg-0ubuntu1.6~jammy2 amd64 tools for extending Samba
ii samba-dsdb-modules:amd64 2:4.15.13+dfsg-0ubuntu1.6~jammy2 amd64 Samba Directory Services Database
ii samba-libs:amd64 2:4.15.13+dfsg-0ubuntu1.6~jammy2 amd64 Samba core libraries
ii samba-testsuite 2:4.15.13+dfsg-0ubuntu1.6~jammy2 amd64 test suite from Samba
ii samba-vfs-modules:amd64 2:4.15.13+dfsg-0ubuntu1.6~jammy2 amd64 Samba Virtual FileSystem plugins

[Mitchell Dzurick (mitchdz)]

Hi Dan, I'm going through with the process to getting the changes in Jammy, but I don't have a mac to test for Focal. Once I get the patch for Focal would you be willing to test that out for me? No worries if not.

[Dan Smith (dansmith)]

Sorry, I don't have a focal machine set up anymore to test this with. It's a little more than just a focal machine and samba, you need elasticsearch and fscrawler to digest some information in order to confirm. When I first saw that it was fixed upstream, I tested the patch on focal (patched and rebuilt the latest ubuntu package samba-4.15.13+dfsg with the patch) and it worked fine. Then I upgraded that machine to jammy in order to confirm that (a) it wasn't in the jammy package and (b) that the fix worked for jammy as well, but that means I don't have a focal machine to test with.

So, my feeling is that it was tested in focal and it's pretty benign, so it'd be safe to just assume it's good, but I know that may be riskier than you want. Personally, as long as it makes it into jammy I'm happy :D

[Mitchell Dzurick (mitchdz)]

Thanks a lot Dan! I'll proceed with Jammy then. Once the Jammy package is in the -proposed archive, would you be willing to test that package out as well? It should be identical to the one in my PPA, but it's always nice to double check :)

[Dan Smith (dansmith)]

Yep, sure, glad to.

[Dan Bungert (dbungert)]

Uploading for Focal and Jammy. For purposes of clarity for the SRU for Focal, note that this build is yet untested.

[Timo Aaltonen (tjaalton)]

Hello Dan, or anyone else affected,

Accepted samba into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/samba/2:4.15.13+dfsg-0ubuntu1.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Dan, or anyone else affected,

Accepted samba into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/samba/2:4.15.13+dfsg-0ubuntu0.20.04.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dan Smith (dansmith)]

ii samba 2:4.15.13+dfsg-0ubuntu1.6 amd64 SMB/CIFS file, print, and login server for Unix

Installed from jammy-proposed and confirmed that the fix is working like my locally-built ones and the one from the PPA tested earlier using a Mac running Ventura.

Thanks!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted samba (2:4.15.13+dfsg-0ubuntu1.6) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.48.2-0ubuntu1 (amd64)
samba/2:4.15.13+dfsg-0ubuntu1.6 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#samba

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted samba (2:4.15.13+dfsg-0ubuntu0.20.04.8) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1.2 (amd64, ppc64el)
vlc/3.0.9.2-1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#samba

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Mitchell Dzurick (mitchdz)]

Reran gvfs/vlc autopkgtest and they now pass. The failure logs looked like flaky issues (dpkg being locked up for vlc and network not being up for gvfs)

[Andreas Hasenack (ahasenack)]

I retried the failed jammy dep8 tests.

[Andreas Hasenack (ahasenack)]

The jammy tests are still failing. gvfs is known to be flaky, but this is a bit too much. I triggered a migration-reference/0 run.

[Mitchell Dzurick (mitchdz)]

I see gvfs is still failing even with migration-reference/0.

[Mitchell Dzurick (mitchdz)]

Can we still call 9 autopkgtest run failures and 2 migration-refrence/0 failures flaky? I see gvfs failed in the past, but not this much.

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for samba has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:4.15.13+dfsg-0ubuntu1.6

---------------
samba (2:4.15.13+dfsg-0ubuntu1.6) jammy; urgency=medium

  * d/p/lp2046994-spotlight-doesnt-work-with-latest-macos-ventura.patch: fix
    spotlight search function on macos ventura (LP: #2046994).

 -- Mitchell Dzurick <email address hidden> Fri, 05 Jan 2024 14:23:01 -0700

[Andreas Hasenack (ahasenack)]

Launchpad notified me there was an OOPS before when I ran the release script for this package in jammy:

Launchpad encountered an internal error during the following operation: copying package samba. It was logged with id OOPS-506a4e55f3c1962e07e46d85360f5ffe. Sorry for the inconvenience.

I therefore ran it again:
--- Releasing samba ---
Proposed: 2:4.15.13+dfsg-0ubuntu1.6
Security: 2:4.15.13+dfsg-0ubuntu1.5
Updates: 2:4.15.13+dfsg-0ubuntu1.5
Copied to jammy-updates
LP: #2046994 was not commented on