Be able to query with multiple terms

[Robert Ancell]

The current snapd interface allows snaps to be queried with one search term, i.e.

GET /2.0/snaps?q=foo

 It would be useful to be able to search for more than one term (i.e. only return results that match both terms)

GET /2.0/snaps?q=foo+bar

[John Lenton (chipaca)]

Adding CPI because currently if I did this I'd get OR semantics afaik

[Robert Ancell (robert-ancell)]

This is causing lots of bad search results to show in GNOME Software (bug 1661983). For example, searching for "World Clock" returns all the hello world results because it matches "World".

You can also see this on the command line:
$ snap find "World Clock"
Name Version Developer Notes Summary
ubuntu-clock-app 3.6+snap3 ubuntucoredev - Ubuntu Clock application for the Unity desktop
qt5-hello-world 1.1 gareth-hulse - Qt5 CLI hello world program.
world 4.0 barry - Where in the world is...?
hello-world 6.3 canonical - The 'hello-world' of snaps
dwarf-fortress 0.43.05 mterry - Dwarf world simulator
hello-world-om26er 0.2 om26er - A great snap
hello-snap-test 0.2 aegloszenit - A hello world app
minetest-luk3yx 0.4.15 luk3yx - luk3yx's unofficial Minetest builds. Credit to the snappy playpen.
hello-node-snap 1.0.2 bhdouglass - A simple hello world command
so-hello-world 0.2 shadowen - the old classic
hello-feroz 2.10 feroz - GNU Hello, the "hello world" snap
hello-will 2.11 will - GNU Hello, the "hello world" snap
hello-mpf 2.12 mpf - GNU hello, the "hello world" snap
kgeography 16.12.1 kde - A Geography Learning Program
hello-atawil 2.10 atawil - GNU Hello, the "hello world" snap
hello-iahmad2 2.10 iahmad2 - GNU Hello, the "hello world" snap
hello-pcoca 2.10 pcoca - GNU Hello, the "hello world" snap
ubuntu-social-kit 2 keshavnrj - Bring social media apps to Ubuntu Desktop
ipfs v0.4.4 elopio - global, versioned, peer-to-peer filesystem
ultimate-media-downloader 6.0 keshavnrj - Ultimate media downloader is a Gui based video audio downloader which support download media from various sites.
pingus-game 0.1 larryprice - Free Lemmings(TM) clone
hello-alculete 2.12 alculete - GNU Hello, the "hello world" snap
hello-jonnyk 2.12 jonnyk - GNU Hello, the "hello world" snap
hello-tcarrondo 2.1 tcarrondo - GNU Hello, the "hello world" snap
hello-daniel 2.10 cumulocity-demelo - GNU Hello, the "hello world" snap
wire 0.4 dalmirdasilva - wiring...
ogre 1.10.1 paroj - Object-Oriented Graphics Rendering Engine
landing-team-tools 0.2 sil2100 - Various tools useful in the ubuntu-touch landing world.
gatertest 1.2 ralsina - GNU Hello, the "hell...

[Gustavo Niemeyer (niemeyer)]

AND semantics is definitely what we want in those cases, and what I expected was happening already.

[James Tait (jamestait)]

Right. So there's a mix of things going on here:

 * There is what I would consider to be an error in the way the Elasticsearch query is being generated. The `default_operator` is being provided to the `Elasticsearch.search()` method call, but it seems that only takes effect in the case of a `query_string` search via the `q` argument, which we're note using. Instead, we should add the `default_operator` property to the `simple_query_string` objects in the JSON request body.
 * Applying that fix gets the AND semantics, and significantly restricts the number of results - possibly too much. As a simple example, searching for "minecraft" (`/api/v1/snaps/search?q=minecraft`) will show results for `minecraft-nsg`, `minecraft-server-jdstrand`, `cuberite`, `konstructs-client` and `minetest-luk3yx`. `minecraft-nsg` and `konstructs-client` are both client packages; `minecraft-server-jdstrand` and `cuberite` are both Minecraft server packages, the latter being complete Free software; `minetest-luk3yx` isn't Minecraft, but a Minecraft-like game that can operate as a network server. However, searching for "minecraft server" (`/api/v1/snaps/search?q=minecraft+server`) only shows `minecraft-server-jdstrand` and `cuberite`. It would be nice to see Minetest appear there, even as the third option. This could be solvable with a mention of the server capabilities in the description, but maybe we could be a bit more forgiving?
 * Way back in the dim and distant past, when CPI was first conceived, we were targeting people using a touch screen on a phone. As such, presenting results that only prefix-matched one of several search terms wasn't much of a problem, as long as more complete matches against more terms resulted in a higher ranking - most relevant results were shown first, and less relevant results being displayed potentially helps with discoverability by browsing. Now we're *also* targeting users typing on a computer keyboard in a CLI - lots of results in a problem in this scenario. There was, at one point, an intention to have "curated" or "promoted" snaps and filter general query results to include only those, which would help - I don't know if that's still the case. Either way, we *still* need to cater to those users on phones.

Now there are a couple of initiatives under way to address the disparity between the phone use case and the CLI use case. In the short term, work is in progress to untangle them, such that we can optimise the CLI use case without having an adverse effect on the phone. Longer term, Snap searches will go to an entirely different service, written from the ground up and designed from day one with Snaps in mind. We'll work this fix or an equivalent in sooner rather than later.

[Colin Watson (cjwatson)]

snapfind currently does this to build the text search query:

  tsquery = search_term.replace(' ', ' & ')
  tsquery = tsquery + ' | ' + tsquery + ':*'

This seems to have a precedence problem. Shouldn't we be going for '(world & clock) | (world:* & clock:*)' rather than 'world & clock | world & clock:*'? My reading of the code is that it will currently produce the latter, which I think is incorrect.

[Colin Watson (cjwatson)]

Please disregard the above: apparently I have a hard time with precedence after midnight, and only doing prefix-matching on the last term is deliberate for the sake of partial phrase matching.

[John Lenton (chipaca)]

Closing as CPI now implements AND semantics of the space-separated terms in a query, and snapd exposes that.

[Robert Ancell (robert-ancell)]

Updated https://github.com/snapcore/snapd/wiki/REST-API to mention the AND semantics.