Search URL is long and hard to paste

[Jonathan Lange]

If you search for bugs within a project, you get a URL that looks like this:
    https://bugs.edge.launchpad.net/launchpad-bazaar/+bugs?field.searchtext=cloud&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package=

This is a very long URL that's hard to read, and hard to paste on IRC. It would be great if the URL just included the search data that I had explicitly specified. e.g.
    https://bugs.edge.launchpad.net/launchpad-bazaar/+bugs?field.searchtext=cloud

One way to do this would be to use javascript to doctor the location after the page has loaded.

[Eleanor Berger (intellectronica)]

For bonus points, we can try to get rid of the zopey 'field' prefix

[Jonathan Lange (jml)]

Tom and I talked about this a while ago on IRC. The trick is to have some javascript update the location after the page has loaded so that the URL contains only the fields that actually have data.

The 'field.' thing is probably a separate bug.

[Karl Fogel (kfogel)]

micahg and I had a long conversation about this in IRC.

Summary is: even though most of the fields in that URL just represent defaults, they still hold information. If Launchpad's defaults were to change, the meaning of the URL wouldn't change -- unless we shortened the URL, thus depending on Launchpad to know what the defaults should be: then if Launchpad's defaults changed, the meaning of the query would change too.

I'm not sure whether that means the proposed fix is a good or bad idea. However, it does mean that this bug might be alleviated by a fix to bug #317136 ("Launchpad should offer a 'Get tinyURL link' option for each page"), which has its own rather intricate discussion.

Here's the conversation between micahg and me today:

-----------------------------------------------

  <micahg> kfogel: what do you think of that idea of bookmarked
             queries?

  <micahg> then you can use post for everything

  <micahg> in the app at least

  <micahg> and leave $_GET for the people who want to pull data

  <kfogel> micahg: can you explain in more detail? I'm not sure I
             understand.

  <kfogel> micahg: I mean, this is more about the URLs themselves --
             making them more palatable when pasted into emails or other
             forums.

  <micahg> kfogel: that would make the url smaller
             https://bugs.launchpad.net/?bookmarkq=1589654

  <kfogel> micahg: AH, I see what you mean.

  <kfogel> micahg: I think implementation-wise that would have a
             greater impact on the rest of Launchpad (because now we
             have to have the DB remember all those bookmarks), and it
             would make the meaning of the URL more obscure (== less
             hackable).

  <kfogel> micahg: IOW, I think jml's suggestion in
             https://bugs.edge.launchpad.net/malone/+bug/325982/comments/2
             is a cleaner fix.

  <micahg> kfogel: but it wouldn't shorten it much

  <kfogel> micahg: well, the length would now be proportional to the
             complexity of the query, sure.

  <kfogel> micahg: all the unspecified fields would be left off.

  <micahg> yes, but most queries do have a lot ofs option

  <kfogel> micahg: I think the idea behind the bug is that when the
             specified value is the same as the default, you can leave
             it off the URL.

  <kfogel> micahg: but you do have a point -- what if the defaults
             change? Should the meaning of the query then change?

  <micahg> because the url has the defaults right now

  <kfogel> micahg: yeah, the state is carried in the URL, not in
             Launchpad -- even if Launchpad's search query defaults
             change, the URL will still mean the same thing.

  <micahg> right, that's what I meant by not shortening it too much

  <kfogel> micahg: this ties into another bug we had about making a
             dedicated tinyurl service, which got swamped by discussion
             of why it's a bad idea IIRC. Let me see if I can dig it
             up.

  <micahg> kfogel: it seems like if people could "send" queries to
     ...

[Eleanor Berger (intellectronica)]

2009/9/9 Karl Fogel <email address hidden>:
> micahg and I had a long conversation about this in IRC.
>
> Summary is: even though most of the fields in that URL just represent
> defaults, they still hold information.  If Launchpad's defaults were to
> change, the meaning of the URL wouldn't change -- unless we shortened
> the URL, thus depending on Launchpad to know what the defaults should
> be: then if Launchpad's defaults changed, the meaning of the query would
> change too.
>
> I'm not sure whether that means the proposed fix is a good or bad idea.
> However, it does mean that this bug might be alleviated by a fix to bug
> #317136 ("Launchpad should offer a 'Get tinyURL link' option for each
> page"), which has its own rather intricate discussion.

There are two other things to do, which are related to this problem.

1. Get rid of the 'field.' prefix for form fields.
2. Make it possible to save searches in Launchpad itself. If we do
that, people don't have to rely on keeping search URLs for referring
back to searches, and then we have to worry less about the defaults
changing for absent fields.

Finally, the problem with defaults is only manifested in some fields -
enumeration fields mostly. Fields that are empty can safely be dropped
without worrying that the semantics will change later on.

[Karl Fogel (kfogel)]

Tom, I think your suggestion (2) above is either the same as bug #317136 or is a subset of it. Re (1): getting rid of the "field" prefix would help, and not showing empty fields would help. But still, using the original URL in this report as an example, how many fields are actually empty? 5 out of 15. Here's the URL divided into parts:

  https://bugs.edge.launchpad.net/launchpad-bazaar/+bugs
  ?field.searchtext=cloud
  &orderby=-importance
  &search=Search
  &field.status%3Alist=NEW
  &field.status%3Alist=INCOMPLETE_WITH_RESPONSE
  &field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE
  &field.status%3Alist=CONFIRMED
  &field.status%3Alist=TRIAGED
  &field.status%3Alist=INPROGRESS
  &field.status%3Alist=FIXCOMMITTED
  &field.assignee=
  &field.bug_reporter=
  &field.omit_dupes=on
  &field.has_patch=
  &field.has_no_package=

So 15 fields total -- 10 with values, only 5 without. If we removed those 5, the URL would still be quite long:

  https://bugs.edge.launchpad.net/launchpad-bazaar/+bugs?field.searchtext=cloud&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED

Hence my emphasis on bug bug #317136 :-).

[Deryck Hodge (deryck)]

I think it's a mistake to bring in the tiny url discussion here, or to worry about other fixes. Stripping "field" and saving searches in LP should be tracked in other bugs, if they're not already (sorry haven't looked yet). I'm not criticizing the ideas or the discussion, just pointing out that were drifting away from the primary request here.

I think it's fairly easy to do a JavaScript fix, either before submitting the form or on the URL afterwards to make the URL shorter, and we could do this now. It's still a valuable fix, even if it has to be updated after future changes to default search params.

[Eleanor Berger (intellectronica)]

2009/9/11 Deryck Hodge <email address hidden>:

> I think it's fairly easy to do a JavaScript fix, either before
> submitting the form or on the URL afterwards to make the URL shorter,
> and we could do this now.  It's still a valuable fix, even if it has to
> be updated after future changes to default search params.

The Javascript fix is easy to do indeed. The problem, if i understand
correctly, is not that the code will have to be updated if the form
changes, but that saved URLs will become obsolete if the form changes.
In the past we always tried to make it so saved URLs retain their
semantics even when we change the form and people are relying on that
to save searches. If we removed enumerated fields that are set to
their default value, we will never be able to change the default value
without rendering search URLs with these fields omitted incorrect.

[Jonathan Lange (jml)]

An interesting conundrum!

Some thoughts:
  - permanent URLs are nice

  - it might be helpful to distinguish between a search URL giving correct results and a search URL giving predictable results.

  - For any given search, there are two possible intents: search for exactly this, or search for what I've specified within the defaults. We don't know which intent any given user has on any given search.

  - doing an advanced search merely for a particular text string gives a URL without enumerated fields, e.g. https://bugs.edge.launchpad.net/malone/+bugs?field.searchtext=foo&orderby=-importance&search=Search&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_supervisor=&field.bug_commenter=&field.subscriber=&field.omit_dupes.used=&field.omit_dupes=on&field.has_patch.used=&field.has_cve.used=&field.tag=&field.tags_combinator=ANY

  - AIUI, the current defaults for status search correspond to "open bugs". If we changed the default, would it be changing the "open" bit (e.g to "open and recently closed") or would it be to change what we mean by "open" (e.g. "fix committed" is no longer open)? If the first, keeping the enums matters more. If the latter, then maybe it doesn't matter quite so much.

  - There are open bugs and compelling arguments for removing some of our current statuses. Surely doing this would break existing search URLs? Surely this is an insufficient reason to keep our current statuses?

  - If we omitted the fields that were set to default values, we could also have add 'permalink' to the search results page. This would allow us to serve both kinds of search intentions.

jml

[Eleanor Berger (intellectronica)]

2009/9/14 Jonathan Lange <email address hidden>:
>  - If we omitted the fields that were set to default values, we could
> also have add 'permalink' to the search results page. This would allow
> us to serve both kinds of search intentions.

Yes, I think that'a a terrific idea!

[Matthew Paul Thomas (mpt)]

It seems there are three main approaches to shortening the canonical URLs.

1. Remove fields with default values. (75% reduction in length for the original example URL.)

2. Shorten the parameter names and their possible values, starting with those that are repeated most often. For example, "status%3Alist=NEW&status%3Alist=INCOMPLETE_WITH_RESPONSE&status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&status%3Alist=CONFIRMED&status%3Alist=TRIAGED&status%3Alist=INPROGRESS&status%3Alist=FIXCOMMITTED" could be shortened to "s=N&s=IWR&s=IWOR&s=C&s=T&s=IP&s=FC". (36% reduction in length for the original example URL, shortening only the bug statuses.)

3. Remove the "field." prefixes, which I reported as bug 87229. (18% reduction for the original example URL.)

After that changes start getting more incremental. One example is making beta-testing entirely cookie-based rather than URL-based, so post-redirect URLs would no longer ever contain "edge." (though that would fix several other bugs too).

There may be relevant implementation ideas in the equivalent bug report for Bugzilla <https://bugzilla.mozilla.org/show_bug.cgi?id=15809>, though it seems they only ever partially fixed it.

[Martin Pitt (pitti)]

> Summary is: even though most of the fields in that URL just represent defaults, they still hold information. If Launchpad's defaults were to change, the meaning of the URL wouldn't change

While that's true, I think it is mostly an academic argument. Bug search results are inherently subject to change -- after all, that's their entire point! I think it is not only a bad side effect, but even a good thing that changed defaults are taken into consideration. Just take the example of the recently introduced "Opinion" bug state -- with an old hardcoded URL which explicitly enumerates all the bug states (even if the requestor didn't explicitly specify them), there would be no way how we could control the default behaviour of the new bug state. Similarly, adding internal hidden objects like "search=Search" will only require you to not change this field, ever. If it wasn't part of the URL, then you could change the implementation without breaking existing URLs out there.

TBH I don't like all the proposed workarounds with tinyurl. Not only would that inordinately spam their service, but giving proper LP URLs is so much clearer for the recipient, as well as me looking in my browser. tinyurl/DEADBEEF doesn't carry any information, while http://bugs.launchpad.net/apport/+bugs?orderby=status is quite clear what I get.

[Martin Pitt (pitti)]

(Oh, and a big "YES" for having this fixed -- a lot of people spend lots of time shortening URLs manually for pasting into emails or bookmarking, etc.)

[Paul Sladen (sladen)]

Example copied over from duplicate. Since I only touched/modified one field, that is the only field that should be present, eg:

   http://bugs.launchpad.net/ubuntu-font-family/+bugs?field.tag=uff-greek

Karl (comment #3) notes that the defaults might change; in this case I think they should and quite freely (and I do not see that as a problem). At the moment an "Advanced search" immediately defaults to a subset of the bug space, so the first thing is to tick the remaining Closed/Released/Expired... ideally these should be ticked by default—if a Simple search had found the answer, I would /not/ be resorting to the Advanced search screen.

Each keypress, or mouse click, I make should be about /narrowing/ the search space (rather than needing to expand it before narrowing).
The worry (about changing defaults) is I believe moot as at the moment—the URLs are generally too big to post/save/email for longevity anyway without modification; the URLs that /are/ making it into the wild with longevity (Pitti, comment #12) are those that /have/ been manually hand-hacked down to a usable length.

On top of mpt's estimated savings (comment #10), having "field.status:list=any" (insert of per-item enumeration) would remove most of the length, and then going further and having 'any' as default (per paragraph immediately above) but allow elimination.