Not Searching for Reference Manuals on Advanced Search Results Page

[Nat Katin-Borland]

Reference manuals should be included in the content being searched under the Wiki filter - matching what we have in LiveSearch. Right now, the wiki filter only shows wiki pages in communities. This was our compromise for still being able to search reference manuals without adding a new category to LiveSearch or the advanced search filters.

[Paul Everitt (paul-agendaless)]

Don't know if we can get this in in time. If it's easy, let's give it a shot.

[Chris Rossi (chris-archimedeanco)]

In order to accomplish this, we would need to switch from using facets that are linked only to content type and use something more like what's done for LiveSearch where more complex queries can be built up. During the phone call a couple of weeks ago, I asked if we should do that and the consensus at the time was stick to just the plain content type facets. Unless we want to revisit that decision, there's not really anything I can do here.

[Chris Rossi (chris-archimedeanco)]

Actually it looks like I can replace 'Wikis' with 'Pages' and there is a content type I can use that will get both Wiki Pages. Is this what you want?

[Chris Rossi (chris-archimedeanco)]

And now after having tried it, I realize it is somewhat more complex than I thought, but is still doable in about an hour. I would probably end up refactoring how we decide which facets to show up on the left somewhat. Rather than rely on the tagged values of the interfaces we'd just need to build up a mapping explicitly in the search results view. Let me know if I should go this route.

[Robert Marianski (rmarianski)]

My vote is to build the mapping explicitly and not tie it to the tagged values. This will open up more possibilities for us and let us search on different things that aren't tied to facets explicitly.

[Chris Rossi (chris-archimedeanco)]

The trade off, of course, is that we lose extensibility. Karl has mostly
been designed such that most of its functionality can be changed via a
customization package. Since currently, the facets are built up from
information that lives in the component registry, we can do this. Doing the
facets explicitly, breaks this. We're already sort of breaking this with
the arbitrary sort order and the pluralization of the names. But this would
break it completely. May or may not be a consideration.

 This will open up more possibilities for us and let us search on
> different things that aren't tied to facets explicitly.

Actually, it won't. Using LiveSearch style machinery for the facets would.
 And it would also be extensible. That really is the best option in terms
of flexibility and extensibility. The above is the best option in terms of
cost and releasing something this week.

Chris

[Robert Marianski (rmarianski)]

We're just emphasizing different things.

You're seeing it from an implementation perspective, that a more flexible solution would be to just generate the list completely from tagged values, which would allow customization packages to influence what appears and even add to it by tweaking the tagged values. And having the list come from the view would be constraining because overriding the filters would not be as easy in customization packages.

I'm seeing it from a user perspective, that a more flexible solution would allow us to search on whatever we wanted, and wouldn't be tied to the underlying implementation of content types. For example, separating reference manuals and wiki pages in communities, or distinguishing between comments in forum topics and comments on blog posts from communities. I was seeing tying the filters to content types as a constraint.

[Robert Marianski (rmarianski)]

Just so there's a next step decision point, I spoke with Nat and for this Thursday it would be great if we could get the search results page to return results from reference manuals in some way that wasn't all content.

Our ideal case would be that reference manuals is split up into another filter that the user can click on. But if that's a big pain and it's much easier to include those results as part of the wiki filter, let's go with that. From what we can tell, there's no way to get reference manual results outside of all content. Searching for information within a policy document is a use case we think is common here.

[Chris Rossi (chris-archimedeanco)]

On Wed, May 18, 2011 at 11:04 AM, Robert Marianski <
<email address hidden>> wrote:

> We're just emphasizing different things.
>
> You're seeing it from an implementation perspective, that a more
> flexible solution would be to just generate the list completely from
> tagged values, which would allow customization packages to influence
> what appears and even add to it by tweaking the tagged values. And
> having the list come from the view would be constraining because
> overriding the filters would not be as easy in customization packages.
>
> I'm seeing it from a user perspective, that a more flexible solution
> would allow us to search on whatever we wanted, and wouldn't be tied to
> the underlying implementation of content types. For example, separating
> reference manuals and wiki pages in communities, or distinguishing
> between comments in forum topics and comments on blog posts from
> communities. I was seeing tying the filters to content types as a
> constraint.
>
>
Well, the explicit way that I'm proposing doesn't let you execute arbitrary
queries. That's what the live search way does allow you to do, though. The
explicit way that is on the table now, would look like this:

facets = [{
    'name': 'Pages'
    'interface': 'IPages',
    ...: etc...
    },
    'name': 'People',
    'interface': 'IProfile',
    ...: etc...
    },
    etc...]

The reason we can't do this with the tagged values is IPages is not
registered as a content type. But we could just build up the list
explicitly in the view code.

The LiveSearch way gives you true flexibility, both from a programmer and
user point of view by allowing any arbitrary query to be executed as a
result of clicking on a facet.

Chris

[Nat Katin-Borland (nborland)]

Hi Chris,

Did you see Robert's last comment? I just want to make sure we're all on the same page about what we're doing (or not doing) for this week's roll-out.

-Nat

[Paul Everitt (paul-agendaless)]

Didn't get to this in time with the short workweek.

[Chris Rossi (chris-archimedeanco)]

Ok, I have completed this on a branch which is ready for testing on branch5:

https://karlstaging.gocept.com/branch5/osf/

This involved some refactoring, which I'll try to discuss briefly at a high level:

1) The "types" in the left sidebar of the search results page now use the same machinery as the groupings in live search. This means that a grouping in the search results can now execute an arbitrary catalog query. Previously the types were tied to a particular content type and couldn't be used to perform any queries more elaborate than just the content type.

2) The catalog now has a new index that tracks the content types of the folders a document is contained in. This opens a new type of query that can be performed, eg "give me any documents contained inside of a reference manual."

These refactors were required to create the "Reference Manuals" type that is clickable from the sidebar of the search results page. Much care was taken to preserve all current functionality with regards to the other types in the sidebar, as well as the relationship between types in the livesearch and types on the search results page. So everything besides the new "Reference Manuals" link should function exactly as before, but because the way these work underneath the hood has been fundamentally altered, it is worth spending some extra testing effort verifying this is the case.

This also, now, opens up the possibility of altering the behavior of search in ways not previously possible. For example, it is now possible to create a "Blogs" search type that searches for blog entries and blog comments but not forum comments. Because all comments, be they blog comments or forum comments, have the same content type in Karl, this was not possible previously as a search for comments would necessarily return comments from anywhere on the site. One possible upshot of this is we could get rid of the (probably seldom used) "Comments" type and enhance the "Blogs" and "Forums" types to include comments contained in their respective types.

[Nat Katin-Borland (nborland)]

Hi Chris,

Thanks for taking a crack at this. I just took a quick look at your fix and found a few bugs.

-The "Reference Manuals" filter should be moved up the list so that it sits just below "People" (see screenshot)
-When you select the Reference Manual filter and it becomes bolded, Reference Manual becomes misaligned and drops to a second line (also see screenshot). We should make sure Reference Manual stays on one line even when it's selected.
-Your fix seems to have triggered some bizarre behavior in LiveSearch. Specifically: the 5 results per category cap on All Content is being ignored, categories like Files and Wiki are appearing multiple times in the results drop-down and there is a new category appearing called "Unknown" (I have no idea what this is).

-Nat

--
Nathaniel Katin-Borland
Support Specialist
Knowledge Management Initiative
KARL Support Team

Open Society Foundations - New York Office
400 West 59th Street
New York, NY 10019
Email: <email address hidden>
Phone: [phone]
http://www.soros.org/
http://www.karlproject.org

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Chris Rossi
Sent: Thursday, June 23, 2011 1:24 PM
To: Nathaniel Katin-Borland
Subject: [Bug 784222] Re: Not Searching for Reference Manuals on Advanced Search Results Page

Ok, I have completed this on a branch which is ready for testing on
branch5:

https://karlstaging.gocept.com/branch5/osf/

This involved some refactoring, which I'll try to discuss briefly at a high level:

1) The "types" in the left sidebar of the search results page now use the same machinery as the groupings in live search. This means that a grouping in the search results can now execute an arbitrary catalog query. Previously the types were tied to a particular content type and couldn't be used to perform any queries more elaborate than just the content type.

2) The catalog now has a new index that tracks the content types of the folders a document is contained in. This opens a new type of query that can be performed, eg "give me any documents contained inside of a reference manual."

These refactors were required to create the "Reference Manuals" type that is clickable from the sidebar of the search results page. Much care was taken to preserve all current functionality with regards to the other types in the sidebar, as well as the relationship between types in the livesearch and types on the search results page. So everything besides the new "Reference Manuals" link should function exactly as before, but because the way these work underneath the hood has been fundamentally altered, it is worth spending some extra testing effort verifying this is the case.

This also, now, opens up the possibility of altering the behavior of search in ways not previously possible. For example, it is now possible to create a "Blogs" search type that searches for blog entries and blog comments but not forum comments. Because all comments, be they blog comments or forum comments, have the same content type in Karl, this was not possible previously as a search for comments would necessarily return ...

[Chris Rossi (chris-archimedeanco)]

Hi Nat,

Thanks for taking a look at that. I have addressed everything you brought up. The fixes are up on branch5.

Thanks,
Chris

[JimPGlenn (jpglenn09)]

did not appear to have made it into release...

[Chris Rossi (chris-archimedeanco)]

I have merged this change to trunk and it is ready for testing on branch1. The merge was a little hairy with a lot of conflicts due to svn doing some deeply moronic things, so this is worth some extra testing.

[Nat Katin-Borland (nborland)]

This looks good and is ready to roll out!

[JimPGlenn (jpglenn09)]

fixed....rolled out