Performance problems with LiveSearch

[Nat Katin-Borland]

Try to improve the performance of LiveSearch. Note that the "add member" component of the original ticket was moved to 385283.

Originally
===========

The auto-suggest features on the global Live Search and the Add Existing Member field in the community section are both very slow. Is there something we can do to optimize the performance?

[Paul Everitt (paul-agendaless)]

Hi Nat. I just fired up IE7, logged in, and typed: "ever" into
LiveSearch.

I got results in under 2 seconds.

Can you go type that and see what performance you see?

Also, can you give an illustration of what LiveSearches you see that
are slow?

--Paul

On Jun 4, 2009, at 1:33 PM, Nat Katin-Borland wrote:

> Public bug reported:
>
> The auto-suggest features on the global Live Search and the Add
> Existing
> Member field in the community section are both very slow. Is there
> something we can do to optimize the performance?
>
> ** Affects: karl3
> Importance: Medium
> Assignee: Paul Everitt (paul-agendaless)
> Status: New
>
> ** Changed in: karl3
> Importance: Undecided => Medium
>
> ** Changed in: karl3
> Assignee: (unassigned) => Paul Everitt (paul-agendaless)
>
> --
> peformance problems with auto-suggest fields
> https://bugs.launchpad.net/bugs/383672
> You received this bug notification because you are a bug assignee.
>
> Status in Porting KARL to a new architecture: New
>
> Bug description:
> The auto-suggest features on the global Live Search and the Add
> Existing Member field in the community section are both very slow.
> Is there something we can do to optimize the performance?

[Nat Katin-Borland (nborland)]

It just took me about 5 seconds on the "ever" search. I was waiting
20-30 seconds yesterday for results and it was even worse for the Add
Existing Members field. Anthony and Yalan noted similar delays.

-Nat

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
Paul Everitt
Sent: Thursday, June 04, 2009 1:44 PM
To: Nathaniel Katin-Borland
Subject: Re: [Bug 383672] [NEW] peformance problems with auto-suggest
fields

Hi Nat. I just fired up IE7, logged in, and typed: "ever" into
LiveSearch.

I got results in under 2 seconds.

Can you go type that and see what performance you see?

Also, can you give an illustration of what LiveSearches you see that
are slow?

--Paul

On Jun 4, 2009, at 1:33 PM, Nat Katin-Borland wrote:

> Public bug reported:
>
> The auto-suggest features on the global Live Search and the Add
> Existing
> Member field in the community section are both very slow. Is there
> something we can do to optimize the performance?
>
> ** Affects: karl3
> Importance: Medium
> Assignee: Paul Everitt (paul-agendaless)
> Status: New
>
> ** Changed in: karl3
> Importance: Undecided => Medium
>
> ** Changed in: karl3
> Assignee: (unassigned) => Paul Everitt (paul-agendaless)
>
> --
> peformance problems with auto-suggest fields
> https://bugs.launchpad.net/bugs/383672
> You received this bug notification because you are a bug assignee.
>
> Status in Porting KARL to a new architecture: New
>
> Bug description:
> The auto-suggest features on the global Live Search and the Add
> Existing Member field in the community section are both very slow.
> Is there something we can do to optimize the performance?

--
peformance problems with auto-suggest fields
https://bugs.launchpad.net/bugs/383672
You received this bug notification because you are a direct subscriber
of the bug.

Status in Porting KARL to a new architecture: New

Bug description:
The auto-suggest features on the global Live Search and the Add Existing
Member field in the community section are both very slow. Is there
something we can do to optimize the performance?

[Paul Everitt (paul-agendaless)]

Some points to note on this:

1) If it is chronic (meaning, it happens more than 20% of the time),
then that's certainly worth investigating immediately.

2) There have been a number of times during the day in the last 3 days
that we have restarted the server during the day. When that happens,
the first few requests will be slow. But I doubt that is a factor here.

3) We have isolated LiveSearch and the People directory requests into
a separate app server process. It is possible that LiveSearch is
being interrupted by People searches, but I'd have to prove that.

Anyway, I'll slot this for getting looked at next week. Can I get an
estimate on how much is the maximum time we should spend looking at
this? We might find an easy answer, but we might also have to do a
good bit of investigation.

--Paul

On Jun 4, 2009, at 2:01 PM, Nat Katin-Borland wrote:

> It just took me about 5 seconds on the "ever" search. I was waiting
> 20-30 seconds yesterday for results and it was even worse for the Add
> Existing Members field. Anthony and Yalan noted similar delays.
>
> -Nat
>
> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of
> Paul Everitt
> Sent: Thursday, June 04, 2009 1:44 PM
> To: Nathaniel Katin-Borland
> Subject: Re: [Bug 383672] [NEW] peformance problems with auto-suggest
> fields
>
>
> Hi Nat. I just fired up IE7, logged in, and typed: "ever" into
> LiveSearch.
>
> I got results in under 2 seconds.
>
> Can you go type that and see what performance you see?
>
> Also, can you give an illustration of what LiveSearches you see that
> are slow?
>
> --Paul
>
> On Jun 4, 2009, at 1:33 PM, Nat Katin-Borland wrote:
>
>> Public bug reported:
>>
>> The auto-suggest features on the global Live Search and the Add
>> Existing
>> Member field in the community section are both very slow. Is there
>> something we can do to optimize the performance?
>>
>> ** Affects: karl3
>> Importance: Medium
>> Assignee: Paul Everitt (paul-agendaless)
>> Status: New
>>
>> ** Changed in: karl3
>> Importance: Undecided => Medium
>>
>> ** Changed in: karl3
>> Assignee: (unassigned) => Paul Everitt (paul-agendaless)
>>
>> --
>> peformance problems with auto-suggest fields
>> https://bugs.launchpad.net/bugs/383672
>> You received this bug notification because you are a bug assignee.
>>
>> Status in Porting KARL to a new architecture: New
>>
>> Bug description:
>> The auto-suggest features on the global Live Search and the Add
>> Existing Member field in the community section are both very slow.
>> Is there something we can do to optimize the performance?
>
> --
> peformance problems with auto-suggest fields
> https://bugs.launchpad.net/bugs/383672
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Porting KARL to a new architecture: New
>
> Bug description:
> The auto-suggest features on the global Live Search and the Add
> Existing
> Member field in the community section are both very slow. Is there
> something we can do to optimize the performance?
>
> --
> peformance problems with auto-suggest fields
> https://bugs.launchpad...

[Paul Everitt (paul-agendaless)]

Shane, sounds like it is related to the "analyze query logs" task for next week.

[Paul Everitt (paul-agendaless)]

Dang, forgot to assign to Shane.

[Jason Lantz (jasontlantz)]

We just received the following comment in the KARL Feedback Community blog from one of our users:

"Adding existing members to a community is PAINFULLY slow. I feel like I am waiting for an AOL dial up for each name entry. Can you add a list function like the Add New option?"

[Paul Everitt (paul-agendaless)]

It is painfully slow this morning because we did an update yesterday.
It should get faster as the ZEO cache warms up.

At the same time, this issue is still open, so the following simply re-
inforces the point of the open issue. Or, is there new information in
the comment?

--Paul

On Jun 9, 2009, at 10:05 AM, Jason Lantz wrote:

> We just received the following comment in the KARL Feedback Community
> blog from one of our users:
>
> "Adding existing members to a community is PAINFULLY slow. I feel
> like
> I am waiting for an AOL dial up for each name entry. Can you add a
> list
> function like the Add New option?"
>
> --
> Performance problems with auto-suggest fields
> https://bugs.launchpad.net/bugs/383672
> You received this bug notification because you are subscribed to
> KARL3.
>
> Status in Porting KARL to a new architecture: New
>
> Bug description:
> The auto-suggest features on the global Live Search and the Add
> Existing Member field in the community section are both very slow.
> Is there something we can do to optimize the performance?

[Chris McDonough (chrism-plope)]

zope.index 3.5.2 (in the lemonade index) may help a bit speedwise for general livesearch

[Shane Hathaway (shane-hathawaymix)]

For the next release, we have moved nearly all text searches to a separate search process. Then we gave the search process a cache that is much larger than the normal cache, while restricting that cache to only retain certain kinds of objects, so the cache doesn't get polluted. These combined improvements, along with the zope.index changes, should help.

[Paul Everitt (paul-agendaless)]

Site is updated. This is about as much as we could do near-term. Nat, if problems persist, let us know and we'll propose other steps.