Files Trash Times Out in Justice Initiative Central Community

[Nat Katin-Borland]

Clicking on the Files section Trash link in the Justice Initiative Central community (https://karl.soros.org/communities/justice-initiative-administration/files) times out. Sometimes the Trash page won't load and sometimes the page loads and then crashes if you actually try to restore a file or folder. Robert checked the server logs and he was seeing 80-90% CPU usage after clicking on the Trash link, so it was slowing down all of KARL.

Part of the problem may be with the fact that there are 2 folders with the title 'Staff':

https://karl.soros.org/communities/justice-initiative-administration/files/staff-1/
https://karl.soros.org/communities/justice-initiative-administration/files/staff/ (currently has a General Error)

Items from one of the Staff folders is in the Trash folder and KARL doesn't seem to know which Staff folder to restore the items to.

Please see if you can sort out the problem.

[Paul Everitt (paul-agendaless)]

Shane...help!

[Shane Hathaway (shane-hathawaymix)]

Diagnosis: when Karl displays the trash for that folder, it rummages through 845 archived containers and 7782 archived objects. That's just too many.

I tried to fix it by reducing the number of database round-trips, but even then it still takes 10 seconds to display each page. I think the trash UI code needs an algorithmic overhaul: we can't afford to build a tree of the entire trash hierarchy on each page view.

[Robert Marianski (rmarianski)]

Are there that many objects in the trash? Is it possible to speed it up by adding an index for what the ui is asking for?

[Nat Katin-Borland (nborland)]

The one piece of the story that's missing from Shane's diagnosis is the part about how Trash works with 2 folders with the same name. It seems like KARL isn't able to tell what the original location of the document was if you restore a file from the Trash and there are 2 folders with the same name. We might have to tweak the Trash logic to handle this edge case. Can you guys comment if this is part of the problem?

[Paul Everitt (paul-agendaless)]

Hmm, it looks like two different issues packed under the description of "time out". Can we split those into 2 issues?

--Paul

On Jan 17, 2012, at 1:58 PM, Nat Katin-Borland wrote:

> The one piece of the story that's missing from Shane's diagnosis is the
> part about how Trash works with 2 folders with the same name. It seems
> like KARL isn't able to tell what the original location of the document
> was if you restore a file from the Trash and there are 2 folders with
> the same name. We might have to tweak the Trash logic to handle this
> edge case. Can you guys comment if this is part of the problem?
>
> --
> You received this bug notification because you are subscribed to KARL3.
> https://bugs.launchpad.net/bugs/913990
>
> Title:
> Files Trash Times Out in Justice Initiative Central Community
>
> Status in KARL3:
> In Progress
>
> Bug description:
> Clicking on the Files section Trash link in the Justice Initiative
> Central community (https://karl.soros.org/communities/justice-
> initiative-administration/files) times out. Sometimes the Trash page
> won't load and sometimes the page loads and then crashes if you
> actually try to restore a file or folder. Robert checked the server
> logs and he was seeing 80-90% CPU usage after clicking on the Trash
> link, so it was slowing down all of KARL.
>
> Part of the problem may be with the fact that there are 2 folders with
> the title 'Staff':
>
> https://karl.soros.org/communities/justice-initiative-administration/files/staff-1/
> https://karl.soros.org/communities/justice-initiative-administration/files/staff/ (currently has a General Error)
>
> Items from one of the Staff folders is in the Trash folder and KARL
> doesn't seem to know which Staff folder to restore the items to.
>
> Please see if you can sort out the problem.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/913990/+subscriptions

[Paul Everitt (paul-agendaless)]

Nat moved part of this out to another ticket.

Shane, can you propose the UI you're thinking of that would reduce the need to load so many objects?

[Shane Hathaway (shane-hathawaymix)]

I don't want to change the UI, I just want to change the implementation of the UI. There is no need to load the whole tree for each page view of the trash.

[Paul Everitt (paul-agendaless)]

Ahh, good, in that case fire away.

--Paul

On Jan 20, 2012, at 3:49 PM, Shane Hathaway wrote:

> I don't want to change the UI, I just want to change the implementation
> of the UI. There is no need to load the whole tree for each page view of
> the trash.
>
> --
> You received this bug notification because you are subscribed to KARL3.
> https://bugs.launchpad.net/bugs/913990
>
> Title:
> Files Trash Times Out in Justice Initiative Central Community
>
> Status in KARL3:
> In Progress
>
> Bug description:
> Clicking on the Files section Trash link in the Justice Initiative
> Central community (https://karl.soros.org/communities/justice-
> initiative-administration/files) times out. Sometimes the Trash page
> won't load and sometimes the page loads and then crashes if you
> actually try to restore a file or folder. Robert checked the server
> logs and he was seeing 80-90% CPU usage after clicking on the Trash
> link, so it was slowing down all of KARL.
>
> Part of the problem may be with the fact that there are 2 folders with
> the title 'Staff':
>
> https://karl.soros.org/communities/justice-initiative-administration/files/staff-1/
> https://karl.soros.org/communities/justice-initiative-administration/files/staff/ (currently has a General Error)
>
> Items from one of the Staff folders is in the Trash folder and KARL
> doesn't seem to know which Staff folder to restore the items to.
>
> Please see if you can sort out the problem.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/913990/+subscriptions

[Paul Everitt (paul-agendaless)]

I'm going to consider this one done for the first part of the timeout, so we can get some immediate relief into production. The rest of the work is in lp:920668

[Chris Rossi (chris-archimedeanco)]

Deployed for staging on branch2.

[JimPGlenn (jpglenn09)]

Community does not exist on branch2 and it appears to work fine.

[JimPGlenn (jpglenn09)]

fixed