Cve:+index timeouts

Bug #727023 reported by Robert Collins on 2011-03-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Critical
Robert Collins

Bug Description

      3 https://bugs.launchpad.net/bugs/cve/2010-4080/+index (Cve:+index)
 OOPS-1906C1974
      3 https://launchpad.net/bugs/cve/2009-0745/+index (Cve:+index)
OOPS-1906D1997

Still doing lazy lookups:

1 67 433 6 427 SQL-launchpad-main-master
SELECT Distribution.answers_usage,
       Distribution.blueprints_usage,
       Distribution.translations_usage,
       Distribution.bug_reported_acknowledgement,
       Distribution.bug_reporting_guidelines,
       Distribution.bug_supervisor,
       Distribution.date_created,
       Distribution.description,
       Distribution.displayname,
       Distribution.domainname,
       Distribution.driver,
       Distribution.enable_bug_expiration,
       Distribution.homepage_content,
       Distribution.icon,
       Distribution.id,
       Distribution.language_pack_admin,
       Distribution.logo,
       Distribution.max_bug_heat,
       Distribution.members,
       Distribution.mirror_admin,
       Distribution.mugshot,
       Distribution.name,
       Distribution.official_answers,
       Distribution.official_blueprints,
       Distribution.official_malone,
       Distribution.official_rosetta,
       Distribution.OWNER, Distribution.security_contact,
                           Distribution.summary,
                           Distribution.title,
                           Distribution.translation_focus,
                           Distribution.translationgroup,
                           Distribution.translationpermission
FROM Distribution
WHERE id IN ($INT)
ORDER BY Distribution.name
2 54 942 17 925 SQL-launchpad-main-master
SELECT Milestone.active, Milestone.codename, Milestone.dateexpected, Milestone.distribution, Milestone.distroseries, Milestone.id, Milestone.name, Milestone.product, Milestone.productseries, Milestone.summary
FROM Milestone
WHERE Milestone.distroseries = $INT
  AND Milestone.active = TRUE
ORDER BY milestone_sort_key(Milestone.dateexpected, Milestone.name) DESC
3 53 442 8 434 SQL-launchpad-main-master
SELECT SourcePackagePublishingHistory.ancestor,
       SourcePackagePublishingHistory.archive,
       SourcePackagePublishingHistory.component,
       SourcePackagePublishingHistory.datecreated,
       SourcePackagePublishingHistory.datemadepending,
       SourcePackagePublishingHistory.datepublished,
       SourcePackagePublishingHistory.dateremoved,
       SourcePackagePublishingHistory.datesuperseded,
       SourcePackagePublishingHistory.distroseries,
       SourcePackagePublishingHistory.id,
       SourcePackagePublishingHistory.pocket,
       SourcePackagePublishingHistory.removal_comment,
       SourcePackagePublishingHistory.removed_by,
       SourcePackagePublishingHistory.scheduleddeletiondate,
       SourcePackagePublishingHistory.section,
       SourcePackagePublishingHistory.sourcepackagerelease,
       SourcePackagePublishingHistory.status,
       SourcePackagePublishingHistory.supersededby
FROM SourcePackagePublishingHistory
WHERE distroseries = $INT
  AND archive IN ($INT, $INT)
  AND sourcepackagerelease = $INT
ORDER BY SourcePackagePublishingHistory.datecreated DESC LIMIT $INT

Related branches

description: updated
j.c.sackett (jcsackett) on 2011-03-09
Changed in launchpad:
assignee: nobody → j.c.sackett (jcsackett)
j.c.sackett (jcsackett) on 2011-03-11
Changed in launchpad:
status: Triaged → In Progress
Robert Collins (lifeless) wrote :

406->58 queries for the first url, times out cold on qastaging, 5 seconds hot.
506->??? for the second url - I couldn't get it to render. it doesn't on prod either though so strictly an improvement.

Robert Collins (lifeless) wrote :

OOPS-1904QS55 from staging may shed light on the cause.

Robert Collins (lifeless) wrote :

hah, I was testing the unfixed code ><

Launchpad QA Bot (lpqabot) wrote :
Changed in launchpad:
milestone: none → 11.04
tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Robert Collins (lifeless) wrote :

ok, still can't get https://qastaging.launchpad.net/bugs/cve/2009-0745/+index to render: OOPS-1904QS73

tags: added: qa-ok
removed: qa-needstesting
Robert Collins (lifeless) wrote :

both referenced CVE's timed out in prod, updating the description with the new OOPS ids

Changed in launchpad:
status: Fix Committed → Triaged
milestone: 11.04 → none
description: updated
tags: removed: qa-ok
Robert Collins (lifeless) wrote :

Note that the total SQL time may look low, but IME this is a symptom of late evaluation - query compilation etc times drive this up, and making things eager load, cacheproperty etc will often drive the non-sql time down too.

description: updated
Robert Collins (lifeless) wrote :

so, this is a problem page for the same reason bugtask:+index is I think; I need to check but I suspect its reusing the same nested template + per-row view logic - see https://bugs.qastaging.launchpad.net/++profile++show/bugs/cve/2010-4080/+index

Robert Collins (lifeless) wrote :

OOPS-1909EA1446

Robert Collins (lifeless) wrote :
Download full text (3.4 KiB)

notes:
lib/lp/bugs/browser/buglinktarget.py

  File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/browser/buglinktarget.py", line 87, in buglinks
    for bug in self.context.bugs:

 - that triggers

SELECT Bug.date_last_message, Bug.date_last_updated, Bug.date_made_private, Bug.datecreated, Bug.description, Bug.duplicateof, Bug.heat, Bug.heat_last_updated, Bug.id, Bug.latest_patch_uploaded, Bug.message_count, Bug.name, Bug.number_of_duplicates, Bug.owner, Bug.private, Bug.security_related, Bug.title, Bug.users_affected_count, Bug.users_unaffected_count, Bug.who_made_private FROM Bug, BugCve WHERE BugCve.cve = 1 AND Bug.id = BugCve.bug ORDER BY Bug.id

which doesn't have eager loading attached; possibly we need that to be a cachedproperty or some such.

  File "/home/robertc/launchpad/lp-branches/working/lib/lp/services/propertycache.py", line 116, in __get__
    value = self.populate(instance)
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/browser/bugtask.py", line 3500, in target_has_milestones
    return len(self._visible_milestones) > 0
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/services/propertycache.py", line 116, in __get__
    value = self.populate(instance)
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/browser/bugtask.py", line 3474, in _visible_milestones
    return self.milestone_source(self.context).visible_milestones
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/services/propertycache.py", line 116, in __get__
    value = self.populate(instance)
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/registry/vocabularies.py", line 1241, in visible_milestones
    milestones = shortlist(milestones_source, longest_expected=40)
  File "/home/robertc/launchpad/lp-branches/working/lib/canonical/launchpad/helpers.py", line 320, in shortlist
    results = list(sequence[:last])

SELECT Milestone.active, Milestone.codename, Milestone.dateexpected, Milestone.distribution, Milestone.distroseries, Milestone.id, Milestone.name, Milestone.product, Milestone.productseries, Milestone.summary FROM Milestone WHERE Milestone
.distribution = 1 AND Milestone.active = True ORDER BY milestone_sort_key(Milestone.dateexpected, Milestone.name) DESC

- 109 milestone lookups on a local test - and this comes from the
lib/lp/bugs/browser/../templates/bugtask-tasks-and-nominations-table-row.pt template looking to see whether to show the milestone widget

  File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/browser/bugtask.py", line 3512, in user_can_edit_importance
    return self.context.userCanEditImportance(self.user)
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/model/bugtask.py", line 1229, in userCanEditImportance
    return (self._userIsPillarEditor(user) or
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/bugs/model/bugtask.py", line 1220, in _userIsPillarEditor
    pillar.userCanEdit(user))
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/registry/model/distribution.py", line 1774, in userCanEdit
    return user.inTeam(self.owner) or user.inTeam(admins)
  File "/home/robertc/launchpad/lp-branches/working/lib/lp/registry/...

Read more...

j.c.sackett (jcsackett) wrote :

> so, this is a problem page for the same reason bugtask:+index is I think;
> I need to check but I suspect its reusing the same nested > template + per-row view logic
> - see https://bugs.qastaging.launchpad.net/++profile++show/bugs/cve/2010-4080/+index

You're correct; it uses the bugtasks-nominations-tasks-row template, which triggers multiple milestone lookups via a very peculiar function in the milestone vocabulary.

On Sat, Mar 26, 2011 at 3:08 AM, j.c.sackett <email address hidden> wrote:
>> so, this is a problem page for the same reason bugtask:+index is I think;
>> I need to check but I suspect its reusing the same nested > template + per-row view logic
>> - see https://bugs.qastaging.launchpad.net/++profile++show/bugs/cve/2010-4080/+index
>
> You're correct; it uses the bugtasks-nominations-tasks-row template,
> which triggers multiple milestone lookups via a very peculiar function
> in the milestone vocabulary.

It actually seems to be worse than bugtask:+index - probably because
that has lots of partial optimisations and I suspect cve:+index has
been largely ignored.

So we have two tasks:
 - get cve:+index doing the same eager loading bugtask:+index already does
 - fix the milestone pathology in both places.

The latter has had me headdesking when I have looked at it for a
couple weeks, I suspect the right answer is:
 - strip stuff back to bare bones
 - rewrite the vocab on the underlying layer
 - get milestones in the view page using the underlying layer

j.c.sackett (jcsackett) on 2011-03-28
Changed in launchpad:
assignee: j.c.sackett (jcsackett) → nobody
Robert Collins (lifeless) wrote :

I spoke to deryck and kees (nominated as a key user) about this and removing the edit widgets is simple and ok to do.

Changed in launchpad:
assignee: nobody → Robert Collins (lifeless)
Launchpad QA Bot (lpqabot) wrote :
Changed in launchpad:
milestone: none → 11.04
tags: added: qa-needstesting
Changed in launchpad:
status: Triaged → Fix Committed
tags: added: qa-ok
removed: qa-needstesting
William Grant (wgrant) on 2011-03-31
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers