ProductRelease:+rdf timeout

[Robert Collins]

 1 SELECT Milestone.active, Milestone.codename, Milestone.dateexpected, Milestone.distribution, Mile ... t, Milestone.productseries, Milestone.summary FROM Milestone WHERE Milestone.id = $INT LIMIT $INT:
   GET: 1 Robots: 1 Local: 0
      1 https://launchpad.net/libpng/main/1.2.9beta6/+rdf (ProductRelease:+rdf)
       OOPS-1900M664

The repeated statements section has
 Reps Total time Average time Saving Database id Statement
1 501 10856 21 10835 SQL-launchpad-main-slave
SELECT Milestone.active,
       Milestone.codename,
       Milestone.dateexpected,
       Milestone.distribution,
       Milestone.distroseries,
       Milestone.id,
       Milestone.name,
       Milestone.product,
       Milestone.productseries,
       Milestone.summary
FROM Milestone
WHERE Milestone.id = $INTLIMIT $INT

Thats 501 late-lookups of milestones. There are 777 milestones for libpng on production, and as this rdf is for a single release/milestone it suggests that the traversal is examining all the milestones in a list rather than doing a dict-style dereference.

[Curtis Hovey (sinzui)]

Death by late evaluation of milestones.

[Curtis Hovey (sinzui)]

I did not finish thinking here. The page timed out iterating over milestones, but the URL is for a ProductRelease which has exactly 1 milestone.

/me looks at rdf

No milestones is listed. This is a very short file. Neither the template or the view are explicitly working with milestones. Nor are there any iterations in the view or template. The template uses the simple values of the release most of the time. It gets the releases Product twice to make URLs, and the productseries once to get its name.

I suspect then that
    string:${context/product/fmt:url}/${context/productseries/name}/+rdf"
is implicitly the cause. I can see in the SQL log that the Product is retrieved, then the ProductSeries is retrieved...which starts a long repetition to get each milestone. The only code that appears to select by milestone is a property:

    @property
    def releases(self):
        """See `IProductSeries`."""
        store = Store.of(self)
        result = store.find(
            ProductRelease,
            And(Milestone.productseries == self,
                ProductRelease.milestone == Milestone.id))
        return result.order_by(Desc('datereleased'))

Oh, is this an open join, or is there an odd query plan created by the And()? I see one product release in the sql log, then hundreds of unmatched milestones. I do not think the And() is needed. I would have written the method as:

    @property
    def releases(self):
        """See `IProductSeries`."""
        store = Store.of(self)
        result = store.find(
            ProductRelease,
            Milestone.productseries == self,
            ProductRelease.milestone == Milestone.id)
        return result.order_by(Desc('datereleased'))

[Robert Collins (lifeless)]

So when late evaluation of milestones is happening, something is loading *references* to milestones but not the milestones. So selection by milestone isn't necessary. The sql log is often a good hint to the cause of this sort of thing: right before the 501 milestones start we have
7. 51 61ms SQL-launchpad-main-slave
SELECT ProductRelease.changelog,
       ProductRelease.datecreated,
       ProductRelease.datereleased,
       ProductRelease.id,
       ProductRelease.milestone,
       ProductRelease.OWNER, ProductRelease.release_notes
FROM Milestone,
     ProductRelease
WHERE Milestone.productseries = 4155
  AND ProductRelease.milestone = Milestone.id
ORDER BY datereleased DESC

which is grabbing all the releases of productseries 'main' for libpng: 777 rows.

My guess is that the traversal is being done via looking in a python property rather than a direct lookup for the specific release.

[Robert Collins (lifeless)]

Oh, and the thing being formatted is a diversion - we're getting past the traversal only just, I think.

[Curtis Hovey (sinzui)]

I do not follow the diversion remark. The series line is the second to last code we evaluate. The last thing is the link to the product release registrant. BTW, the series has more than 700 milestones, so I am not sure we passed that line accessing the series.

[Curtis Hovey (sinzui)]

ProductSeriesNavigation.traverse() call ProductSeries.getRelease()

    def getRelease(self, version):
        for release in self.releases:
            if release.version == version:
                return release
        return None

The ProductSeries,getRelease or ProductSeriesNavigation.traverse() could:

        return getUtility(IProductReleaseSet).getBySeriesAndVersion(series, name)

where series is wither self or self.context. None is if there are not matches.

[Curtis Hovey (sinzui)]

ProductReleaseSet.getBySeriesAndVersion() Has been broken for about 2 years. It is using a non-existent column in the where condition.

[Robert Collins (lifeless)]

return getUtility(IProductReleaseSet).getBySeriesAndVersion(series, name) certainly looks like it would be an improvement (if fixed ;))

[Launchpad QA Bot (lpqabot)]

Fixed in stable r13060 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13060>.