Including binaries when copying pkgs with lots of binaries oopses
Bug #447138 reported by
Michael Nelson
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
Medium
|
Michael Nelson |
Bug Description
OOPS-1378C1323
Nicolas experienced the above oops while trying to copy openoffice.org with binaries included - and mentioned that someone in the security team also experienced this recently?
It seems that we create and (re)fetch each BinaryPackagePu
Related branches
lp:~michael.nelson/launchpad/copy-binaries-timeout
Merged
into
lp:launchpad
- Muharem Hrnjadovic (community): Approve
-
Diff: 196 lines3 files modifiedlib/lp/soyuz/interfaces/publishing.py (+17/-0)
lib/lp/soyuz/model/publishing.py (+69/-26)
lib/lp/soyuz/scripts/packagecopier.py (+6/-25)
Changed in soyuz: | |
status: | New → Triaged |
importance: | Undecided → Medium |
tags: | added: oops |
tags: | added: tech-debt |
tags: | added: oem-services |
Changed in soyuz: | |
assignee: | nobody → Michael Nelson (michael.nelson) |
status: | Triaged → In Progress |
milestone: | none → 3.1.10 |
Changed in soyuz: | |
status: | In Progress → Triaged |
Changed in soyuz: | |
status: | Fix Committed → Fix Released |
To post a comment you must log in.
In terms of a pre-implementation discussion, here's what I'm planning:
1. Look at adding a PublishingSet. copyBinariesTo( binaries, series, pocket, archive) that ensures certain queries are not evaluated for each binary. For example, the biggest timesaver according to the oops is from the PPA component override to main - we look up the 'main' component before creating each binary.
2. The next worst offender is the repeated inserts - can we batch these with storm? would there be much of a benefit? - Hrm, no, it seems storm.store.Store only has a singular add method. Not much to do here afaics.
3. The third worst offender is a select on SecureBinaryPac kagePublishingH istory - and I'm not sure where this is coming from? At first I thought it was an artifact of the SBPPH insert, but there's another query listed for that. This one only selects 3 fgields on the SBPPH.
4. The fourth being to get the corresponding BPPH - at the end of PackageSet. newBinaryPublic ation() we have BinaryPackagePu blishingHistory .get(pub. id). This could also be a single query to get all the corresponding BPPHs in one hit if we have PS.copyBinaries To().
5. Refactor SPPH.getBuiltBi naries( ) getting rid of the code that currently iterates the results to get the BPPH for unique BPRs. We could instead return a result set of unique (BPPH, BPR) tuples directly from the database as the callsites all use the related BPRs anyway. (I'm assuming this is contributing to the non-sql time).