Launchpad itself

Update Storm

Bug #352965 reported by Stuart Bishop
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Stuart Bishop
Launchpad itself 2.2.7

Bug Description

We want to sync our version of Storm to upstream. Of particular interest is jtv's generational cache work which should help some of our memory exhaustion issues.

Related branches

lp:~stub/launchpad/update-storm
Launchpad code reviewers: Pending requested
Diff: 2644 lines (+362/-446)
44 files modified
lib/canonical/launchpad/database/account.py (+7/-7)
lib/canonical/launchpad/database/stormsugar.py (+1/-19)
lib/canonical/launchpad/database/tests/test_stormextensions.py (+0/-101)
lib/canonical/launchpad/helpers.py (+45/-0)
lib/canonical/launchpad/vocabularies/dbobjects.py (+7/-2)
lib/canonical/launchpad/webapp/pgsession.py (+16/-14)
lib/canonical/launchpad/webapp/vocabulary.py (+5/-4)
lib/lp/archivepublisher/ftparchive.py (+15/-9)
lib/lp/bugs/model/bug.py (+25/-17)
lib/lp/bugs/model/bugtask.py (+13/-8)
lib/lp/bugs/model/bugwatch.py (+20/-24)
lib/lp/code/model/branch.py (+2/-2)
lib/lp/hardwaredb/model/hwdb.py (+17/-13)
lib/lp/registry/browser/distributionsourcepackage.py (+2/-1)
lib/lp/registry/doc/vocabularies.txt (+1/-1)
lib/lp/registry/model/distribution.py (+9/-8)
lib/lp/registry/model/distributionsourcepackage.py (+2/-4)
lib/lp/registry/model/person.py (+18/-33)
lib/lp/registry/model/pillar.py (+4/-1)
lib/lp/registry/model/projectgroup.py (+2/-3)
lib/lp/registry/model/sourcepackage.py (+2/-3)
lib/lp/registry/model/sourcepackagename.py (+3/-1)
lib/lp/registry/vocabularies.py (+15/-16)
lib/lp/scripts/garbo.py (+5/-5)
lib/lp/scripts/utilities/sanitizedb.py (+1/-1)
lib/lp/services/database/bulk.py (+1/-2)
lib/lp/services/worlddata/model/language.py (+5/-4)
lib/lp/soyuz/doc/gina.txt (+1/-1)
lib/lp/soyuz/doc/packageset.txt (+2/-2)
lib/lp/soyuz/model/archivepermission.py (+12/-13)
lib/lp/soyuz/model/binarypackagebuild.py (+2/-3)
lib/lp/soyuz/model/binarypackagename.py (+9/-5)
lib/lp/soyuz/model/distroarchseriesbinarypackage.py (+4/-10)
lib/lp/soyuz/model/packagediff.py (+5/-6)
lib/lp/soyuz/model/packageset.py (+17/-17)
lib/lp/soyuz/model/publishing.py (+22/-30)
lib/lp/soyuz/model/queue.py (+28/-40)
lib/lp/soyuz/scripts/initialise_distroseries.py (+3/-1)
lib/lp/translations/model/pofile.py (+1/-2)
lib/lp/translations/model/potemplate.py (+1/-2)
lib/lp/translations/model/translationimportqueue.py (+1/-2)
lib/lp/translations/scripts/migrate_variants.py (+9/-9)
setup.py (+1/-0)
versions.cfg (+1/-0)
Jonathan Lange (community): Approve
Christian Reis: Pending (release-critical) requested
Diff: None lines
Stuart Bishop (stub)
Changed in launchpad-foundations:
milestone: none → 2.2.4
status: New → Triaged
Revision history for this message
Stuart Bishop (stub) wrote :

All our local changes relate to Bug #241267, which is now fix committed.

I'm attempting to run our test suite against trunk now on ec2.

Francis J. Lacoste (flacoste)
Changed in launchpad-foundations:
milestone: 2.2.4 → 2.2.6
Revision history for this message
Stuart Bishop (stub) wrote :

Salgado and bac have also been looking into this:

https://pastebin.canonical.com/17460/

Newer Storm doesn't register Stores with the transaction manager until they are actually used, so transaction.abort() doesn't call Store.rollback() on these Stores. This is good, since it stops noop ROLLBACK statements being issued to the database. However, it breaks some tests apparently - probably sniffing to confirm that commits or rollbacks occur when they are expected to. We need to fix these tests, or apply the join transaction patch above to SQLBase._reset_store if we can confirm that method is only called by the test suite.

Changed in launchpad-foundations:
importance: Undecided → Medium
Revision history for this message
Stuart Bishop (stub) wrote :

Other fixes are already on lp:~stub/launchpad/update-storm

Stuart Bishop (stub)
Changed in launchpad-foundations:
assignee: nobody → Stuart Bishop (stub)
status: Triaged → In Progress
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

JamesH notes that the lazy registering of stores can also improve test times. The Landscape folks have seen hefty speedups, apparently, whereas LP's test suite slowed down when we added more stores.

Stuart Bishop (stub)
Changed in launchpad-foundations:
status: In Progress → Fix Committed
Revision history for this message
Stuart Bishop (stub) wrote :

I had to revert this branch. Tests pass locally and under ec2test.py, but buildbot has failures. http://buildbot.devpad.info/builders/lp/builds/432/steps/shell_7/logs/stdio.

lp:~stub/launchpad/update-storm is prepped and ready to land again.

I'm reassigning to Francis as I don't have the ability to investigate further.

Changed in launchpad-foundations:
assignee: Stuart Bishop (stub) → Francis J. Lacoste (flacoste)
status: Fix Committed → In Progress
Revision history for this message
Stuart Bishop (stub) wrote :

The failing script was reset_sequences.py. This was one of the scripts fixed in Bug #389065. This script was not running with the correct path, so had no hope of finding Storm. I'll resubmit now.

Changed in launchpad-foundations:
assignee: Francis J. Lacoste (flacoste) → Stuart Bishop (stub)
Stuart Bishop (stub)
Changed in launchpad-foundations:
status: In Progress → Fix Committed
Revision history for this message
Stuart Bishop (stub) wrote :

This was reverted again due to insufficient QA time and suspect memory usage.

We will land it again as soon as we are out of release mode for testing on staging and edge.

Changed in launchpad-foundations:
status: Fix Committed → In Progress
Revision history for this message
Stuart Bishop (stub) wrote :

Landed on lp:~launchpad-pqm/launchpad/db-devel r8210 for testing on staging. We should also merge lp:~stub/launchpad/update-storm into lp:~launchpad-pqm/launchpad/devel for testing on edge.

Changed in launchpad-foundations:
importance: Medium → High
milestone: 2.2.6 → 2.2.7
Revision history for this message
Stuart Bishop (stub) wrote :

Landed. Fallout being dealt with in Bug #390861

Changed in launchpad-foundations:
status: In Progress → Fix Committed
Stuart Bishop (stub)
Changed in launchpad-foundations:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.