Launchpad itself

Remove Popen from buildstatus_OK

Bug #506256 reported by Michael Nelson
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Medium
Jelmer Vernooij
Launchpad itself 10.10

Bug Description

Currently Popen is used to upload source packages and essentially means that the builder infrastructure can't scale.

A side-effect is that there are also flush_database_updates, removeSecurityProxy, clear_current_connetion_cache etc. within buildstatus_OK which will be able to be removed too.

We should also be able to remove "# Quick and dirty hack to carry on on process-upload failures" which puts uploads into the failed-to-move folder when Popen returns and the folder hasn't moved.

See original description

Related branches

lp:~jelmer/launchpad/simplify-uploadprocess
Rejected for merging into lp:launchpad
Canonical Launchpad Engineering: Pending requested
Diff: 1056 lines (+192/-267)
13 files modified
database/schema/security.cfg (+5/-2)
lib/lp/archiveuploader/tests/__init__.py (+9/-3)
lib/lp/archiveuploader/tests/test_buildduploads.py (+2/-3)
lib/lp/archiveuploader/tests/test_ppauploadprocessor.py (+2/-5)
lib/lp/archiveuploader/tests/test_recipeuploads.py (+2/-3)
lib/lp/archiveuploader/tests/test_securityuploads.py (+3/-7)
lib/lp/archiveuploader/tests/test_uploadprocessor.py (+31/-38)
lib/lp/archiveuploader/uploadprocessor.py (+39/-23)
lib/lp/buildmaster/interfaces/buildbase.py (+6/-5)
lib/lp/buildmaster/model/buildbase.py (+50/-103)
lib/lp/buildmaster/tests/test_buildbase.py (+31/-53)
lib/lp/soyuz/doc/buildd-slavescanner.txt (+4/-20)
lib/lp/soyuz/scripts/soyuz_process_upload.py (+8/-2)
lp:~jelmer/launchpad/506256-remove-popen
Graham Binns (community): Approve (release-critical)
Brad Crittenden (community): Approve (code)
Julian Edwards (community): Approve
Diff: 1176 lines (+222/-405)
17 files modified
lib/canonical/launchpad/webapp/tales.py (+1/-0)
lib/lp/archiveuploader/tests/test_uploadprocessor.py (+53/-24)
lib/lp/archiveuploader/uploadprocessor.py (+25/-9)
lib/lp/buildmaster/enums.py (+7/-3)
lib/lp/buildmaster/interfaces/buildfarmjob.py (+4/-0)
lib/lp/buildmaster/interfaces/packagebuild.py (+2/-20)
lib/lp/buildmaster/model/buildfarmjob.py (+10/-0)
lib/lp/buildmaster/model/packagebuild.py (+35/-153)
lib/lp/buildmaster/tests/test_buildfarmjob.py (+15/-0)
lib/lp/buildmaster/tests/test_packagebuild.py (+29/-67)
lib/lp/code/browser/sourcepackagerecipebuild.py (+1/-0)
lib/lp/code/model/tests/test_sourcepackagerecipebuild.py (+6/-6)
lib/lp/registry/model/sourcepackage.py (+6/-2)
lib/lp/soyuz/browser/archive.py (+1/-0)
lib/lp/soyuz/doc/buildd-slavescanner.txt (+12/-116)
lib/lp/soyuz/model/archive.py (+3/-2)
lib/lp/soyuz/model/binarypackagebuild.py (+12/-3)
lp:~jelmer/launchpad/506256-remove-popen-2
Michael Nelson (community): Approve (code)
Diff: 1585 lines (+374/-273)
27 files modified
database/schema/security.cfg (+2/-0)
lib/lp/archiveuploader/dscfile.py (+0/-29)
lib/lp/archiveuploader/nascentupload.py (+31/-16)
lib/lp/archiveuploader/nascentuploadfile.py (+59/-35)
lib/lp/archiveuploader/tests/__init__.py (+5/-7)
lib/lp/archiveuploader/tests/nascentupload.txt (+4/-5)
lib/lp/archiveuploader/tests/test_buildduploads.py (+7/-8)
lib/lp/archiveuploader/tests/test_nascentuploadfile.py (+61/-0)
lib/lp/archiveuploader/tests/test_ppauploadprocessor.py (+11/-12)
lib/lp/archiveuploader/tests/test_recipeuploads.py (+8/-12)
lib/lp/archiveuploader/tests/test_uploadprocessor.py (+99/-30)
lib/lp/archiveuploader/tests/uploadpolicy.txt (+1/-8)
lib/lp/archiveuploader/uploadpolicy.py (+12/-14)
lib/lp/archiveuploader/uploadprocessor.py (+16/-19)
lib/lp/buildmaster/interfaces/packagebuild.py (+8/-4)
lib/lp/buildmaster/model/packagebuild.py (+11/-2)
lib/lp/buildmaster/tests/test_packagebuild.py (+1/-1)
lib/lp/code/configure.zcml (+1/-5)
lib/lp/code/model/sourcepackagerecipebuild.py (+4/-29)
lib/lp/code/model/tests/test_sourcepackagerecipebuild.py (+6/-0)
lib/lp/soyuz/doc/build-failedtoupload-workflow.txt (+2/-3)
lib/lp/soyuz/doc/buildd-slavescanner.txt (+0/-3)
lib/lp/soyuz/doc/distroseriesqueue-translations.txt (+3/-5)
lib/lp/soyuz/doc/soyuz-set-of-uploads.txt (+3/-20)
lib/lp/soyuz/model/binarypackagebuild.py (+4/-0)
lib/lp/soyuz/scripts/soyuz_process_upload.py (+6/-6)
lib/lp/soyuz/tests/test_binarypackagebuild.py (+9/-0)
lp:~jelmer/launchpad/cp-async-builddmaster (Merged)
lp:~mbp/launchpad/778437-sprb-spam
Benji York (community): Approve (code)
Diff: 160 lines (+64/-41)
3 files modified
lib/lp/archiveuploader/tests/test_uploadprocessor.py (+5/-5)
lib/lp/code/model/sourcepackagerecipebuild.py (+4/-12)
lib/lp/code/model/tests/test_sourcepackagerecipebuild.py (+55/-24)
Michael Nelson (michael.nelson)
description: updated
Julian Edwards (julian-edwards)
Changed in soyuz:
assignee: nobody → Jelmer Vernooij (jelmer)
Revision history for this message
Launchpad QA Bot (lpqabot) wrote : Bug fixed by a commit

Fixed in stable r11566 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11566>.

Changed in soyuz:
milestone: none → 10.10
tags: added: qa-needstesting
Changed in soyuz:
status: Triaged → Fix Committed
Revision history for this message
Robert Collins (lifeless) wrote :

This was only half-landed, and will break things if deployed; we need to land the other half: thats blocked by the OOPS misbehaviour; I have a lead on that and will be following up with it. See https://dev.launchpad.net/QAProcessContinuousRollouts for info on the tag I added. We possibly need to tune what we do with this, but it should stop deploys for now.

tags: added: bad-commit-11566
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r11579 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11579>.

Revision history for this message
Robert Collins (lifeless) wrote :

The CP for this is now in production-stable, but it doesn't include an ancestry reference for 11566 so am tagging this to let the qatagger report move on.

tags: added: qa-ok
removed: bad-commit-11566 qa-needstesting
Curtis Hovey (sinzui)
Changed in soyuz:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.