The buildd-manager should scan each builder individually
Bug #583221 reported by
Julian Edwards
This bug affects 1 person
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Launchpad itself |
Fix Released
|
High
|
Julian Edwards | ||
Bug Description
The buildd-manager currently does a scan of the whole build farm in one fell swoop, every 5 seconds. If there's a failure at any point during the scan, it causes the scan to stop where it is and any remaining transactions are aborted.
Ideally, each builder should be scanned periodically in its own right. The current mass-scan is an artefact of the old slave scanner and the underlying model code.
We can re-architect the buildd-manager slightly so that there's a timer event for each builder and scan each on separately.
Related branches
lp:~julian-edwards/launchpad/buildd-manager-parallel-scan
- Jonathan Lange (community): Approve (code)
- Launchpad code reviewers: Pending requested
-
Diff: 12097 lines (+4672/-2183)193 files modifieddatabase/sampledata/current-dev.sql (+411/-394)
database/sampledata/current.sql (+395/-378)
database/schema/Makefile (+2/-0)
database/schema/comments.sql (+31/-10)
database/schema/patch-2207-64-0.sql (+27/-0)
database/schema/patch-2207-68-0.sql (+22/-0)
database/schema/patch-2207-70-0.sql (+14/-0)
database/schema/patch-2207-73-0.sql (+10/-0)
database/schema/patch-2207-74-0.sql (+18/-0)
database/schema/patch-2207-75-0.sql (+32/-0)
database/schema/patch-2207-76-0.sql (+11/-0)
database/schema/patch-2207-77-0.sql (+7/-0)
database/schema/patch-2207-79-0.sql (+14/-0)
database/schema/security.cfg (+10/-0)
database/schema/trusted.sql (+10/-9)
lib/canonical/database/ftests/script_isolation.py (+2/-2)
lib/canonical/database/ftests/test_isolation.py (+8/-6)
lib/canonical/launchpad/browser/librarian.py (+10/-7)
lib/canonical/launchpad/doc/tales.txt (+1/-1)
lib/canonical/launchpad/doc/webapp-publication.txt (+5/-5)
lib/canonical/launchpad/fields/__init__.py (+2/-28)
lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt (+30/-0)
lib/canonical/launchpad/scripts/garbo.py (+28/-2)
lib/canonical/launchpad/scripts/tests/test_garbo.py (+15/-6)
lib/canonical/launchpad/security.py (+5/-6)
lib/canonical/launchpad/testing/fakepackager.py (+2/-1)
lib/canonical/launchpad/tests/test_libraryfilealias_with_parent.py (+3/-3)
lib/canonical/launchpad/webapp/launchbag.py (+1/-1)
lib/canonical/launchpad/webapp/tests/test_forgiving_vocabulary.py (+28/-0)
lib/canonical/launchpad/webapp/vocabulary.py (+21/-1)
lib/canonical/librarian/client.py (+1/-0)
lib/lp/archivepublisher/domination.py (+4/-1)
lib/lp/archivepublisher/tests/test_dominator.py (+34/-3)
lib/lp/archiveuploader/dscfile.py (+1/-1)
lib/lp/archiveuploader/nascentupload.py (+87/-26)
lib/lp/archiveuploader/nascentuploadfile.py (+11/-19)
lib/lp/archiveuploader/tests/data/suite/debug_1.0-1_broken/debug_1.0-1.dsc (+13/-0)
lib/lp/archiveuploader/tests/data/suite/debug_1.0-1_broken/debug_1.0-1_i386.changes (+31/-0)
lib/lp/archiveuploader/tests/data/suite/debug_1.0-1_broken/debug_1.0-1_source.changes (+26/-0)
lib/lp/archiveuploader/tests/nascentupload-announcements.txt (+20/-19)
lib/lp/archiveuploader/tests/nascentupload-ddebs.txt (+80/-41)
lib/lp/archiveuploader/tests/nascentupload-packageset.txt (+3/-3)
lib/lp/archiveuploader/tests/nascentupload-security-uploads.txt (+2/-2)
lib/lp/archiveuploader/tests/nascentupload.txt (+20/-19)
lib/lp/archiveuploader/tests/test_nascentupload.py (+85/-0)
lib/lp/archiveuploader/tests/test_nascentupload_documentation.py (+8/-3)
lib/lp/archiveuploader/tests/test_recipeuploads.py (+2/-2)
lib/lp/archiveuploader/uploadprocessor.py (+2/-1)
lib/lp/blueprints/browser/sprint.py (+7/-8)
lib/lp/bugs/adapters/bugchange.py (+13/-4)
lib/lp/bugs/browser/bug.py (+27/-7)
lib/lp/bugs/browser/bugattachment.py (+34/-4)
lib/lp/bugs/browser/bugcomment.py (+12/-2)
lib/lp/bugs/browser/bugtarget.py (+5/-0)
lib/lp/bugs/browser/bugtask.py (+6/-5)
lib/lp/bugs/browser/configure.zcml (+6/-1)
lib/lp/bugs/browser/tests/bug-views.txt (+15/-4)
lib/lp/bugs/browser/tests/test_bugattachment_file_access.py (+129/-0)
lib/lp/bugs/browser/tests/test_bugview.py (+3/-2)
lib/lp/bugs/doc/bug-change.txt (+6/-4)
lib/lp/bugs/doc/bug-export.txt (+1/-1)
lib/lp/bugs/doc/bugattachments.txt (+63/-2)
lib/lp/bugs/doc/bugnotification-email.txt (+10/-20)
lib/lp/bugs/interfaces/bug.py (+8/-2)
lib/lp/bugs/interfaces/bugattachment.py (+8/-0)
lib/lp/bugs/interfaces/bugsubscription.py (+4/-3)
lib/lp/bugs/interfaces/bugsupervisor.py (+2/-2)
lib/lp/bugs/interfaces/bugtask.py (+2/-2)
lib/lp/bugs/model/bug.py (+14/-3)
lib/lp/bugs/model/bugactivity.py (+2/-3)
lib/lp/bugs/model/bugattachment.py (+11/-0)
lib/lp/bugs/model/bugsubscription.py (+3/-4)
lib/lp/bugs/model/bugtask.py (+35/-35)
lib/lp/bugs/scripts/bugexport.py (+3/-1)
lib/lp/bugs/stories/bugattachments/10-add-bug-attachment.txt (+1/-1)
lib/lp/bugs/stories/bugattachments/20-edit-bug-attachment.txt (+3/-2)
lib/lp/bugs/stories/bugattachments/xx-attachments-to-bug-report.txt (+6/-2)
lib/lp/bugs/stories/bugattachments/xx-delete-bug-attachment.txt (+4/-2)
lib/lp/bugs/stories/bugs/xx-bug-text-pages.txt (+24/-5)
lib/lp/bugs/stories/guided-filebug/xx-filebug-attachments.txt (+1/-1)
lib/lp/bugs/templates/bug-portlet-attachments.pt (+6/-6)
lib/lp/bugs/templates/bugcomment-box.pt (+2/-2)
lib/lp/bugs/templates/bugtarget-patches.pt (+2/-2)
lib/lp/bugs/tests/test_bugchanges.py (+10/-3)
lib/lp/buildmaster/interfaces/builder.py (+6/-0)
lib/lp/buildmaster/interfaces/packagebuild.py (+0/-1)
lib/lp/buildmaster/manager.py (+312/-238)
lib/lp/buildmaster/model/builder.py (+12/-0)
lib/lp/buildmaster/tests/test_builder.py (+20/-0)
lib/lp/buildmaster/tests/test_manager.py (+211/-132)
lib/lp/code/browser/sourcepackagerecipebuild.py (+4/-4)
lib/lp/code/browser/tests/test_sourcepackagerecipe.py (+15/-14)
lib/lp/code/browser/tests/test_sourcepackagerecipebuild.py (+2/-3)
lib/lp/code/configure.zcml (+1/-1)
lib/lp/code/doc/branch.txt (+13/-10)
lib/lp/code/doc/revision.txt (+26/-18)
lib/lp/code/interfaces/branch.py (+12/-10)
lib/lp/code/interfaces/branchrevision.py (+0/-1)
lib/lp/code/interfaces/branchsubscription.py (+3/-3)
lib/lp/code/interfaces/branchvisibilitypolicy.py (+2/-2)
lib/lp/code/interfaces/sourcepackagerecipe.py (+2/-2)
lib/lp/code/interfaces/sourcepackagerecipebuild.py (+5/-10)
lib/lp/code/mail/sourcepackagerecipebuild.py (+5/-5)
lib/lp/code/mail/tests/test_sourcepackagerecipebuild.py (+2/-3)
lib/lp/code/model/branch.py (+30/-28)
lib/lp/code/model/branchmergeproposal.py (+43/-30)
lib/lp/code/model/branchrevision.py (+30/-17)
lib/lp/code/model/branchsubscription.py (+3/-4)
lib/lp/code/model/branchvisibilitypolicy.py (+2/-3)
lib/lp/code/model/revision.py (+4/-4)
lib/lp/code/model/sourcepackagerecipe.py (+29/-16)
lib/lp/code/model/sourcepackagerecipebuild.py (+51/-75)
lib/lp/code/model/tests/test_branchjob.py (+14/-15)
lib/lp/code/model/tests/test_sourcepackagerecipe.py (+11/-8)
lib/lp/code/model/tests/test_sourcepackagerecipebuild.py (+17/-40)
lib/lp/code/templates/sourcepackagerecipebuild-index.pt (+17/-17)
lib/lp/codehosting/scanner/tests/test_bzrsync.py (+19/-12)
lib/lp/registry/browser/person.py (+7/-10)
lib/lp/registry/browser/team.py (+0/-22)
lib/lp/registry/browser/tests/mailinglist-views.txt (+1/-1)
lib/lp/registry/doc/person.txt (+1/-1)
lib/lp/registry/doc/private-team-roles.txt (+1/-1)
lib/lp/registry/doc/private-team-visibility.txt (+2/-3)
lib/lp/registry/doc/vocabularies.txt (+5/-13)
lib/lp/registry/interfaces/person.py (+14/-47)
lib/lp/registry/interfaces/product.py (+3/-3)
lib/lp/registry/interfaces/productrelease.py (+4/-2)
lib/lp/registry/interfaces/productseries.py (+3/-3)
lib/lp/registry/interfaces/structuralsubscription.py (+13/-7)
lib/lp/registry/model/distribution.py (+3/-3)
lib/lp/registry/model/person.py (+1/-17)
lib/lp/registry/model/product.py (+4/-4)
lib/lp/registry/model/productrelease.py (+2/-2)
lib/lp/registry/model/productseries.py (+3/-4)
lib/lp/registry/model/structuralsubscription.py (+4/-3)
lib/lp/registry/tests/mailinglists_helper.py (+3/-2)
lib/lp/registry/tests/test_person.py (+1/-14)
lib/lp/registry/tests/test_user_vocabularies.py (+1/-1)
lib/lp/registry/vocabularies.py (+2/-4)
lib/lp/scripts/helpers.py (+2/-1)
lib/lp/scripts/utilities/sanitizedb.py (+0/-8)
lib/lp/services/database/prejoin.py (+5/-4)
lib/lp/services/features/__init__.py (+10/-0)
lib/lp/services/features/flags.py (+85/-0)
lib/lp/services/features/model.py (+42/-0)
lib/lp/services/features/scopes.py (+5/-0)
lib/lp/services/features/tests/__init__.py (+2/-0)
lib/lp/services/features/tests/test_db_settings.py (+25/-0)
lib/lp/services/features/tests/test_flags.py (+95/-0)
lib/lp/services/memcache/doc/restful-cache.txt (+1/-1)
lib/lp/soyuz/browser/archive.py (+14/-13)
lib/lp/soyuz/browser/archivesubscription.py (+3/-6)
lib/lp/soyuz/browser/tests/distroseriesqueue-views.txt (+1/-1)
lib/lp/soyuz/configure.zcml (+3/-3)
lib/lp/soyuz/doc/archive-files.txt (+8/-0)
lib/lp/soyuz/doc/binarypackagebuild.txt (+3/-2)
lib/lp/soyuz/doc/build-failedtoupload-workflow.txt (+1/-1)
lib/lp/soyuz/doc/distroarchseriesbinarypackage.txt (+4/-2)
lib/lp/soyuz/doc/distroseriesqueue-ddtp-tarball.txt (+3/-3)
lib/lp/soyuz/doc/distroseriesqueue-debian-installer.txt (+1/-1)
lib/lp/soyuz/doc/distroseriesqueue-dist-upgrader.txt (+4/-4)
lib/lp/soyuz/doc/distroseriesqueue-translations.txt (+1/-1)
lib/lp/soyuz/doc/distroseriesqueue.txt (+2/-2)
lib/lp/soyuz/interfaces/archive.py (+22/-16)
lib/lp/soyuz/interfaces/archivejob.py (+55/-0)
lib/lp/soyuz/interfaces/archivesubscriber.py (+2/-2)
lib/lp/soyuz/interfaces/binarypackagebuild.py (+9/-1)
lib/lp/soyuz/interfaces/binarypackagerelease.py (+5/-1)
lib/lp/soyuz/model/archive.py (+5/-3)
lib/lp/soyuz/model/archivejob.py (+130/-0)
lib/lp/soyuz/model/archivesubscriber.py (+2/-3)
lib/lp/soyuz/model/binarypackagebuild.py (+17/-10)
lib/lp/soyuz/model/binarypackagebuildbehavior.py (+3/-4)
lib/lp/soyuz/model/binarypackagerelease.py (+2/-0)
lib/lp/soyuz/model/copyarchivejob.py (+140/-0)
lib/lp/soyuz/model/publishing.py (+35/-1)
lib/lp/soyuz/scripts/tests/test_queue.py (+1/-1)
lib/lp/soyuz/tests/test_archive.py (+34/-2)
lib/lp/soyuz/tests/test_archivejob.py (+48/-0)
lib/lp/soyuz/tests/test_copyarchivejob.py (+391/-0)
lib/lp/soyuz/tests/test_publishing.py (+64/-6)
lib/lp/testing/_webservice.py (+12/-4)
lib/lp/testing/factory.py (+7/-3)
lib/lp/testing/sampledata.py (+2/-0)
lib/lp/translations/browser/hastranslationimports.py (+7/-3)
lib/lp/translations/interfaces/potemplate.py (+14/-2)
lib/lp/translations/interfaces/translationimportqueue.py (+2/-2)
lib/lp/translations/model/potemplate.py (+29/-2)
lib/lp/translations/model/translationimportqueue.py (+2/-2)
lib/lp/translations/scripts/tests/test_cache_suggestive_templates.py (+119/-0)
lib/lp/translations/stories/importqueue/xx-translation-import-queue.txt (+22/-0)
scripts/close-account.py (+1/-3)
utilities/report-database-stats.py (+76/-21)
| Changed in soyuz: | |
| status: | New → In Progress |
| importance: | Undecided → High |
| assignee: | nobody → Julian Edwards (julian-edwards) |
| tags: | added: buildd-manager |
Revision history for this message
| Launchpad QA Bot (lpqabot) wrote Bug fixed by a commit | #1 |
| Changed in soyuz: | |
| milestone: | none → 10.08 |
| tags: | added: qa-needstesting |
| Changed in soyuz: | |
| status: | In Progress → Fix Committed |
| tags: |
added: qa-ok removed: qa-needstesting |
| Changed in soyuz: | |
| status: | Fix Committed → Fix Released |
To post a comment you must log in.
Fixed in db-stable r9632 <http://