lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails fails intermittently/rarely in parallel tests

Bug #993510 reported by Gary Poster on 2012-05-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Francesco Banconi

Bug Description

I can reliably reproduce this error in a Lucid LXC container with ./bin/test --load-list tests.txt if tests.txt contains these two tests:

lp.archivepublisher.tests.test_publish_ftpmaster.TestPublishFTPMasterScript.test_getDirtySecuritySuites_returns_security_suites
lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails

(The test passes by itself.)

======================================================================
FAILURE: lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails
worker ID: worker-2
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: librarian-log: {{{
2012-05-03 00:50:59+0530 [-] Log opened.
2012-05-03 00:50:59+0530 [-] twistd 11.1.0 (/usr/bin/python2.6 2.6.5) starting up.
2012-05-03 00:50:59+0530 [-] reactor class: twisted.internet.pollreactor.PollReactor.
2012-05-03 00:50:59+0530 [-] FileUploadFactory starting on 47202
2012-05-03 00:50:59+0530 [-] Starting factory <lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at 0xb8e338c>
2012-05-03 00:50:59+0530 [-] Site starting on 39654
2012-05-03 00:50:59+0530 [-] Starting factory <twisted.web.server.Site instance at 0xb8e328c>
2012-05-03 00:50:59+0530 [-] FileUploadFactory starting on 36628
2012-05-03 00:50:59+0530 [-] Starting factory <lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at 0xb8e38ec>
2012-05-03 00:50:59+0530 [-] Site starting on 51591
2012-05-03 00:50:59+0530 [-] Starting factory <twisted.web.server.Site instance at 0xb8e3d2c>
2012-05-03 00:50:59+0530 [-] Not using upstream librarian
2012-05-03 00:50:59+0530 [-] daemon ready!
}}}

oops-0: {{{
Oops-Id: OOPS-3aacc5868ee52ab4a348b0c20c017d86
Exception-Type: RaisingJobException
Exception-Value: oops notifying users
Date: 2012-05-02T19:27:56.540247+00:00
Branch: devel
Revision: 15189
URL: None
Duration: -1
Oops-Reporter: T

%3Coops-message-2%3E=%7B%27foo%27: %27bar%27%7D
%3Coops-message-3%3E=Failure in _doOops: oops notifying users
path=/var/lib/buildbot/slaves/slave/lucid-devel/build/bin/test
script_name=publishftpmaster

00000-00003@memcache-set MemcachedLayer__live_test
00079-00081@memcache-set MemcachedLayer__live_test
00181-00183@memcache-set MemcachedLayer__live_test
05340-05352@librarian-connection http://localhost:39654/93/filename-100098
05353-05354@librarian-read http://localhost:39654/93/filename-100098
05357-05362@librarian-connection http://localhost:39654/94/filename-100124
05362-05362@librarian-read http://localhost:39654/94/filename-100124
05727-05733@librarian-connection http://localhost:39654/95/filename-100150
05734-05734@librarian-read http://localhost:39654/95/filename-100150
05736-05741@librarian-connection http://localhost:39654/94/filename-100124
05741-05741@librarian-read http://localhost:39654/94/filename-100124
06131-06139@librarian-connection http://localhost:39654/96/filename-100176
06140-06140@librarian-read http://localhost:39654/96/filename-100176
06142-06147@librarian-connection http://localhost:39654/94/filename-100124
06147-06147@librarian-read http://localhost:39654/94/filename-100124
14629-14640@librarian-connection http://localhost:39654/94/copyme_2.8-1_source.changes
14641-14642@librarian-read http://localhost:39654/94/copyme_2.8-1_source.changes
41333-41342@librarian-connection http://localhost:39654/93/unique-from-factory-py-line3282-101297.pot
41344-41344@librarian-read http://localhost:39654/93/unique-from-factory-py-line3282-101297.pot
43637-43646@librarian-connection http://localhost:39654/93/testing.pot
43647-43647@librarian-read http://localhost:39654/93/testing.pot
43650-43654@librarian-connection http://localhost:39654/94/sr.po
43655-43655@librarian-read http://localhost:39654/94/sr.po
53779-53820@librarian-connection http://localhost:39654/1/netapplet-1.0.0.tar.gz
53821-53821@librarian-read http://localhost:39654/1/netapplet-1.0.0.tar.gz
117810-117822@librarian-connection http://localhost:39654/95/foo_666.dsc
117823-117823@librarian-read http://localhost:39654/95/foo_666.dsc
117824-117824@librarian-read http://localhost:39654/95/foo_666.dsc
163327-163336@librarian-connection http://localhost:39654/95/foo_666.dsc
163337-163337@librarian-read http://localhost:39654/95/foo_666.dsc
163337-163337@librarian-read http://localhost:39654/95/foo_666.dsc
168474-168487@librarian-connection http://localhost:39654/93/hwinfo.xml
168487-168487@librarian-read http://localhost:39654/93/hwinfo.xml
190640-190649@librarian-connection http://localhost:39654/94/boing_1.0_source.changes
190650-190650@librarian-read http://localhost:39654/94/boing_1.0_source.changes
196903-196912@librarian-connection http://localhost:39654/95/foo_1.dsc
196913-196913@librarian-read http://localhost:39654/95/foo_1.dsc
196914-196914@librarian-read http://localhost:39654/95/foo_1.dsc
262559-262596@librarian-connection http://localhost:39654/1/netapplet-1.0.0.tar.gz
262596-262596@librarian-read http://localhost:39654/1/netapplet-1.0.0.tar.gz
273754-273762@librarian-connection http://localhost:39654/93/hwinfo.xml
273763-273763@librarian-read http://localhost:39654/93/hwinfo.xml
285616-285626@librarian-connection http://localhost:39654/93/lazy-building_1.0_source.changes
285626-285626@librarian-read http://localhost:39654/93/lazy-building_1.0_source.changes
335127-335136@librarian-connection http://localhost:39654/93/filename-105666
335137-335137@librarian-read http://localhost:39654/93/filename-105666
358219-358230@librarian-connection http://localhost:39654/100/zeca_1.0.orig.tar.gz
358230-358230@librarian-read http://localhost:39654/100/zeca_1.0.orig.tar.gz
358231-358231@librarian-read http://localhost:39654/100/zeca_1.0.orig.tar.gz
358233-358238@librarian-connection http://localhost:39654/101/zeca_1.0-2.diff.gz
358239-358239@librarian-read http://localhost:39654/101/zeca_1.0-2.diff.gz
358240-358240@librarian-read http://localhost:39654/101/zeca_1.0-2.diff.gz
358243-358249@librarian-connection http://localhost:39654/102/zeca_1.0-2.dsc
358249-358249@librarian-read http://localhost:39654/102/zeca_1.0-2.dsc
358250-358250@librarian-read http://localhost:39654/102/zeca_1.0-2.dsc
358253-358258@librarian-connection http://localhost:39654/95/zeca_1.0.orig.tar.gz
358259-358259@librarian-read http://localhost:39654/95/zeca_1.0.orig.tar.gz
358260-358260@librarian-read http://localhost:39654/95/zeca_1.0.orig.tar.gz
358263-358268@librarian-connection http://localhost:39654/96/zeca_1.0-1.diff.gz
358269-358269@librarian-read http://localhost:39654/96/zeca_1.0-1.diff.gz
358270-358270@librarian-read http://localhost:39654/96/zeca_1.0-1.diff.gz
358272-358278@librarian-connection http://localhost:39654/97/zeca_1.0-1.dsc
358279-358279@librarian-read http://localhost:39654/97/zeca_1.0-1.dsc
358280-358280@librarian-read http://localhost:39654/97/zeca_1.0-1.dsc
381046-381056@librarian-connection http://localhost:39654/93/filename-106394
381057-381057@librarian-read http://localhost:39654/93/filename-106394
386709-386718@librarian-connection http://localhost:39654/93/foo_1.0_source.changes
386719-386719@librarian-read http://localhost:39654/93/foo_1.0_source.changes

Traceback (most recent call last):
  Module lazr.jobrunner.jobrunner, line 199, in runJobHandleError
    job.notifyUserError(e)
  Module lp.services.job.tests.test_runner, line 122, in notifyUserError
    raise RaisingJobException('oops notifying users')
RaisingJobException: oops notifying users
}}}

oops-1: {{{
Oops-Id: OOPS-be9faf3485a50ec7731f2d6dc78dc886
Exception-Type: RaisingJobException
Exception-Value: oops notifying users
Date: 2012-05-02T19:27:56.559661+00:00
Branch: devel
Revision: 15189
Duration: -1
Oops-Reporter: T

%3Coops-message-2%3E=%7B%27foo%27: %27bar%27%7D

Traceback (most recent call last):
  Module lazr.jobrunner.jobrunner, line 199, in runJobHandleError
    job.notifyUserError(e)
  Module lp.services.job.tests.test_runner, line 122, in notifyUserError
    raise RaisingJobException('oops notifying users')
RaisingJobException: oops notifying users
}}}

Traceback (most recent call last):
  File "/var/lib/buildbot/slaves/slave/lucid-devel/build/lib/lp/services/job/tests/test_runner.py", line 366, in test_runJobHandleErrors_oops_generated_user_notify_fails
    self.assertEqual(1, len(self.oopses))
MismatchError: 1 != 2

Related branches

Brad Crittenden (bac) on 2012-05-03
description: updated
Gary Poster (gary) wrote :

We though this was a duplicate of bug 992692 but since the fix for that bug landed, we've seen this error again. The failure looks identical. http://pastebin.ubuntu.com/1001180/ I'm seeing if I can identify a way to replicate this reliably.

Gary Poster (gary) wrote :

I can reliably dupe the failure using --load-list with a file containing only these two tests:

lp.archivepublisher.tests.test_publish_ftpmaster.TestPublishFTPMasterScript.test_script_is_happy_with_no_publications
lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails

Changed in launchpad:
assignee: nobody → Francesco Banconi (frankban)
status: Triaged → In Progress
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r15318 (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15318) by a commit, but not testable.

tags: added: qa-untestable
Changed in launchpad:
status: In Progress → Fix Committed
William Grant (wgrant) on 2012-05-30
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers