Repeated timeouts when trying to accept packages into intrepid-backports

[Scott Kitterman]

See (Error ID: OOPS-1148G1692) for one example. It seems to work about one time in 20. This is blocking work and needs to be addressed immediately.

[Scott Kitterman (kitterman)]

Or maybe even fewer. (Error ID: OOPS-1148D1805)

[Scott Kitterman (kitterman)]

It seems to have been a transient issue, but I think it still ought to be investigated.

[Scott Kitterman (kitterman)]

Happening again:
(Error ID: OOPS-1150E542)
(Error ID: OOPS-1150E543)
(Error ID: OOPS-1150C597)
(Error ID: OOPS-1150F576)
(Error ID: OOPS-1150G579)

And then I gave up.

[Diogo Matsubara (matsubara)]

Celso, can you take a look at this?

[Scott Kitterman (kitterman)]

Happened again.

[16:47:22] <ScottK> It seems I've just lost my ability to accept packages via LP (Error ID: OOPS-1203A1853)
[16:47:23] <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1203A1853
[16:47:44] <ScottK> This is somewhat bad timing as we're about a week from a release ....
[16:47:56] <ScottK> I'd appreciate it if someone would take a look.
[16:48:37] <cody-somerville> What page gives that oops?
[16:48:55] --> thumper (n=[e-mail]) has joined #launchpad
[16:49:26] <ScottK> https://launchpad.net/ubuntu/jaunty/+queue?queue_state=1
[16:49:35] <ScottK> Right after I mash the accept button.
[16:50:25] <-- gianmt (<email address hidden>) has quit ("Leaving")
[16:50:39] <cody-somerville> Its a time out
[16:50:44] <ScottK> (Error ID: OOPS-1203D1950) if multiple copies help.
[16:50:45] <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1203D1950
[16:50:52] <ScottK> cody-somerville: I'm aware of this.
[16:50:59] <ScottK> It's an internal problem though.
[16:51:05] <ScottK> It's happened before.
[16:54:17] <cody-somerville> ScottK, is it happening for all packages you try to accept or just a specific one?
[16:55:21] <ScottK> Bug #332529 is similar
[16:55:22] <ubottu> Launchpad bug 332529 in soyuz "Repeated timeouts when trying to accept packages into intrepid-backports" [High,Triaged] https://launchpad.net/bugs/332529
[16:55:35] <ScottK> There's only one that needs accepting right now.
[16:55:40] <-- magcius (<email address hidden>) has quit (Connection timed out)
[16:56:22] <-- BjornT (<email address hidden>) has quit (Read error: 110 (Connection timed out))
[16:57:06] <-- dominiks (<email address hidden>) has quit ("...")
[16:57:15] --> luke-jr_ (<email address hidden>) has joined #launchpad
[16:58:51] <ScottK> It finally went through.

[Julian Edwards (julian-edwards)]

Sorry about this, it was caused by accepting a package that had closed a bug which had a lot of subscribers. A similar optimisation was done elsewhere to make that process quicker so we'll fix this page too.

[Björn Tillenius (bjornt)]

On Mon, Apr 20, 2009 at 10:57:51AM -0000, Julian Edwards wrote:
> Sorry about this, it was caused by accepting a package that had closed a
> bug which had a lot of subscribers. A similar optimisation was done
> elsewhere to make that process quicker so we'll fix this page too.

The fix was done in the single place that inserted bug notification
recipients, so things should be better on edge already. I'm not sure
that fix is enough to stop the page from timing out, though.

[Celso Providelo (cprov)]

Looking at https://devpad.canonical.com/~matsubara/oops.cgi/2009-04-17/A1853 you can see that the major problem are the SPR queries issues in SPR.createBuild() for having a better build-ETA.

I don't know exactly why/how a SPR.select(SPN.name='foobar') takes ~500 ms, but that code is executed 8 times (number of architectures) which certainly pushes sql-time above the timeout threshold.

[Scott Kitterman (kitterman)]

Still happens on edge: (Error ID: OOPS-1213EC246)

[Scott Kitterman (kitterman)]

Also on edge (in case you need more than one):

 (Error ID: OOPS-1213EB1171)
 (Error ID: OOPS-1213EB1172)