Signing of multiple LRM KMOD objects is horribly slow
Bug #2025617 reported by
Andy Whitcroft
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Triaged
|
High
|
Unassigned | ||
lp-signing |
Triaged
|
High
|
Unassigned |
Bug Description
When we perform an LRM respin we sign out a large number of KMOD objects. For each LRM there are 4-5 .kos per version of Nvidia built which is of the order of 11 at the current time. When signing a full set there are 54 LRMs. Overall signing is taking about 2 wall-clock hours for this set and is tripping publisher alerts.
tags: | added: lp-soyuz performance |
Changed in lp-signing: | |
status: | New → Triaged |
importance: | Undecided → High |
Changed in launchpad: | |
status: | New → Triaged |
importance: | Undecided → High |
To post a comment you must log in.
The main problem is that the blobs to sign are large. Cryptographically speaking, we might be able to avoid this by only sending the hash, since that's what's actually signed; but the tools we're calling don't generally support that, and in any case that would only work for detached signatures.
This is a little speculative, but I think there's a good chance we could improve this by changing the way we send blobs to the signing service. At the moment they're sent in the request body, base64-encoded because the request body is JSON, and the request body is encrypted and authenticated using NaCl. For large blobs, the encoding/decoding and encryption/ decryption are going to take a while, probably enough to make a significant difference here given the large amount of time being spent in bulk. This probably needs some performance testing, but I think we'd get a substantial win by having the caller ensure that the blobs to sign are in the librarian (as restricted objects with very limited visibility) and then sending a librarian URL with a suitable token and the checksum of its contents instead.