The main problem is that the blobs to sign are large. Cryptographically speaking, we might be able to avoid this by only sending the hash, since that's what's actually signed; but the tools we're calling don't generally support that, and in any case that would only work for detached signatures.
This is a little speculative, but I think there's a good chance we could improve this by changing the way we send blobs to the signing service. At the moment they're sent in the request body, base64-encoded because the request body is JSON, and the request body is encrypted and authenticated using NaCl. For large blobs, the encoding/decoding and encryption/decryption are going to take a while, probably enough to make a significant difference here given the large amount of time being spent in bulk. This probably needs some performance testing, but I think we'd get a substantial win by having the caller ensure that the blobs to sign are in the librarian (as restricted objects with very limited visibility) and then sending a librarian URL with a suitable token and the checksum of its contents instead.
The main problem is that the blobs to sign are large. Cryptographically speaking, we might be able to avoid this by only sending the hash, since that's what's actually signed; but the tools we're calling don't generally support that, and in any case that would only work for detached signatures.
This is a little speculative, but I think there's a good chance we could improve this by changing the way we send blobs to the signing service. At the moment they're sent in the request body, base64-encoded because the request body is JSON, and the request body is encrypted and authenticated using NaCl. For large blobs, the encoding/decoding and encryption/ decryption are going to take a while, probably enough to make a significant difference here given the large amount of time being spent in bulk. This probably needs some performance testing, but I think we'd get a substantial win by having the caller ensure that the blobs to sign are in the librarian (as restricted objects with very limited visibility) and then sending a librarian URL with a suitable token and the checksum of its contents instead.