consistent timeouts of lp built snaps uploading to store

[kevin gunn]

unity8-session snap in the store is currently on a quite old version, even though our current launchpad project is set to build and upload regularly.
It fails upload with:
"Store upload failed: timeout exceeded."
As an example
https://code.launchpad.net/~unity-team/+snap/unity8-session-silo/+build/12633

I understand this is happening with other snaps (i.e. inkscape)

not sure which side is having the issue, store or lp.

this is going to become an issue as we go wider and external with the unity8 snap and application snap work.

[Michael Terry (mterry)]

Some more facts:

- The snap is about 380MB

- The LP side doesn't seem particularly interesting. From cjwatson:
  """
  [2016-12-06 19:19:51,073: INFO/Worker-3] Starting new HTTPS connection (1): upload.apps.ubuntu.com
  [2016-12-06 19:24:49,265: WARNING/MainProcess] Soft time limit (300.0s) exceeded for lp.services.job.celeryjob.CeleryRunJobIgnoreResult[SnapStoreUploadJob_34011252_4f15615a-3667-49ca-8641-830274abb55b]
  [2016-12-06 19:24:49,265: ERROR/Worker-3] Failure in _doOops: SoftTimeLimitExceeded()
  (with a few uninteresting things from the same worker in between, basically just transaction management waffling)
  the traceback is otherwise uninformative
  """

- The CUD logs are also not helpful. From thomi:
  "I've completely failed to find anything interesting in the CUD logs. <snip> I couldn't find any POST in the CUD logs that remotely matched that timestamp so I wonder if this was something 'earlier' than the app units"

- Ted reports that sometimes just retrying the upload in LP works. I haven't seen that myself, but apparently it can work.

- I've even seen LP get stuck in the "uploading" state, never noticing a timeout. For example, https://launchpad.net/~unity-team/+snap/unity8-session-silo/+build/12636 -- as of the time of this writing -- is stuck at " Store upload in progress"

[Colin Watson (cjwatson)]

It seems reasonably clear that Launchpad isn't directly at fault here, other than enforcing what really should be a fairly reasonable timeout (five minutes). Our best guess at the moment is that there's a network fault at the PS4.5 border; there's an RT although I don't have the number handy at the moment.

[Bret Barker (noise)]

Yes, this is most likely being caused by the network issues covered in RT #97941 and actively being worked on by IS.

[William Grant (wgrant)]

Has this been seen recently? PS4½ networking should have been pretty happy for the last couple of weeks.

[Michael Terry (mterry)]

It seems to be better indeed. Thanks all!