PPA publishing is slow

[Andreas Hasenack]

When uploading a source package to a PPA, the build usually happens very quickly, but the publishing is very slow, taking 1h or more most of the time.

[Jürgen Gmach (jugmac00)]

Hi Andreas,

Did this happen only once or is this a permanent issue?

I am only back from PTO today. Has the publishing been successfully? How long did it take?

If the problem is persistent, I am more than happy to investigate, but sometimes there is just a lot of packages which needs to be published.

[Andreas Hasenack (ahasenack)]

It's persistent, and I heard in conversations that the problem is that there is only one "runner/publisher" for all of launchpad ppas.

[Jürgen Gmach (jugmac00)]

Ok, thanks for clarifying. We are aware of that limitation. That said we are currently not working on that issue.

Thanks for creating this ticket so we can collect some more feedback.

For our reference, could you give me some numbers how long it used to take and how long it takes now?

Thank you.

[Andreas Hasenack (ahasenack)]

It used to be that if I hit the timing right, I could get a package published in a few minutes. In my mind, that timing was "I was lucky to upload right before the cron job ran". But in general, I'd say it took less than 30min. Now it's always over 1h.

[Ravi Kant Sharma (ravi-sharma)]

Thanks for raising this, I experience more than 1hr on a regular basis.

[Bryce Harrington (bryce)]

I was helping Andreas with some slow publication related issues, and he mentioned this bug.

We're curious about what causes the slowness. I perused the codebase (e.g. https://git.launchpad.net/launchpad/tree/lib/lp/archivepublisher/scripts/processaccepted.py) and it looks like it is processing all PPAs that have pending records. So I gather if we have a million PPAs in Launchpad but only 10 that have been actively uploaded to, that would mean it is processing only those 10? Is that correct?

Also, if we have one PPA with only a single package and another with many (already built or obsolete) packages, but each PPA has only a single pending publication, will they both take the same time to process, or will the former take the publisher less time to process? In other words, would it be better for the Launchpad publisher for there to be fewer PPAs overall, or better for the number of items in PPAs to be kept to a minimum? Or does it matter?

A third question: Does the publisher cronjob run PPA publishing in parallel or in series with the primary archive publishing? IOW, does heavy publication activity in the primary archive impact the PPA publishing speed, or are they independent? From what I'm inferring from looking at the cron.publish-copy-archives and cron.publish-ppa scripts at https://git.launchpad.net/launchpad/tree/cronscripts/publishing, I'm guessing they are not run in sequence and thus their performance shouldn't be coupled. Is that correct? If they run in parallel, are they run on separate vm's or in the same one?

More broadly, do you have thoughts (or plans) on how the publication processing time could be reduced for PPAs?