Upload to PPA from bb-cloud-images jenkaas nodes taking over an hour per upload

[Philip Roche]

As requested in internal Canonical RT https://rt.admin.canonical.com/Ticket/Display.html?id=149592 creating bug due to very slow uploads to PPA.

https://rt.admin.canonical.com/Ticket/Display.html?id=149592 is internal to Canonical but the relevant contents of the RT are:

We have a jenkins job https://jenkins.canonical.com/bb-cloud-images/job/18.04-Prepare-Secret-Sauce-Livecd-Rootfs/ which builds packages and uploads to private PPA https://launchpad.net/~cloudware/+archive/ubuntu/cpc-livecd-rootfs. We have recently noticed that uploads to the PPA take about an hour for each upload.

```
22:10:21 Uploading to ppa-secret-sauce (via sftp to ppa.launchpad.net):
22:10:21 livecd-rootfs_2.525.58~202204182110.upstream-7bef05d.cpc-cef6388.maas-aa52770.dsc: done.
22:10:22 livecd-rootfs_2.525.58~202204182110.upstream-7bef05d.cpc-cef6388.maas-aa52770.tar.xz: done.
22:10:22 livecd-rootfs_2.525.58~202204182110.upstream-7bef05d.cpc-cef6388.maas-aa52770_source.changes: Connection to ppa.launchpad.net closed by remote host.
23:10:22 done.
23:10:22 Successfully uploaded packages.
```

Do you have any ideas why this is taking so long or how we can debug? These are not large packages (fully build is only 256.5 KiB).

Nvidia driver PPA uploads are also taking the same about of time so it isn't bb-cloud-images specific.

See https://stable-cloud-images-jenkins.canonical.com/job/22.04-Minimal-Nvidia-Driver-Build-Part-1-nvidia-modules/default/20/console

```
13:29:53 Uploading to ppa (via sftp to ppa.launchpad.net):
13:29:53 nvidia-modules_0.68+22.04-5.15-470-82.dsc: done.
13:29:54 nvidia-modules_0.68+22.04-5.15-470-82.tar.xz: done.
13:29:54 nvidia-modules_0.68+22.04-5.15-470-82_source.changes: done.
14:29:54 Connection to ppa.launchpad.net closed by remote host.
14:29:54 Successfully uploaded packages.
```

We did not see these issues on https://jenkins.canonical.com/devel-cloud-images

See https://jenkins.canonical.com/devel-cloud-images/job/22.04-Minimal-Nvidia-Driver-Build-Part-1-nvidia-modules/default/6/console

```
15:51:58 Uploading to ppa (via ftp to ppa.launchpad.net):
15:51:58 Uploading nvidia-modules_0.68+22.04-5.15-470-80.dsc: done.
15:51:58 Uploading nvidia-modules_0.68+22.04-5.15-470-80.tar.xz: done.
15:51:58 Uploading nvidia-modules_0.68+22.04-5.15-470-80_source.buildinfo: done.
15:51:58 Uploading nvidia-modules_0.68+22.04-5.15-470-80_source.changes: done.
15:51:58 Successfully uploaded packages.
```

Could it be SFTP vs FTP upload related?

Note that some of the links above will not be accessible to non Canonical folk.

[Colin Watson (cjwatson)]

Please do try FTP if you can. It exercises sufficiently different code paths in txpkgupload that there's a good chance it'll avoid this particular bug.

[Thomas Bechtold (toabctl)]

Looks like **not** using a dput profile and instead just doing

dput ppa:xxxxx *.changes

just works. our dput.cf profile looked like:

cat << DPUT_CONFIG_EOF > $HOME/.dput.cf
-[my-profile]
-method = sftp
-login = my-lp-username
-fqdn = ppa.launchpad.net
-incoming = ppa:cloudware/xxxxxx
-allow_unsigned_uploads = 0
-DPUT_CONFIG_EOF