407 Proxy Authentication errors from snap build on s390x and PPC64el

[Adam Collard]

Some recent LP snap builds of MAAS on PPC64el and s390x failed with a HTTP 407 trying to fetch libpython3-dev from ftpmaster

https://launchpadlibrarian.net/597662141/buildlog_snap_ubuntu_focal_ppc64el_maas-master_BUILDING.txt.gz

has

E: Failed to fetch http://ftpmaster.internal/ubuntu/pool/main/p/python3-defaults/libpython3-dev_3.8.2-0ubuntu2_ppc64el.deb 407 Proxy Authentication Required [IP: 10.10.10.1 8222]

and https://launchpadlibrarian.net/597662144/buildlog_snap_ubuntu_focal_s390x_maas-master_BUILDING.txt.gz has:

E: Failed to fetch http://ftpmaster.internal/ubuntu/pool/main/p/python3-defaults/libpython3-dev_3.8.2-0ubuntu2_s390x.deb 407 Proxy Authentication Required [IP: 10.10.10.1 8222]

[Colin Watson (cjwatson)]

There's not much obvious from logs, but my best guess is that this may be because of bad template substitution in a charm (that I fixed a little while ago in https://code.launchpad.net/~cjwatson/rutabaga/+git/rutabaga/+merge/418939, but hadn't got round to rolling out to production yet) that means that the builder proxy authenticator's database of valid tokens has been bloating without bound ever since it was installed, which is somewhat likely to cause random failures like this.

I've filed https://portal.admin.canonical.com/C149735 to get the fix rolled out. That should also give us better access to Squid logs, giving us a better chance of debugging future problems of this kind.

[Adam Collard (adam-collard)]

Hit again on https://launchpadlibrarian.net/597887356/buildlog_snap_ubuntu_focal_arm64_maas-master_BUILDING.txt.gz (this time npm, so added confidence that it's not particular to the resource being fetched)

npm ERR! code E407
npm ERR! 407 Proxy Authentication Required - GET https://registry.npmjs.org/find-up

[22/Apr/2022:08:16:17 +0000] "CONNECT registry.npmjs.org:443 HTTP/1.1" 200 9604 "-" "-"

[Colin Watson (cjwatson)]

While reducing cruft in the DB seemed to help a bit, we're generally seeing `sqlite3.OperationalError: database is locked` at about the same time as these errors. I think we really need to finally get round to converting rutabaga to postgresql.