Network failures when downloading assets from Launchpad are not notified about and can result in broken images
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu CD Images |
New
|
High
|
Unassigned | ||
debian-cd |
New
|
High
|
Unassigned |
Bug Description
See today's groovy daily, log here:
https:/
The amd64 image failed to build
mv: cannot stat '/srv/cdimage.
make: *** [Makefile:903: /srv/cdimage.
ERROR WHILE BUILDING OFFICIAL IMAGES !!
This is because some of the assets failed to be downloaded from Launchpad
===== Downloading live filesystem images =====
Fri Sep 18 08:28:21 UTC 2020
failed: Network is unreachable.
failed: Network is unreachable.
But there was no notification of this failure to people subscribed to receive them.
I think that we fail to bubble download failures up, possibly somewhere around here
(we could also do with some backoff/retry logic, maybe in osextras.py/fetch itself)
or maybe very slightly later on we should assert that all the files we need are in place.
Also, in debian-cd we have, since roughly forever, essentially ignored images failing in favour of continuing to build any other arches
https:/
I wonder if we should revisit this in some way? e.g. continue to build all the arches but store the bad error code and exit with it later on. Or if that's not desirable, get cdimage to check for the output file(s) being present and notify if they're not?
Worst thing: such an asset download failure can actually result in a successful image build, but with missing assets on the image. As per:
https:/
summary: |
- Network failures when downloading the build do not abort the build + Network failures when downloading assets from Launchpad are not notified + about |
We just hit the same bug but in a bit of a worse way. This time there was a networking error while building ubuntu-server images that resulted in the image build to SUCCEED but with certain image bits missing as not pulled from LP:
https:/ /people. canonical. com/~ubuntu- archive/ cd-build- logs/ubuntu- server/ groovy/ daily-live- 20200930. log
(in this case it was amd64.modules. squashfs- generic, so no modules on the image!)
I'm modifying the bug description to include this info. I think this should be prioritized properly as it can result in us wasting a lot of time. No feedback, no image build failure, nothing - just a broken image shipped 'successfully'.