Activity log for bug #1896215

Date Who What changed Old value New value Message
2020-09-18 10:22:42 Iain Lane bug added bug
2020-09-18 10:22:59 Iain Lane bug task added debian-cd
2020-09-18 10:23:42 Iain Lane summary Network failures when downloading the build do not abort the build Network failures when downloading assets from Launchpad are not notified about
2020-10-01 19:10:35 Łukasz Zemczak summary Network failures when downloading assets from Launchpad are not notified about Network failures when downloading assets from Launchpad are not notified about and can result in broken images
2020-10-01 19:13:17 Łukasz Zemczak ubuntu-cdimage: importance Undecided High
2020-10-01 19:13:21 Łukasz Zemczak debian-cd: importance Undecided High
2020-10-01 19:14:26 Łukasz Zemczak description See today's groovy daily, log here: https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu/groovy/daily-live-20200918.log The amd64 image failed to build mv: cannot stat '/srv/cdimage.ubuntu.com/scratch/ubuntu/groovy/daily-live/tmp/groovy-amd64/CD1/casper/filesystem.kernel-generic': No such file or directory make: *** [Makefile:903: /srv/cdimage.ubuntu.com/scratch/ubuntu/groovy/daily-live/tmp/groovy-amd64/bootable-stamp] Error 1 ERROR WHILE BUILDING OFFICIAL IMAGES !! This is because some of the assets failed to be downloaded from Launchpad ===== Downloading live filesystem images ===== Fri Sep 18 08:28:21 UTC 2020 failed: Network is unreachable. failed: Network is unreachable. But there was no notification of this failure to people subscribed to receive them. I think that we fail to bubble download failures up, possibly somewhere around here https://bazaar.launchpad.net/~ubuntu-cdimage/ubuntu-cdimage/mainline/view/head:/lib/cdimage/livefs.py#L651 (we could also do with some backoff/retry logic, maybe in osextras.py/fetch itself) or maybe very slightly later on we should assert that all the files we need are in place. Also, in debian-cd we have, since roughly forever, essentially ignored images failing in favour of continuing to build any other arches https://bazaar.launchpad.net/~ubuntu-cdimage/debian-cd/ubuntu/view/head:/build_all.sh#L110 I wonder if we should revisit this in some way? e.g. continue to build all the arches but store the bad error code and exit with it later on. Or if that's not desirable, get cdimage to check for the output file(s) being present and notify if they're not? See today's groovy daily, log here:   https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu/groovy/daily-live-20200918.log The amd64 image failed to build mv: cannot stat '/srv/cdimage.ubuntu.com/scratch/ubuntu/groovy/daily-live/tmp/groovy-amd64/CD1/casper/filesystem.kernel-generic': No such file or directory make: *** [Makefile:903: /srv/cdimage.ubuntu.com/scratch/ubuntu/groovy/daily-live/tmp/groovy-amd64/bootable-stamp] Error 1 ERROR WHILE BUILDING OFFICIAL IMAGES !! This is because some of the assets failed to be downloaded from Launchpad ===== Downloading live filesystem images ===== Fri Sep 18 08:28:21 UTC 2020 failed: Network is unreachable. failed: Network is unreachable. But there was no notification of this failure to people subscribed to receive them. I think that we fail to bubble download failures up, possibly somewhere around here   https://bazaar.launchpad.net/~ubuntu-cdimage/ubuntu-cdimage/mainline/view/head:/lib/cdimage/livefs.py#L651 (we could also do with some backoff/retry logic, maybe in osextras.py/fetch itself) or maybe very slightly later on we should assert that all the files we need are in place. Also, in debian-cd we have, since roughly forever, essentially ignored images failing in favour of continuing to build any other arches   https://bazaar.launchpad.net/~ubuntu-cdimage/debian-cd/ubuntu/view/head:/build_all.sh#L110 I wonder if we should revisit this in some way? e.g. continue to build all the arches but store the bad error code and exit with it later on. Or if that's not desirable, get cdimage to check for the output file(s) being present and notify if they're not? Worst thing: such an asset download failure can actually result in a successful image build, but with missing assets on the image. As per: https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu-server/groovy/daily-live-20200930.log