ubuntu-cpc parallel builds produce unused files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
livecd-rootfs (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Xenial |
Fix Released
|
Undecided
|
Unassigned | ||
Bionic |
Fix Released
|
Undecided
|
Unassigned | ||
Disco |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Impact]
In parallel builds where a list of image targets are provided, the build
may produce binaries that are not part of the named set of targets but
are created by series dependencies. These implicitly created binaries
may be generated by multiple builds but are unused as our convention for
the ubuntu-cpc project is to only consume binaries from the explicitly
named image targets; this avoids overwriting the same object by multiple
parallel builds. The problem is that as we build out parallel jobs for
a number of image targets (multiplied by each supported architecture)
we're wasting a lot of resources.
[Test Case]
* Specify an image target within the ubuntu-cpc project for a build
* Observe artifacts in addition to the ones for the named image target are produced and downloaded
[Regression Potential]
* An artifact from an explicitly named artifact is missing and builds will begin to remove desired output. Testing is being done to address this regression.
[Other Info]
* This has been running in production in Eoan for a few weeks and we're satisfied with the results. We have seen overall time reduced by 20-50% on some builds.
[Original Description]
In parallel builds where a list of image targets are provided, the build
may produce binaries that are not part of the named set of targets but
are created by series dependencies. These implicitly created binaries
may be generated by multiple builds but are unused as our convention for
the ubuntu-cpc project is to only consume binaries from the explicitly
named image targets; this avoids overwriting the same object by multiple
parallel builds. The problem is that as we build out parallel jobs for
a number of image targets (multiplied by each supported architecture)
we're wasting a lot of resources.
Related branches
- Łukasz Zemczak: Approve
- Philip Roche (community): Approve
-
Diff: 276 lines (+117/-11)12 files modifieddebian/changelog (+6/-0)
live-build/auto/build (+5/-0)
live-build/ubuntu-cpc/hooks.d/base/series/disk-image (+7/-0)
live-build/ubuntu-cpc/hooks.d/base/series/qcow2 (+1/-0)
live-build/ubuntu-cpc/hooks.d/base/series/root-dir (+0/-2)
live-build/ubuntu-cpc/hooks.d/base/series/squashfs (+2/-2)
live-build/ubuntu-cpc/hooks.d/base/series/tarball (+2/-2)
live-build/ubuntu-cpc/hooks.d/base/series/vagrant (+1/-0)
live-build/ubuntu-cpc/hooks.d/base/series/vmdk (+2/-0)
live-build/ubuntu-cpc/hooks.d/base/series/wsl (+2/-0)
live-build/ubuntu-cpc/hooks.d/make-hooks (+48/-5)
live-build/ubuntu-cpc/hooks.d/remove-implicit-artifacts (+41/-0)
- Łukasz Zemczak: Approve
- Philip Roche (community): Approve
-
Diff: 282 lines (+123/-11)12 files modifieddebian/changelog (+6/-0)
live-build/auto/build (+5/-0)
live-build/ubuntu-cpc/hooks.d/base/series/disk-image (+9/-0)
live-build/ubuntu-cpc/hooks.d/base/series/qcow2 (+3/-0)
live-build/ubuntu-cpc/hooks.d/base/series/root-dir (+0/-2)
live-build/ubuntu-cpc/hooks.d/base/series/squashfs (+2/-2)
live-build/ubuntu-cpc/hooks.d/base/series/tarball (+3/-2)
live-build/ubuntu-cpc/hooks.d/base/series/vagrant (+1/-0)
live-build/ubuntu-cpc/hooks.d/base/series/vmdk (+3/-0)
live-build/ubuntu-cpc/hooks.d/base/series/wsl (+2/-0)
live-build/ubuntu-cpc/hooks.d/make-hooks (+48/-5)
live-build/ubuntu-cpc/hooks.d/remove-implicit-artifacts (+41/-0)
- Łukasz Zemczak: Approve
- Philip Roche (community): Approve
-
Diff: 276 lines (+117/-11)12 files modifieddebian/changelog (+6/-0)
live-build/auto/build (+5/-0)
live-build/ubuntu-cpc/hooks.d/base/series/disk-image (+7/-0)
live-build/ubuntu-cpc/hooks.d/base/series/qcow2 (+1/-0)
live-build/ubuntu-cpc/hooks.d/base/series/root-dir (+0/-2)
live-build/ubuntu-cpc/hooks.d/base/series/squashfs (+2/-2)
live-build/ubuntu-cpc/hooks.d/base/series/tarball (+2/-2)
live-build/ubuntu-cpc/hooks.d/base/series/vagrant (+1/-0)
live-build/ubuntu-cpc/hooks.d/base/series/vmdk (+2/-0)
live-build/ubuntu-cpc/hooks.d/base/series/wsl (+2/-0)
live-build/ubuntu-cpc/hooks.d/make-hooks (+48/-5)
live-build/ubuntu-cpc/hooks.d/remove-implicit-artifacts (+41/-0)
- Łukasz Zemczak: Approve
- Tobias Koch (community): Approve
-
Diff: 289 lines (+116/-12)13 files modifieddebian/changelog (+3/-0)
live-build/auto/build (+5/-0)
live-build/ubuntu-cpc/hooks.d/base/series/disk-image (+6/-0)
live-build/ubuntu-cpc/hooks.d/base/series/kvm (+3/-1)
live-build/ubuntu-cpc/hooks.d/base/series/qcow2 (+1/-0)
live-build/ubuntu-cpc/hooks.d/base/series/root-dir (+0/-2)
live-build/ubuntu-cpc/hooks.d/base/series/squashfs (+2/-2)
live-build/ubuntu-cpc/hooks.d/base/series/tarball (+2/-2)
live-build/ubuntu-cpc/hooks.d/base/series/vagrant (+1/-0)
live-build/ubuntu-cpc/hooks.d/base/series/vmdk (+2/-0)
live-build/ubuntu-cpc/hooks.d/base/series/wsl (+2/-0)
live-build/ubuntu-cpc/hooks.d/make-hooks (+48/-5)
live-build/ubuntu-cpc/hooks.d/remove-implicit-artifacts (+41/-0)
description: | updated |
description: | updated |
tags: | added: id-5d31721934a49770eea1ff43 |
description: | updated |
description: | updated |
I have attached an MP. In testing with the ubuntu-cpc project on eoan, when building the amd64 qcow2 image target we eliminate 1.38GiB in unnecessary transfer and storage. This saves a lot of time, storage, and transit for each parallel build on each architecture (so it adds up fast).