Debian: build-pkgs -c does not always regenerate *.dsc files

[M. Vefa Bicakci]

Brief Description
-----------------

When a Debian-based StarlingX package makes use of a "dl_hook" and when the dl_files and src_files entries in the meta_data.yaml do /not/ take into account all of the files fed into the build system, then using "build-pkgs -c -p mypkg" does not regenerate the *.dsc and the *.orig.tar.gz files for the package, which may cause older versions of the "untracked" source files to be used.

In these cases, the only way to ensure a fresh build is to remove the *.dsc file from "/localdisk/loadbuild/<user>/<project>/<"std"|"rt">/<package>/*.dsc".

I have prepared a fix for this shortcoming, which is more easily triggered with some of my kernel packaging changes. This is why I am opening this bug.

Severity
--------
Minor: This is a build issue affecting packages whose meta_data.yaml files have a "dl_hook" entry and whose input files are only partially covered by meta_data.yaml.

I did not acquire the number of packages matching this criteria, but my thinking is that this is a minor issue, especially as the official nightly builds are always non-incremental builds (i.e., nightly builds always build from scratch and hence are not affected by this issue).

Note
----
I already have a patch for this, and I am opening this bug report as a placeholder to refer to. This is why this template is incomplete.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/855389
Committed: https://opendev.org/starlingx/root/commit/d7c5e1c8ba6f7871fe7e40ce901805f159e0af8d
Submitter: "Zuul (22348)"
Branch: master

commit d7c5e1c8ba6f7871fe7e40ce901805f159e0af8d
Author: M. Vefa Bicakci <email address hidden>
Date: Fri Aug 26 18:37:53 2022 +0000

    stx: build-pkgs: Ensure that -c rebuilds packages

    Prior to this commit, the "build-pkgs -c -p <pkg>" command would not
    rebuild the DSC file and the orig.tar.gz archive of packages (especially
    those that use the "dl_hook" build method) in case the corresponding
    input files tracked by meta_data.yaml had not changed.

    This appears expected at first, but there are use cases which require
    regenerating the DSC file and the orig.tar.gz archive. For example, the
    dl_hook script may make use of directories and/or files that are not
    tracked by the meta_data.yaml file, and it may not be viable to track
    such directories and files using meta_data.yaml.

    As a supporting example, calculating the MD5 hash of all of the files in
    a StarlingX kernel git repository (after local changes to optimize
    memory utilization of build-pkgs) takes about 4.5 minutes in the
    author's build VM, which makes it unviable to track changes in the
    kernel repository using meta_data.yaml due to the excessive time added
    to most build-pkgs invocations.

    Hence, this commit makes build-pkgs regenerate DSC files and orig.tar.gz
    archives when the "-c" option is in use.

    Note that the "-c" option sets the self.attrs['avoid'] member flag of
    the BuildController class to False, which is checked by the change made
    in this commit to skip the input file change detection code, which
    allows forcibly regenerating the aforementioned build artifacts.

    Verification
    - 'build-pkgs -c -p linux' regenerates the *.dsc an *.orig.tar.gz files
      as expected when used with (currently unpublished) packaging changes
      that make use of kernel git repository clones instead of downloaded
      tar archive snapshots.
    - No adverse effects were observed with incremental builds (e.g.,
      "build-pkgs -c -b std,rt -p linux,linux-rt") and builds from scratch
      (e.g., "build-pkgs -c -a") as well as incremental builds carried out
      during the author's day-to-day work in the past three business days.

    Closes-Bug: 1988348
    Change-Id: Ie186aee5ba7391a8278ed54acb6e7492122c02e7
    Signed-off-by: M. Vefa Bicakci <email address hidden>