Debian: Abruptly interrupting build leaves behind an Aptly snapshot backup

[M. Vefa Bicakci]

Brief Description
-----------------
When building Debian-based StarlingX, abruptly interrupting the build system (downloader or build-pkgs scripts) with Ctrl+C (SIGINT) occasionally results in the build system leaving behind an Aptly snapshot backup, such as an Aptly snapshot named "backup-deb-local-binary".

Restarting the build is not able to clean up the backup snapshot due to a bug in build-pkgs, which in turn results in build failures that are not obvious as to how to resolve.

Here is an example error message snippet output by apt-get during the chroot set-up phase of the build, caused by the issue discussed above:

      E: Failed to fetch http://stx-deb-bld-1-stx-repomgr:80/deb-local-binary/dists/bullseye/main/binary-amd64/Packages 404 Not Found [IP: 10.98.96.147 80]
      E: Some index files failed to download. They have been ignored, or old ones used instead.
      W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list:1 and /etc/apt/sources.list.d/sbuild-extra-repositories.list:1
      W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:1 and /etc/apt/sources.list.d/sbuild-extra-repositories.list:1
      E: apt-get update failed

Severity
--------
Minor: Only visible if the build system user interrupts the build abruptly during the build, and only if the build was processing Aptly snapshots at the time of the interruption.

Note
----
I already have a patch for this, and I am opening this bug report as a placeholder to refer to. This is why this template is incomplete.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/852034
Committed: https://opendev.org/starlingx/root/commit/e91e6f77b70dd6718b5f65d5fc5e6ab71f878977
Submitter: "Zuul (22348)"
Branch: master

commit e91e6f77b70dd6718b5f65d5fc5e6ab71f878977
Author: M. Vefa Bicakci <email address hidden>
Date: Tue Apr 19 21:06:29 2022 +0000

    stx: Remove backup snapshot if it already exists

    This commit avoids the issue where downloader and/or build-pkgs are
    interrupted abruptly using Ctrl+C (SIGINT) and leave behind an Aptly
    snapshot named "backup-<snapshot name>" (such as
    "backup-deb-local-binary").

    Restarting the build afterwards does not allow recovery, because the
    snapshot whose backup was left behind cannot be published into an Aptly
    repository, and this causes the corresponding repository to be
    unavailable for the build.

    Here are the error and warning messages reported by apt-get running
    under schroot/sbuild, where the "deb-local-binary" Aptly repository is
    not found:

      E: Failed to fetch http://stx-deb-bld-1-stx-repomgr:80/\
        deb-local-binary/dists/bullseye/main/binary-amd64/Packages \
        404 Not Found [IP: 10.98.96.147 80]
      E: Some index files failed to download. They have been ignored, \
        or old ones used instead.
      W: Target Packages (main/binary-amd64/Packages) is configured \
        multiple times in /etc/apt/sources.list:1 and \
        /etc/apt/sources.list.d/sbuild-extra-repositories.list:1
      W: Target Packages (main/binary-all/Packages) is configured \
        multiple times in /etc/apt/sources.list:1 and \
          /etc/apt/sources.list.d/sbuild-extra-repositories.list:1
      E: apt-get update failed

    The fix for this issue involves checking for the existence of the backup
    Aptly snapshot and removing the pre-existing backup snapshot. This
    allows the build to rename the currently existing normal/non-backup
    snapshot to a backup snapshot. With this change, the __create_snapshot
    method is able to return True, which allows build-pkgs to publish the
    snapshot into a repository, recovering from the aforementioned
    situation.

    Verification:
    - No apparent issues observed when using downloader and when building
      incrementally as well as from scratch many times over the course a few
      weeks.

    - Interrupting downloader or build-pkgs abruptly no longer results in
      the inability to remove backup snapshots.

      This issue is admittedly not straightforward to reproduce by manually
      interrupting downloader/build-pkgs due to the timing aspect, but the
      issue can be simulated by creating a backup snapshot ahead of time as
      follows:

        stx shell --container repomgr \
          -c 'aptly snapshot create backup-deb-local-binary \
              from repo deb-local-binary'

      Afterwards, run the following:

        stx shell -c 'sudo apt update; downloader -b -s -c'

      Finally, run the following and observe that "apt update" fails to
      locate the "deb-local-binary" repository, which is also observed in
      golang-1.17's "sb...