Build failures due to snapshot.debian.org

[Scott Little]

Brief Description
-----------------
CENGN's overnight StarlingX builds have failed semi-regularly on network download issues.
The problem usually strikes in the 2:00-3:00 am Eastern time window when accessing snapshot.debian.org to download files. Recent build failures were Nov 18 and Nov 22.

We have seen issues with both construction of build containers and during the download step.

CENGN IT claims no network issues on those time windows, nor does the build servers connectivity audit show any issues with general network access. Whatever the issue, it is closer to the debian side.

We need to look into adding retries and/or falling back to an alternative mirror for these, if not all downloads.

Severity
--------
Minor:

Steps to Reproduce
------------------
Scenario 1)
stx-init-env --rebuild

Scenario 2)
downloader -b -s -B std,rt

Expected Behavior
------------------
Build passes

Actual Behavior
----------------
Build fails

Reproducibility
---------------
Intermittent: Perhaps 20% of builds when triggered between 2-3 am Eastern

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
Branch and the time when code was pulled or git commit or cengn load info

Last Pass
---------
Did this test scenario pass previously? If so, please indicate the load/pull time info of the last pass.
Use this section to also indicate if this is a new test scenario.

Timestamp/Logs
--------------
Scenario 1)
./stx-init-env --rebuild
...
02:20:35 Step 5/19 : RUN cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb
02:20:35 ---> Running in aa077b054066
02:20:35 --2022-11-16 07:20:32-- http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb
02:20:35 Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27, 185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ...
02:20:35 Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected.
02:21:43 HTTP request sent, awaiting response... 503 Backend fetch failed
02:21:43 2022-11-16 07:21:32 ERROR 503: Backend fetch failed.
02:21:43
02:21:43 The command '/bin/sh -c cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb' returned a non-zero code: 8

Scenario 2)
downloader -b -s -B std,rt
...
02:38:45 2022-11-21 07:38:45,386 - repo_manager - DEBUG: upload_pkg: package /import/mirrors/starlingx/binaries/firmware-linux_20210315-3_all.deb been uploaded into deb-local-binary
02:38:45 2022-11-21 07:38:45,386 - downloader - INFO: firmware-linux_20210315-3 is uploaded to deb-local-binary
02:38:45 2022-11-21 07:38:45,386 - downloader - DEBUG: package firmware-misc-nonfree_20210315-3 needs to be downloaded
02:38:45 2022-11-21 07:38:45,386 - downloader - INFO: [ Run - "['rm', '-rf', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" ]
02:38:45 2022-11-21 07:38:45,398 - downloader - INFO: [ Run - "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" ]
02:38:51 2022-11-21 07:38:51,907 - downloader - DEBUG: stderr: % Total % Received % Xferd Average Speed Time Time Time Current
02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: Dload Upload Total Spent Left Speed
02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 1 8742k 1 127k 0 0 52361 0 0:02:50 0:00:02 0:02:48 52340
02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 13 8742k 13 1152k 0 0 321k 0 0:00:27 0:00:03 0:00:24 321k
02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 24 8742k 24 2173k 0 0 482k 0 0:00:18 0:00:04 0:00:14 482k
02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 35 8742k 35 3069k 0 0 515k 0 0:00:16 0:00:05 0:00:11 662k
02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 38 8742k 38 3344k 0 0 515k 0 0:00:16 0:00:06 0:00:10 769k
02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 38 8742k 38 3368k 0 0 518k 0 0:00:16 0:00:06 0:00:10 809k
02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: curl: (18) transfer closed with 5503728 bytes remaining to read
02:38:51 2022-11-21 07:38:51,909 - downloader - ERROR: [ Command failed with a non-zero return code: "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" return code: 18 ]
02:38:51 Traceback (most recent call last):
02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 586, in <module>
02:38:51 binary_dl.start()
02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 400, in start
02:38:51 self.download_list(bin_list)
02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 369, in download_list
02:38:51 ret = self.download(debnames[0], debnames[1], url)
02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 256, in download
02:38:51 utils.run_shell_cmd(["curl", "-k", "-f", dl_url, "-o", tmp_file], logger)
02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/utils.py", line 163, in run_shell_cmd
02:38:52 raise Exception(msg)
02:38:52 Exception: [ Command failed with a non-zero return code: "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" return code: 18 ]

Test Activity
-------------
Build

Workaround
----------
Wait for snapshot.debian.org to come back, and try again.