StarlingX

  1. Bug #1997568
  2. Activity log

Activity log for bug #1997568

Date Who What changed Old value New value Message
2022-11-23 15:03:29 Scott Little bug added bug
2022-11-23 15:11:17 Scott Little description Brief Description ----------------- CENGN's overnight StarlingX builds have failed semi-regularly on network download issues. The problem usually strikes in the 2:00-3:00 am Eastern time window when accessing snapshot.debian.org to download files. Recent build failures were Nov 18 and Nov 22. We have seen issues with both construction of build containers and during the download step. CENGN IT claims no network issues on those time windows, nor does the build servers connectivity audit show any issues with general network access. Whatever the issue, it is closer to the debian side. We need to look into adding retries and/or falling back to an alternative mirror for these, if not all downloads. Severity -------- Minor: Steps to Reproduce ------------------ Scenario 1) stx-init-env --rebuild Scenario 2) Expected Behavior ------------------ Build failes Actual Behavior ---------------- Build passes Reproducibility --------------- Intermittent: Perhaps 20% of builds when triggered between 2-3 am Eastern System Configuration -------------------- <One node system, Two node system, Multi-node system, Dedicated storage, https, IPv4, IPv6 etc.> Branch/Pull Time/Commit ----------------------- Branch and the time when code was pulled or git commit or cengn load info Last Pass --------- Did this test scenario pass previously? If so, please indicate the load/pull time info of the last pass. Use this section to also indicate if this is a new test scenario. Timestamp/Logs -------------- Scenario 1) ./stx-init-env --rebuild ... 02:20:35 Step 5/19 : RUN cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb 02:20:35 ---> Running in aa077b054066 02:20:35 --2022-11-16 07:20:32-- http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb 02:20:35 Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27, 185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ... 02:20:35 Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected. 02:21:43 HTTP request sent, awaiting response... 503 Backend fetch failed 02:21:43 2022-11-16 07:21:32 ERROR 503: Backend fetch failed. 02:21:43 02:21:43 The command '/bin/sh -c cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb' returned a non-zero code: 8 Scenario 2) downloader -b -s -B std,rt ... 02:38:45 2022-11-21 07:38:45,386 - repo_manager - DEBUG: upload_pkg: package /import/mirrors/starlingx/binaries/firmware-linux_20210315-3_all.deb been uploaded into deb-local-binary 02:38:45 2022-11-21 07:38:45,386 - downloader - INFO: firmware-linux_20210315-3 is uploaded to deb-local-binary 02:38:45 2022-11-21 07:38:45,386 - downloader - DEBUG: package firmware-misc-nonfree_20210315-3 needs to be downloaded 02:38:45 2022-11-21 07:38:45,386 - downloader - INFO: [ Run - "['rm', '-rf', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" ] 02:38:45 2022-11-21 07:38:45,398 - downloader - INFO: [ Run - "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" ] 02:38:51 2022-11-21 07:38:51,907 - downloader - DEBUG: stderr: % Total % Received % Xferd Average Speed Time Time Time Current 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: Dload Upload Total Spent Left Speed 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 1 8742k 1 127k 0 0 52361 0 0:02:50 0:00:02 0:02:48 52340 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 13 8742k 13 1152k 0 0 321k 0 0:00:27 0:00:03 0:00:24 321k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 24 8742k 24 2173k 0 0 482k 0 0:00:18 0:00:04 0:00:14 482k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 35 8742k 35 3069k 0 0 515k 0 0:00:16 0:00:05 0:00:11 662k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 38 8742k 38 3344k 0 0 515k 0 0:00:16 0:00:06 0:00:10 769k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 38 8742k 38 3368k 0 0 518k 0 0:00:16 0:00:06 0:00:10 809k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: curl: (18) transfer closed with 5503728 bytes remaining to read 02:38:51 2022-11-21 07:38:51,909 - downloader - ERROR: [ Command failed with a non-zero return code: "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" return code: 18 ] 02:38:51 Traceback (most recent call last): 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 586, in <module> 02:38:51 binary_dl.start() 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 400, in start 02:38:51 self.download_list(bin_list) 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 369, in download_list 02:38:51 ret = self.download(debnames[0], debnames[1], url) 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 256, in download 02:38:51 utils.run_shell_cmd(["curl", "-k", "-f", dl_url, "-o", tmp_file], logger) 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/utils.py", line 163, in run_shell_cmd 02:38:52 raise Exception(msg) 02:38:52 Exception: [ Command failed with a non-zero return code: "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" return code: 18 ] Test Activity ------------- Build Workaround ---------- Wait for snapshot.debian.org to come back, and try again. Brief Description ----------------- CENGN's overnight StarlingX builds have failed semi-regularly on network download issues. The problem usually strikes in the 2:00-3:00 am Eastern time window when accessing snapshot.debian.org to download files. Recent build failures were Nov 18 and Nov 22. We have seen issues with both construction of build containers and during the download step. CENGN IT claims no network issues on those time windows, nor does the build servers connectivity audit show any issues with general network access. Whatever the issue, it is closer to the debian side. We need to look into adding retries and/or falling back to an alternative mirror for these, if not all downloads. Severity -------- Minor: Steps to Reproduce ------------------ Scenario 1) stx-init-env --rebuild Scenario 2) downloader -b -s -B std,rt Expected Behavior ------------------ Build passes Actual Behavior ---------------- Build fails Reproducibility --------------- Intermittent: Perhaps 20% of builds when triggered between 2-3 am Eastern System Configuration -------------------- N/A Branch/Pull Time/Commit ----------------------- Branch and the time when code was pulled or git commit or cengn load info Last Pass --------- Did this test scenario pass previously? If so, please indicate the load/pull time info of the last pass. Use this section to also indicate if this is a new test scenario. Timestamp/Logs -------------- Scenario 1) ./stx-init-env --rebuild ... 02:20:35 Step 5/19 : RUN cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb 02:20:35 ---> Running in aa077b054066 02:20:35 --2022-11-16 07:20:32-- http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb 02:20:35 Resolving snapshot.debian.org (snapshot.debian.org)... 193.62.202.27, 185.17.185.185, 2001:630:206:4000:1a1a:0:c13e:ca1b, ... 02:20:35 Connecting to snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80... connected. 02:21:43 HTTP request sent, awaiting response... 503 Backend fetch failed 02:21:43 2022-11-16 07:21:32 ERROR 503: Backend fetch failed. 02:21:43 02:21:43 The command '/bin/sh -c cd /tmp && wget http://snapshot.debian.org/archive/debian/20211128T160803Z/pool/main/d/debootstrap/debootstrap_1.0.126%2Bnmu1_all.deb && dpkg -i debootstrap_1.0.126+nmu1_all.deb' returned a non-zero code: 8 Scenario 2) downloader -b -s -B std,rt ... 02:38:45 2022-11-21 07:38:45,386 - repo_manager - DEBUG: upload_pkg: package /import/mirrors/starlingx/binaries/firmware-linux_20210315-3_all.deb been uploaded into deb-local-binary 02:38:45 2022-11-21 07:38:45,386 - downloader - INFO: firmware-linux_20210315-3 is uploaded to deb-local-binary 02:38:45 2022-11-21 07:38:45,386 - downloader - DEBUG: package firmware-misc-nonfree_20210315-3 needs to be downloaded 02:38:45 2022-11-21 07:38:45,386 - downloader - INFO: [ Run - "['rm', '-rf', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" ] 02:38:45 2022-11-21 07:38:45,398 - downloader - INFO: [ Run - "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" ] 02:38:51 2022-11-21 07:38:51,907 - downloader - DEBUG: stderr: % Total % Received % Xferd Average Speed Time Time Time Current 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: Dload Upload Total Spent Left Speed 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 1 8742k 1 127k 0 0 52361 0 0:02:50 0:00:02 0:02:48 52340 02:38:51 2022-11-21 07:38:51,908 - downloader - DEBUG: stderr: 13 8742k 13 1152k 0 0 321k 0 0:00:27 0:00:03 0:00:24 321k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 24 8742k 24 2173k 0 0 482k 0 0:00:18 0:00:04 0:00:14 482k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 35 8742k 35 3069k 0 0 515k 0 0:00:16 0:00:05 0:00:11 662k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 38 8742k 38 3344k 0 0 515k 0 0:00:16 0:00:06 0:00:10 769k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: 38 8742k 38 3368k 0 0 518k 0 0:00:16 0:00:06 0:00:10 809k 02:38:51 2022-11-21 07:38:51,909 - downloader - DEBUG: stderr: curl: (18) transfer closed with 5503728 bytes remaining to read 02:38:51 2022-11-21 07:38:51,909 - downloader - ERROR: [ Command failed with a non-zero return code: "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" return code: 18 ] 02:38:51 Traceback (most recent call last): 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 586, in <module> 02:38:51 binary_dl.start() 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 400, in start 02:38:51 self.download_list(bin_list) 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 369, in download_list 02:38:51 ret = self.download(debnames[0], debnames[1], url) 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/downloader", line 256, in download 02:38:51 utils.run_shell_cmd(["curl", "-k", "-f", dl_url, "-o", tmp_file], logger) 02:38:51 File "/localdisk/designer/jenkins/debian-master/cgcs-root/build-tools/stx/utils.py", line 163, in run_shell_cmd 02:38:52 raise Exception(msg) 02:38:52 Exception: [ Command failed with a non-zero return code: "['curl', '-k', '-f', 'https://snapshot.debian.org/archive/debian/20220703T151735Z/pool/non-free/f/firmware-nonfree/firmware-misc-nonfree_20210315-3_all.deb', '-o', '/import/mirrors/starlingx/binaries/firmware-misc-nonfree_20210315-3_all.deb.tmp']" return code: 18 ] Test Activity ------------- Build Workaround ---------- Wait for snapshot.debian.org to come back, and try again.
2022-12-17 02:19:25 Ghada Khalil starlingx: importance Undecided Low
2022-12-17 02:19:34 Ghada Khalil tags stx.build