Debian: build-pkgs uses wget to download tarballs and suffers from time-outs due to firewalls

[M. Vefa Bicakci]

Brief Description
-----------------
Debian-based StarlingX build system makes use of wget to download source code tar archives (tarballs). Furthermore, the default mirrors for source code tar archives was recently changed from the upstream mirrors to the CENGN mirror.

As encountered in the past with https://bugs.launchpad.net/starlingx/+bug/1950017 there is an issue with the use of wget that affects users attempting to download from the CENGN while behind certain security-related network traffic deep inspection equipment. In such scenarios, wget times out after 15 minutes of idle time while downloading from the CENGN mirror.

One option to resolve this issue is to use curl instead of wget, as was done for the CentOS-based StarlingX build system for bug 1950017 (the bug linked above).

Severity
--------
Minor: Kubernetes tar archive downloads take a bit more than 15 minutes due to wget's interaction with the network security equipment. The downloads still succeed though.

Logs depicting the issue
------------------------
Note the jump from 19:27 to 19:42 in the logs below. This occurs due to wget encountering a time-out after 15 minutes and restarting the download afterwards.

2022-08-29 19:27:21,361 - debrepack - INFO: === Debian Package Name: kubernetes-1.21.8
2022-08-29 19:27:21,362 - debrepack - INFO: === Package Version: 1.21.8
2022-08-29 19:27:21,397 - debrepack - INFO: Download http://mirror.starlingx.cengn.ca:80/mirror/debian/github.com/kubernetes/kubernetes/archive/refs/tags/v1.21.8.tar.gz to kubernetes-1.21.8.tar.gz
2022-08-29 19:27:21,398 - debrepack - INFO: [ Run - "wget -t 5 --wait=15 http://mirror.starlingx.cengn.ca:80/mirror/debian/github.com/kubernetes/kubernetes/archive/refs/tags/v1.21.8.tar.gz -O kubernetes-1.21.8.tar.gz" ]
2022-08-29 19:42:29,449 - debrepack - INFO: [ Run - "sha256sum kubernetes-1.21.8.tar.gz |cut -d" " -f1" ]
2022-08-29 19:42:29,732 - debrepack - DEBUG: b585d37fb145de9b91cc934669e64ffd0743f40298a2de970509ac182c70a67a

Note
----
I already have a patch for this, and I am opening this bug report as a placeholder to refer to. This is why this template is incomplete.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/855390
Committed: https://opendev.org/starlingx/root/commit/3ad5c3096a1f121c098ced0a581c8d7915ac9b0d
Submitter: "Zuul (22348)"
Branch: master

commit 3ad5c3096a1f121c098ced0a581c8d7915ac9b0d
Author: M. Vefa Bicakci <email address hidden>
Date: Mon Aug 29 21:47:51 2022 +0000

    stx: Fix-up for Kubernetes tarball download hang

    Certain corporate firewalls inspect Kubernetes source tar archive
    downloads when downloading from mirror.starlingx.cengn.ca with plain
    HTTP, which appears to cause wget to encounter its default 15-minute
    read time-out due to a stalled download. While the version of wget in
    the Debian-based build container does recover successfully from the
    time-out, the 15-minute-long time-out is still undesirable.

    This commit resolves the same issue resolved by commit ac49ff342c71
    ("use curl + avoid partial downloads") in the StarlingX tools
    repository, the code review for which is at:
      https://review.opendev.org/c/starlingx/tools/+/817049

    The change identifier for that commit is:
      Iaa89009ce23efe5b73ecb8163556ce6db932028b

    The aforementioned commit was CentOS-based StarlingX, whereas this
    commit is for Debian-based StarlingX.

    Here are the logs (with additional line-breaks) from build-pkgs'
    debrepack module depicting this issue. Note the 15 minute jump from
    19:27:21 to 19:42:29, indicating that wget encounters a read time-out.

    2022-08-29 19:27:21,361 - debrepack - INFO: \
      === Debian Package Name: kubernetes-1.21.8
    2022-08-29 19:27:21,362 - debrepack - INFO: \
      === Package Version: 1.21.8
    2022-08-29 19:27:21,397 - debrepack - INFO: \
      Download http://mirror.starlingx.cengn.ca:80/\
        mirror/debian/github.com/kubernetes/kubernetes/\
        archive/refs/tags/v1.21.8.tar.gz to kubernetes-1.21.8.tar.gz
    2022-08-29 19:27:21,398 - debrepack - INFO: \
      [ Run - "wget -t 5 --wait=15 \
        http://mirror.starlingx.cengn.ca:80/\
        mirror/debian/github.com/kubernetes/\
        kubernetes/archive/refs/tags/v1.21.8.tar.gz \
        -O kubernetes-1.21.8.tar.gz" ]
    2022-08-29 19:42:29,449 - debrepack - INFO: \
      [ Run - "sha256sum kubernetes-1.21.8.tar.gz |cut -d" " -f1" ]
    2022-08-29 19:42:29,732 - debrepack - DEBUG: \
      b585d37fb145de9b91cc934669e64ffd0743f40298a2de970509ac182c70a67a

    One option to resolve this issue would be to use wget's --read-timeout
    option and set the 15-minute time-out to a smaller value such as 5
    seconds. This commit opts to use curl instead to avoid the time-out
    altogether; curl does not appear to exhibit the same issue. This is
    also the approach taken by the aforementioned commit in the StarlingX
    tools repository.

    Verification

    - The following command line:

        stx shell -c "downloader -c -b -s && build-pkgs -c -a"

      (which forces a fresh download of all source and binary archives) does
      not exhibit any issues, including the Kubernetes tar archive download
      time-out, and the following build-pkgs command to build all packages...