build: downloader: silently ignores errors in some cases

[Davlet Panech]

Brief Description
-----------------

Downloader silently ignores errors while downloading source packages in some cases. See logs below.

Severity
--------
Major

Steps to Reproduce
------------------
Difficult to reproduce - see logs below

Expected Behavior
------------------
Download errors must result in "downloader" exiting with a non-zero status

Actual Behavior
----------------
Downloader exits with 0 even if some downloads failed.

Reproducibility
---------------
Seen once

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
master/2023-10-13

Last Pass
---------
N/A

Timestamp/Logs
--------------

11:37:21 2023-10-13 15:37:21,658 - debrepack - INFO: [ Run - "dget -d http://mirror.starlingx.cengn.ca:80/mirror/debian/snapshot.debian.org/archive/debian/20230927T030735Z/pool/main/l/lldpd/lldpd_1.0.11-1+deb11u2.dsc" ]
11:37:25 2023-10-13 15:37:24,874 - debrepack - DEBUG: stdout: dget: retrieving http://mirror.starlingx.cengn.ca:80/mirror/debian/snapshot.debian.org/archive/debian/20230927T030735Z/pool/main/l/lldpd/lldpd_1.0.11-1+deb11u2.dsc
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stdout: dget: using existing lldpd_1.0.11.orig.tar.gz
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stdout: dget: using existing lldpd_1.0.11.orig.tar.gz.asc
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stdout: dget: retrieving http://mirror.starlingx.cengn.ca:80/mirror/debian/snapshot.debian.org/archive/debian/20230927T030735Z/pool/main/l/lldpd/lldpd_1.0.11-1+deb11u2.debian.tar.xz
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stdout: lldpd_1.0.11-1+deb11u2.dsc:
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stdout: Good signature found
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stdout: validating lldpd_1.0.11.orig.tar.gz
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stdout: validating lldpd_1.0.11.orig.tar.gz.asc
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stdout: All files validated successfully.
11:37:25 2023-10-13 15:37:24,875 - debrepack - DEBUG: stderr: % Total % Received % Xferd Average Speed Time Time Time Current
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: Dload Upload Total Spent Left Speed
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: 100 2473 100 2473 0 0 4639 0 --:--:-- --:--:-- --:--:-- 10991
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: % Total % Received % Xferd Average Speed Time Time Time Current
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: Dload Upload Total Spent Left Speed
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: curl: (22) The requested URL returned error: 404 Not Found
11:37:25 2023-10-13 15:37:24,876 - debrepack - DEBUG: stderr: dget: curl lldpd_1.0.11-1+deb11u2.debian.tar.xz http://mirror.starlingx.cengn.ca:80/mirror/debian/snapshot.debian.org/archive/debian/20230927T030735Z/pool/main/l/lldpd/lldpd_1.0.11-1+deb11u2.debian.tar.xz failed
11:37:25 2023-10-13 15:37:24,877 - debrepack - DEBUG: stderr: skipping lldpd_1.0.11-1+deb11u2.debian.tar.xz (not present)

===========================
This download failed: ttp://mirror.starlingx.cengn.ca:80/mirror/debian/snapshot.debian.org/archive/debian/20230927T030735Z/pool/main/l/lldpd/lldpd_1.0.11-1+deb11u2.debian.tar.xz , yet the script returned 0. This caused build errors down the pipeline.

Note: this file is no longer missing on starlingx mirror

Test Activity
-------------
N/A

Workaround
----------
N/A

[Davlet Panech (dpanech)]

One way to reproduce it is to serve a dsc file from some local web server you have access to (eg, stx-tools build environment includes one). And remove one of the files mentioned in .dsc, so that curl returns the 404 error for it. In this case "dget" would falsely succeed.

To fix: in the downloader script, parse the .dsc file after downloading and make sure each file referenced therein is present locally.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/923313

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/923313
Committed: https://opendev.org/starlingx/root/commit/b60f6a8119034bcdf3d2e929508b680143072820
Submitter: "Zuul (22348)"
Branch: master

commit b60f6a8119034bcdf3d2e929508b680143072820
Author: Romulo Leite <email address hidden>
Date: Tue Jul 2 12:28:01 2024 -0300

    Add check for downloader source downloads

    It was seen in [1] that Downloader was silently ignoring
    errors while downloading source packages. Specifically in
    packages that use `archive` in the meta_data.

    This commit will add a handler for this failure by parsing
    the package .dsc and checking if all necessary files were
    correctly downloaded

    Test plan:
    PASS - downloader -s -b
    PASS - serve a package with 1 .tar missing and notice failure

    Closes-bug: 2039298

    [1] https://bugs.launchpad.net/starlingx/+bug/2039298

    Change-Id: I54bf8e41a5342f96a2ac86cf9587096ef0112e0c
    Signed-off-by: Romulo Leite <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/923165
Committed: https://opendev.org/starlingx/root/commit/a04ab7812540b4b675bc8667f2c6c4b19db752e8
Submitter: "Zuul (22348)"
Branch: master

commit a04ab7812540b4b675bc8667f2c6c4b19db752e8
Author: Romulo Leite <email address hidden>
Date: Mon Jul 1 10:25:44 2024 -0300

    Add package cache update to downloader

    On an existing workspace it is common to get
    downloader errors if the packages cache has
    not being updated for a while. To avoid this
    the `downloader` will now run the `apt-get update`
    command automatically.

    Test plan:

    PASS: Run downloader in a "old" environment
    PASS: Run a fresh new environment downloader
    PASS: Build-pkgs -a

    Closes-Bug: 2039298

    Change-Id: Icdaa942da231434d66fcd1a5ff4f89e3d164abfb
    Signed-off-by: Romulo Leite <email address hidden>