Debian: build-pkgs fails when attempting to download from URLs that have '&' characters

[M. Vefa Bicakci]

Brief Description
-----------------
When a developer adds a URL with '&' characters to dl_files section of meta_data.yaml, the Debian-based StarlingX build system fails with the following errors:

  debrepack - ERROR: 2022-06-29 20:04:31 (3.79 MB/s) - '...' saved [...]
  debrepack - ERROR: [ Failed - "wget -t 5 --wait=15 http://a_url_with_& -O ..." ]

This occurs because the build system passes all arguments to the shell without any quoting, and the '&' character is interpreted as "background-this-command-please" by the shell. The solution is to avoid using Python's subprocess.Popen constructor's "shell" parameter.

Severity
--------
Minor: Only problematic when/if a URL with shell-specific special characters needs to be downloaded by the build.

I already have a patch for this, and I am opening this bug report as a placeholder to refer to. This is why this template is incomplete.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/852033
Committed: https://opendev.org/starlingx/root/commit/c0c89d5f5164078cc69b4e6cd4c3a4a04ec22614
Submitter: "Zuul (22348)"
Branch: master

commit c0c89d5f5164078cc69b4e6cd4c3a4a04ec22614
Author: M. Vefa Bicakci <email address hidden>
Date: Wed Jun 29 20:08:37 2022 +0000

    stx: Do not use shell for curl and wget

    When using a download URL that includes the character '&' (as part of a
    query string), the following build failure is encountered, because the
    '&' character was is interpreted by the shell as a special character for
    backgrounding the 'wget' command.

      debrepack - ERROR: 2022-06-29 20:04:31 (3.79 MB/s) - \
        '...' saved [...]
      debrepack - ERROR: [ Failed - "wget -t 5 --wait=15 \
        http://a_url_with_& -O ..." ]

    This issue is resolved by not using the shell for spawning download
    commands. (That is, by setting "shell=False" in Python's
    subprocess.Popen constructor.) This commit implements an in-place
    solution that detects the type of the argument and sets the "shell="
    keyword argument to subprocess.Popen accordingly.

    Verification:
    - Downloading from URLs with '&' characters works as expected as part of
      the "debrepack" phase of build-pkgs.
    - No negative behaviour observed with a build from scratch.

    Closes-Bug: 1988343
    Change-Id: I2a529f60b9a57b6d139f95d31bcd18c51a0fbecb
    Signed-off-by: M. Vefa Bicakci <email address hidden>