Debian: Build system does not log the stderr of executed commands

[M. Vefa Bicakci]

Brief Description
-----------------

As of this writing, the Debian-based StarlingX build system does not log the standard error stream of the commands it executes via the run_shell_cmd function, unless a command fails. This causes non-critical warning messages to be missed during the build.

In addition, in case a command spawned by run_shell_cmd does fail, its return code is not logged, which can be problematic if the command does not emit output sufficient to debug why it failed.

Severity
--------
Minor: This is a minor build system issue: It causes non-critical warnings to be missed, and it affects debuggability in case a command fails.

Note
----
This bug report exists just so that I can publish/propose a commit fixing this issue. I did not fill out the full bug report template on purpose.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/858434

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/858434
Committed: https://opendev.org/starlingx/root/commit/818830b25e338d7f8b6ac4408c67fd6fc7df29a7
Submitter: "Zuul (22348)"
Branch: master

commit 818830b25e338d7f8b6ac4408c67fd6fc7df29a7
Author: M. Vefa Bicakci <email address hidden>
Date: Fri Sep 16 19:05:42 2022 +0000

    Debian/stx: utils: Minor run_shell_cmd fix-ups

    This commit fixes various minor issues in utils.py Python module's
    run_shell_cmd function:

    - Fix "local variable 'process' referenced before assignment" exception
      that can occur while handling exceptions raised by subprocess.Popen().
      This appears to only occur when a forked/child process cannot execute
      a program, which is a programming error, which in turn is a minor
      issue.

      Before this commit, attempting to run a non-existent program with, for
      example: 'run_shell_command("/bin/abc", ...)' would result in the
      exception "local variable 'process' referenced before assignment"
      while handling the exception "[Errno 2] No such file or directory:
      '/bin/abc'".

      With this commit, only the latter exception is reported, but in the
      following form, as part of a single/non-nested exception:

        [ Failed to execute command: "['/bin/abc']" Exception: "[Errno 2] \
          No such file or directory: '/bin/abc'" ]

    - Avoid making Python print two tracebacks when handling exceptions
      raised by subprocess.Popen(). Logging a one-line description of the
      exception encountered by subprocess.Popen() (via "{e}", where "e" is
      an Exception object) is sufficient on its own to debug the encountered
      issue. (Please see above for an example.)

    - Always log standard error of the command. While most of the messages
      logged to the standard error are debugging-oriented or point at minor
      issues, ignoring standard error still causes us to miss non-critical
      warning messages. In addition, change the log level for standard error
      output from "error" to "debug" to avoid panicking build system users
      due to the non-critical messages that will appear, such as the
      following:

      dpkg-architecture: warning: cannot determine CC system type, falling \
        back to default (native compilation)
      dpkg-source --before-build .
      dpkg-buildpackage: warning: building a source package without \
        cleaning up as you asked; it might contain undesired files
      dpkg-source -b .
      dpkg-genbuildinfo --build=source
      dpkg-genchanges --build=source >../mlnx-ofed-kernel_....changes
      dpkg-genchanges: info: not including original source code in upload

    - In the logs, prefix the command's standard output with "stdout: " and
      its standard error with "stderr: ". This allows distinguishing between
      the command's messages emitted via standard output and standard error.

    - Note that adding the "stdout: " prefix to each line of each command's
      standard output increases the log file's size by about 190 KiB,
      according to the following results from a fresh build with
      'build-...