Debian: Build system floods the console when processing packages that have many files

[M. Vefa Bicakci]

Brief Description
-----------------

The Debian build system's helper function "run_shell_cmd" in "build-tools/stx/utils.py" uses stdout=subprocess.PIPE and stderr=subprocess.PIPE to capture the standard output and standard error of the command being executed, and afterwards dumps all of it to the console and the log file. This makes it problematic to log the output of "tar" when generating orig.tar.gz archives for large software packages with many files (such as the kernel) for two reasons:

- build-pkgs essentially caches the output of tar, increasing memory utilization when tar generates large amount of output. The memory utilization was admittedly not measured, but the builder.log file's contents offer a hint: Excluding the timestamp prefix strings, tar generates about 3,953,054 bytes (i.e., ~3.77 MiB) of output for the standard kernel package. Afterwards, the run_shell_cmd function splits this large string into lines and prints out each line in the string one by one.

- build-pkgs floods the console with arguably unnecessary information. For each kernel package, the output takes about 13 seconds to emit to the console and consists of about 77640 lines:

  2022-09-15 16:20:31,396 - debrepack - INFO: [ Run - "cd .../std/linux; tar czvf linux_5.10.112.orig.tar.gz linux-5.10.112" ]
  2022-09-15 16:21:09,950 - debrepack - DEBUG: linux-5.10.112/
  ...
  2022-09-15 16:21:22,870 - debrepack - DEBUG: linux-5.10.112/CREDITS
  2022-09-15 16:21:22,878 - debrepack - INFO: Apply deb patch: 0001-kernel-std-Remove-the-old-changelog-file.patch

This issue can be resolved by quietening "tar" by removing its "v" (verbose) option from the command line used for generating orig.tar.gz archives in the following file and function:

File "build-tools/stx/debrepack.py":

def create_orig_tarball(self):
  # ...
  run_shell_cmd('cd %s; tar czvf %s %s' % (self.pkginfo["packdir"], origtargz, srcname), self.logger)

Severity
--------
Minor: This is a very minor issue; it only affects the amount of output generated by the build system.

Note
----
This bug report exists just so that I can publish/propose a commit fixing this issue. I did not fill out the full bug report template on purpose.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/root/+/858433

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/root/+/858433
Committed: https://opendev.org/starlingx/root/commit/230afa561bb17857f2aea53f164a57efdc3b02e8
Submitter: "Zuul (22348)"
Branch: master

commit 230afa561bb17857f2aea53f164a57efdc3b02e8
Author: M. Vefa Bicakci <email address hidden>
Date: Thu Sep 15 16:48:47 2022 +0000

    Debian/stx: build-pkgs: Quieten *.orig.tar.gz gen.

    run_shell_cmd uses stdout=subprocess.PIPE and stderr=subprocess.PIPE to
    capture the standard output and standard error of the command being
    executed, and afterwards dumps all of it to the console and the log
    file.

    This is problematic for large software packages with many files (such as
    the kernel) for two reasons:

    - build-pkgs essentially caches the output of tar, increasing memory
      utilization when tar generates large amount of output. The memory
      utilization was admittedly not measured, but the builder.log file's
      contents offer a hint: Excluding the timestamp prefix strings, tar
      generates about 3,953,054 bytes (i.e., ~3.77 MiB) of output for the
      standard kernel package. Afterwards, the run_shell_cmd function splits
      this large string into lines and prints out each line in the string
      one by one.

    - build-pkgs floods the console with arguably unnecessary information.
      For each kernel package, the output takes about 13 seconds to emit to
      the console and consists of about 77640 lines:

      2022-09-15 16:20:31,396 - debrepack - INFO: [ Run - \
        "cd .../std/linux; tar czvf linux_5.10.112.orig.tar.gz \
         linux-5.10.112" ]
      2022-09-15 16:21:09,950 - debrepack - DEBUG: linux-5.10.112/
      ...
      2022-09-15 16:21:22,870 - debrepack - DEBUG: linux-5.10.112/CREDITS
      2022-09-15 16:21:22,878 - debrepack - INFO: Apply deb patch: \
        0001-kernel-std-Remove-the-old-changelog-file.patch

    Hence, this commit quietens the generation of orig.tar.gz archives.

    Verification
    - 'build-pkgs -b rt,std -c -a' is successful.

    Closes-Bug: 1990175
    Change-Id: I9ef2adbb85930fbba542f1e47dc36882d39a5996
    Signed-off-by: M. Vefa Bicakci <email address hidden>