Insufficient space on /scratch for request /v1/loads/import_load

[Guilherme Schons]

Brief Description
-----------------
Failed to import an inactive load after importing an active load due to insufficient space in /scratch.

Severity
--------
Minor

Steps to Reproduce
------------------
Import an inactive load (--inactive) after importing an active load (--active).

Expected Behavior
------------------
The load was imported successfully

Actual Behavior
----------------
Failed due to insufficient space.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Distributed Cloud

Branch/Pull Time/Commit
-----------------------
N/A

Last Pass
---------
N/A

Timestamp/Logs
--------------
Fist time:
[sysadmin@controller-0 ~(keystone_admin)]$ system --os-region-name SystemController load-import --inactive bootiso.iso bootiso.sig
This operation will take a while. Please wait.
.......
Load import failed. Reason: Unexpected error copying load to vault

Could not find corresponding error logs.

Second time:

Load import failed. Reason: Insufficient space on /scratch for request /v1/loads/import_load

Test Activity
-------------
N/A

Workaround
----------
Clean up the /scratch/tmp_load directory.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/892037

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/899865

[OpenStack Infra (hudson-openstack)]

Change abandoned by "John Kung <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/config/+/892037
Reason: Abandon in favour of https://review.opendev.org/c/starlingx/config/+/899865 (as that can be moved out of WIP)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/897563
Committed: https://opendev.org/starlingx/distcloud/commit/37d93277d51e56b6dc079851383c92a446deb664
Submitter: "Zuul (22348)"
Branch: master

commit 37d93277d51e56b6dc079851383c92a446deb664
Author: Lindley Werner <email address hidden>
Date: Fri Oct 6 11:39:18 2023 -0300

    Load-import failed due to lack of space in /scratch

    During a load-import operation, the iso file is copied to /scratch,
    a temporary folder (https://docs.starlingx.io/planning/kubernetes/storage-planning-storage-on-worker-hosts.html#root-filesystem-storage).
    Before the operation, it is checked if this folder has a free space of
    at least 3 times the ISO file, if not the load-import is aborted.

    This change enhances the message displayed to the user in order to
    give him some context and two solutions:
      - Delete some unused files from /scratch
      - Increase /scratch size

    Test Plan:
      PASS: After adding more free space to /scratch, the load-import
    operation must pass without error

    Related to: https://review.opendev.org/c/starlingx/config/+/899865
    Partial-bug: 2031941

    Change-Id: Iff99e8793ab6fa5c812648bba4d93e8eab22a6b5
    Signed-off-by: Lindley Werner <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/899865
Committed: https://opendev.org/starlingx/config/commit/006810132c99808d430d1e7fa6416f043bd88e8c
Submitter: "Zuul (22348)"
Branch: master

commit 006810132c99808d430d1e7fa6416f043bd88e8c
Author: Guilherme Schons <email address hidden>
Date: Fri Aug 18 16:13:27 2023 -0300

    Load-import clean up temp folder

    1- Fix /scratch clean up after loading active import
    2- Fix /scratch clean up if some exception occurs during load-import
    3- Properly display error to the user if something happens during HTTP
    upload request
    4- The /scratch folder can be full with non-related load-import files,
    in this case, a message is displayed to the user in order to give him
    some context and two solutions:
      - Delete some unused files from /scratch
      - Increase /scratch size

    Test Plan:
        PASS: /scratch cleanup after importing an active load
        PASS: /scratch cleanup after importing an active load (DC proxy)
        PASS: /scratch cleanup after exception importing load
        PASS: /scratch cleanup after exception importing load (DC proxy)
        PASS: After adding more free space to /scratch, the load-import
    the operation must pass without error

    Related to: https://review.opendev.org/c/starlingx/distcloud/+/897563
    Closes-bug: 2031941

    Change-Id: I580a59e3c8f91606a2a623cead2d3667fc264435
    Signed-off-by: Lindley Werner <email address hidden>
    Signed-off-by: Guilherme Schons <email address hidden>