Parallel executions of gen-bootloader-iso.sh can cause install failures

[Don Penney]

Brief Description
-----------------
Parallel execution of gen-bootloader-iso.sh can lead to a partial boot setup, if the "shared" files have not finished setup when the subsequent executions run. This can lead to installation failures as some required shared data may not be accessible.

Severity
--------
Minor

Steps to Reproduce
------------------
Run utility in two separate windows, one after the other. The first execution starts setting up the shared files. The second will attempt to use the shared content.

Expected Behavior
------------------
It is expected that all install setups would have access to all shared content.

Actual Behavior
----------------
Some setups may only have partial shared content.

Reproducibility
---------------
Intermittent. A number of steps in DC subcloud installs run prior to the execution of gen-bootloader.sh, and all would need to complete within the same timeframe in order to trigger parallel executions of the utility.

System Configuration
--------------------
distributed cloud

Branch/Pull Time/Commit
-----------------------
Seen in master, June 30, 2020

Workaround
----------
Use serial install for subclouds

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738699

[Ghada Khalil (gkhalil)]

stx.5.0 / medium priority - issue with parallel subcloud installs; serial installs work

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738699
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=7ed511a6a0b861573bd9224a8a2a2ff6a031f463
Submitter: Zuul
Branch: master

commit 7ed511a6a0b861573bd9224a8a2a2ff6a031f463
Author: Don Penney <email address hidden>
Date: Tue Jun 30 17:19:14 2020 -0400

    Protect against parallel exec of gen-bootloader-iso.sh

    The bulk of the work done by gen-bootloader-iso.sh is not safe to be
    executed by two callers in parallel. If the initial execution is
    setting up shared files when a subsequent call starts, the subsequent
    call can end up with an incomplete set of symlinks to shared files.
    This, in turn, can result in installation failures.

    To protect against this, a file lock is added to gen-bootloader-iso.sh
    to prevent subsequent calls from accessing shared files before they're
    completely setup.

    Change-Id: Id6def4527b226b8746b2b8ff74059c0d3937e130
    Closes-Bug: 1885779
    Signed-off-by: Don Penney <email address hidden>

[Ghada Khalil (gkhalil)]

Changing the release tag to stx.4.0 since the fix made it for that release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/utilities/+/792213