Comment 42 for bug 1844446

Reviewed: https://review.opendev.org/688201
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=7994173916ec2b4fd5885944120ce3896b2bd413
Submitter: Zuul
Branch: stable/stein

commit 7994173916ec2b4fd5885944120ce3896b2bd413
Author: Alex Schultz <email address hidden>
Date: Wed Oct 2 09:05:58 2019 -0600

    Make executor type dynamic

    When we run the tripleo-container-image-prepare script, it performs
    better under python2 when the process leverages a ProcessPoolExecutor.
    Rather than using threading, we should be using processes to handle the
    image upload processing. Currently when we're processing the images, we
    end up being single threaded due to the GIL when processing the data. By
    switching to the ProcessPoolExecutor, we eliminate the locking that is
    occuring during the data processing as it'll be handled in each process.

    Unfortunately, we cannot leverage the ProcessPoolExecutor when the same
    code is run under Mistral. In order to make the code work for both
    methods, we need to make the execution type dynamic. This change creates
    two types of lock objects that are used to determine what type of
    executor to ultimately use when processing the images for uploading.

    Additionally this change limits the number of concurrent image upload
    processes to 4 if using the ProcessPoolExecutor and caps the number of
    threads at a max of 8 based on (cpu count / 2)

     Conflicts:
     tripleo_common/image/image_uploader.py

    Change-Id: I60507eba9884a0660fe269da5ad27b0e57a70ca8
    Related-Bug: #1844446
    (cherry picked from commit 60afc0eec44f698dd95d9d6ec80dad94a4b07329)