multiple tripleo jobs timing out upstream causing gate resets in train
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Alex Schultz |
Bug Description
http://
possible root cause:
patch to increase the verbosity of container download / upload to resolve ovh failures: https:/
and
close httpd sessions
https:/
tags: | added: alert |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (master) | #1 |
OpenStack Infra (hudson-openstack) wrote : | #2 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master) | #3 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master) | #4 |
Reviewed: https:/
Committed: https:/
Submitter: Clark Boylan (<email address hidden>)
Branch: master
commit 97e0c940f317f3a
Author: Emilien Macchi <email address hidden>
Date: Wed Sep 18 17:25:46 2019 -0400
fs010: disable validations on the undercloud
The validations cause timeouts, we should have them tested outside of
gate/check for now.
Change-Id: I198939058b9d09
Related-Bug: #1844446
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (master) | #5 |
Reviewed: https:/
Committed: https:/
Submitter: Clark Boylan (<email address hidden>)
Branch: master
commit 89be0d17c5d3807
Author: Emilien Macchi <email address hidden>
Date: Wed Sep 18 10:07:59 2019 -0400
Disable inflight validations by default
The inflight validations caused timeouts in the TripleO CI.
They should not be enabled by default. New features should not be
enabled by default in general, they should be optional.
So this patch changes the option to be --inflight-
and make sure the inflights are disabled by default everywhere it's
called.
Change-Id: I082628a9480686
Related-Bug: #1844446
OpenStack Infra (hudson-openstack) wrote : | #6 |
Reviewed: https:/
Committed: https:/
Submitter: Clark Boylan (<email address hidden>)
Branch: master
commit 64af3ae1fb9adda
Author: Emilien Macchi <email address hidden>
Date: Wed Sep 18 12:15:39 2019 -0400
Introduce --inflight-
Like we do for the overcloud, add the --inflight-
disabled by default. Disable by default, we'll skip the
"opendev-
Related-Bug: #1844446
Change-Id: Ia37b3d4cc657d9
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (stable/stein) | #7 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (stable/stein) | #8 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 499b487c51aa91e
Author: Cédric Jeanneret <email address hidden>
Date: Thu Aug 15 11:44:32 2019 +0200
(squash) Clean backport of inflight-
Adds new "--no-inflight-
This provides an independant way to activate or not the in-flight
validations within a deploy.
The default is to have them running, and this option allows to
deactivate the in-flight ones.
Change-Id: I81e934e2978cad
(cherry picked from commit bf48dbc84405208
Disable inflight validations by default
The inflight validations caused timeouts in the TripleO CI.
They should not be enabled by default. New features should not be
enabled by default in general, they should be optional.
So this patch changes the option to be --inflight-
and make sure the inflights are disabled by default everywhere it's
called.
Change-Id: I082628a9480686
Related-Bug: #1844446
(cherry picked from commit 89be0d17c5d3807
Introduce --inflight-
Like we do for the overcloud, add the --inflight-
disabled by default. Disable by default, we'll skip the
"opendev-
Related-Bug: #1844446
Change-Id: Ia37b3d4cc657d9
(cherry picked from commit 64af3ae1fb9adda
tags: | added: in-stable-stein |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master) | #9 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master) | #10 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #11 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #12 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #13 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master) | #14 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 5cbb6bee6a15ed8
Author: Alex Schultz <email address hidden>
Date: Tue Sep 24 09:58:55 2019 -0600
Remove chunk size for url stream
Related-Bug: #1844446
Change-Id: I0eea178cc4fe03
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/stein) | #15 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (master) | #16 |
Change abandoned by Cédric Jeanneret (Tengu) (<email address hidden>) on branch: master
Review: https:/
Reason: abandoned in favor of https:/
Bogdan Dobrelya (bogdando) wrote : | #17 |
Also addressed-by: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/stein) | #18 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit c5729bb84704dfc
Author: Alex Schultz <email address hidden>
Date: Tue Sep 24 09:58:55 2019 -0600
Remove chunk size for url stream
Related-Bug: #1844446
Change-Id: I0eea178cc4fe03
(cherry picked from commit 5cbb6bee6a15ed8
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master) | #19 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #20 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (master) | #21 |
Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master) | #22 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 3adfefa13aebd7b
Author: Alex Schultz <email address hidden>
Date: Thu Sep 26 11:05:15 2019 -0600
Randomize the container list for uploads
When we work through the list of containers in an alphabetical fashion,
we end up duplicating much of the layer fetching because it can occur at
the same time. Things like cinder-api, cinder-backup, cinder-volume
share many of the same layers. Since we don't ensure that we only do
a single fetching of a layer hash durring the multiprocessing, we end
up duplicating the fetches of layers. By randomizing the fetches, we
reduce the likelihood that we'll be fetching the same family of service
containers concurrently.
Change-Id: Ifbcd55de52c9e2
Related-Bug: #1844446
OpenStack Infra (hudson-openstack) wrote : | #23 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit a1d89c7d6359a38
Author: Alex Schultz <email address hidden>
Date: Tue Sep 24 12:04:42 2019 -0600
Improve ThreadPoolExecutor usage
This change addresses a few different issues with out ThreadPoolExecutor
usage.
Previously we were inefficiently looping on the job results which was a
blocking call and not asynchronous. This change switched the threadpool
usage to use as_completed so we'll handle jobs as they complete rather
than blocking until the the specific job we're looking at completes.
Additionally this switches to use a with statement for the executor so
the threads get cleaned up correctly when we're done with the block.
See example comments:
https:/
The _inspect call was using a thread pool executor to run session calls
but was essentially a serial executing function so it has been updated
to remove the executor and just call the requests as needed.
Change-Id: Ia7abec997f4e50
Related-Bug: #1844446
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (master) | #24 |
Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https:/
Reason: sounds like these tweakings aren't needed now, and they didn't seem to help.
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master) | #25 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit e57116d9c45c53a
Author: Alex Schultz <email address hidden>
Date: Thu Sep 26 14:30:13 2019 -0600
Implement threading locks around layers
When we fetch layers, we shouldn't fetch the same layers multiple times.
This change adds some locking basked on layer hashes to prevent multiple
threads from trying to fetch the same layer at the same time.
Change-Id: I477219b7dca1e6
Related-Bug: #1844446
OpenStack Infra (hudson-openstack) wrote : | #26 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 577334235a49abb
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Sep 25 17:22:31 2019 +0200
Always close src/dst sessions and raise to retry
When something goes awry for the image upload process,
always ensure the sessions are getting closed, something gets
logged and whenever necessary, an exception is re-raised for the future
retries with tenacity wrappers.
Change-Id: Id0602b0f51d437
Related-Bug: #1844446
Signed-off-by: Bogdan Dobrelya <email address hidden>
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master) | #27 |
Related fix proposed to branch: master
Review: https:/
Bogdan Dobrelya (bogdando) wrote : | #28 |
It still times out from time to time, see https:/
https:/
OpenStack Infra (hudson-openstack) wrote : | #29 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/stein) | #30 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #31 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #32 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #33 |
Related fix proposed to branch: stable/stein
Review: https:/
Changed in tripleo: | |
status: | Triaged → In Progress |
assignee: | nobody → Alex Schultz (alex-schultz) |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master) | #34 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master) | #35 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 60afc0eec44f698
Author: Alex Schultz <email address hidden>
Date: Wed Oct 2 09:05:58 2019 -0600
Make executor type dynamic
When we run the tripleo-
better under python2 when the process leverages a ProcessPoolExec
Rather than using threading, we should be using processes to handle the
image upload processing. Currently when we're processing the images, we
end up being single threaded due to the GIL when processing the data. By
switching to the ProcessPoolExec
occuring during the data processing as it'll be handled in each process.
Unfortunately, we cannot leverage the ProcessPoolExecutor when the same
code is run under Mistral. In order to make the code work for both
methods, we need to make the execution type dynamic. This change creates
two types of lock objects that are used to determine what type of
executor to ultimately use when processing the images for uploading.
Additionally this change limits the number of concurrent image upload
processes to 4 if using the ProcessPoolExecutor and caps the number of
threads at a max of 8 based on (cpu count / 2)
Change-Id: I60507eba9884a0
Related-Bug: #1844446
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (master) | #36 |
Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https:/
Reason: We no longer execute the subject code via Mistral, so there is no more need to tweak Multi-Threading cooperation. Multi Process had shown much better results (expected for Python with its GIL problems)
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/stein) | #37 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/stein) | #38 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 0c3c2623de8418a
Author: Alex Schultz <email address hidden>
Date: Thu Sep 26 11:05:15 2019 -0600
Randomize the container list for uploads
When we work through the list of containers in an alphabetical fashion,
we end up duplicating much of the layer fetching because it can occur at
the same time. Things like cinder-api, cinder-backup, cinder-volume
share many of the same layers. Since we don't ensure that we only do
a single fetching of a layer hash durring the multiprocessing, we end
up duplicating the fetches of layers. By randomizing the fetches, we
reduce the likelihood that we'll be fetching the same family of service
containers concurrently.
Change-Id: Ifbcd55de52c9e2
Related-Bug: #1844446
(cherry picked from commit 3adfefa13aebd7b
OpenStack Infra (hudson-openstack) wrote : | #39 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit c882901f992f325
Author: Alex Schultz <email address hidden>
Date: Tue Sep 24 12:04:42 2019 -0600
Improve ThreadPoolExecutor usage
This change addresses a few different issues with out ThreadPoolExecutor
usage.
Previously we were inefficiently looping on the job results which was a
blocking call and not asynchronous. This change switched the threadpool
usage to use as_completed so we'll handle jobs as they complete rather
than blocking until the the specific job we're looking at completes.
Additionally this switches to use a with statement for the executor so
the threads get cleaned up correctly when we're done with the block.
See example comments:
https:/
The _inspect call was using a thread pool executor to run session calls
but was essentially a serial executing function so it has been updated
to remove the executor and just call the requests as needed.
Change-Id: Ia7abec997f4e50
Related-Bug: #1844446
(cherry picked from commit a1d89c7d6359a38
OpenStack Infra (hudson-openstack) wrote : | #40 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 55df4dec4efdc93
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Sep 25 17:22:31 2019 +0200
Always close src/dst sessions and raise to retry
When something goes awry for the image upload process,
always ensure the sessions are getting closed, something gets
logged and whenever necessary, an exception is re-raised for the future
retries with tenacity wrappers.
Change-Id: Id0602b0f51d437
Related-Bug: #1844446
Signed-off-by: Bogdan Dobrelya <email address hidden>
(cherry picked from commit 577334235a49abb
OpenStack Infra (hudson-openstack) wrote : | #41 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 74abbd1f0e6eb0e
Author: Alex Schultz <email address hidden>
Date: Thu Sep 26 14:30:13 2019 -0600
Implement threading locks around layers
When we fetch layers, we shouldn't fetch the same layers multiple times.
This change adds some locking basked on layer hashes to prevent multiple
threads from trying to fetch the same layer at the same time.
Change-Id: I477219b7dca1e6
Related-Bug: #1844446
(cherry picked from commit e57116d9c45c53a
OpenStack Infra (hudson-openstack) wrote : | #42 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 7994173916ec2b4
Author: Alex Schultz <email address hidden>
Date: Wed Oct 2 09:05:58 2019 -0600
Make executor type dynamic
When we run the tripleo-
better under python2 when the process leverages a ProcessPoolExec
Rather than using threading, we should be using processes to handle the
image upload processing. Currently when we're processing the images, we
end up being single threaded due to the GIL when processing the data. By
switching to the ProcessPoolExec
occuring during the data processing as it'll be handled in each process.
Unfortunately, we cannot leverage the ProcessPoolExecutor when the same
code is run under Mistral. In order to make the code work for both
methods, we need to make the execution type dynamic. This change creates
two types of lock objects that are used to determine what type of
executor to ultimately use when processing the images for uploading.
Additionally this change limits the number of concurrent image upload
processes to 4 if using the ProcessPoolExecutor and caps the number of
threads at a max of 8 based on (cpu count / 2)
Conflicts:
tripleo_
Change-Id: I60507eba9884a0
Related-Bug: #1844446
(cherry picked from commit 60afc0eec44f698
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master) | #43 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 746d0700c7487f2
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Sep 23 14:01:32 2019 +0200
Use cache for yum update in containers prepare
For the releases later than Stein, whereas a better podman versions
used, use the cached package manager contents.
In order to enable the container-
specify a cache directory. Let's use
/var/
for our container modification process in CI.
Related-Bug: #1844446
Change-Id: I5ce878205ddb18
Signed-off-by: Bogdan Dobrelya <email address hidden>
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master) | #44 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #45 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (master) | #46 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (master) | #47 |
Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : | #48 |
Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https:/
Reason: this doesn't work
Changed in tripleo: | |
milestone: | train-rc1 → ussuri-1 |
OpenStack Infra (hudson-openstack) wrote : | #49 |
Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https:/
Reason: this doesn't work, I ran out of ideas
wes hayutin (weshayutin) wrote : | #50 |
Closing this out.. let's open new bugs on more specific issues
Changed in tripleo: | |
status: | In Progress → Fix Released |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master) | #51 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 46f81298948865a
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 15 11:33:16 2019 +0200
Make upload workers faster on processing layers
Make upload workers processing image layers only once (as the best
effort). This also reworks and simplifies locks management for
individual tasks now managed for the PythonImageUploader class
namespace only.
When fetching source layer, cross-link it for the target
local image, whenever that source is already exists. When pushing a
layer to a target registry, do not repeat transfering the same data,
if already pushed earlier for another image.
The 1st time a layer gets uploaded/fetched for an image, that image and
its known path (local or remote) becomes a reference for future
cross-
Store such information about already processed layers in global view
shared for all workers to speed-up data transfering jobs they execute.
Having that global view, uploading the 1st image in the tasks list as a
separate (and non-concurrent) job becomes redundant and now will be
executed concurently with other images.
Based on the dynamically picked multi-workers mode, provide the global
view as a graf with its MP/MT state synchronization as the following:
* use globally shared locking info also containing global layers view
for MP-workers. With the shared global view state we can no longer
use local locking objects individual for each task.
* if cannot use multi-process workers, like when executing it via
Mistral by monkey patched eventlet greenthreads, choose threadinglock
and multi-threads-safe standard dictionary in the shared class
namespace to store the global view there
* if it can do MP, pick processlock also containing a safe from data
races Manager().dict() as the global view shared among cooperating OS
processes.
* use that global view in a transparent fashion, provided by a special
classmethod proxying access to the internal state shared for workers.
Ultimately, all that optimizes:
* completion time
* re-fetching of the already processed layers
* local deduplication of layers
* the amount of outbound HTTP requests to registries
* if-layer-exists and other internal logic check executed against the
in-memory cache firstly.
As layers locking and unlocking becomes a popular action, reduce the
noise of the debug messages it produces.
Closes-bug: #1847225
Related-bug: #1844446
Change-Id: Ie5ef4045b7e22c
Signed-off-by: Bogdan Dobrelya <email address hidden>
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/train) | #52 |
Related fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/stein) | #53 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/train) | #54 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/train
commit a5768f0d7563832
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 15 11:33:16 2019 +0200
Make upload workers faster on processing layers
Make upload workers processing image layers only once (as the best
effort). This also reworks and simplifies locks management for
individual tasks now managed for the PythonImageUploader class
namespace only.
When fetching source layer, cross-link it for the target
local image, whenever that source is already exists. When pushing a
layer to a target registry, do not repeat transfering the same data,
if already pushed earlier for another image.
The 1st time a layer gets uploaded/fetched for an image, that image and
its known path (local or remote) becomes a reference for future
cross-
Store such information about already processed layers in global view
shared for all workers to speed-up data transfering jobs they execute.
Having that global view, uploading the 1st image in the tasks list as a
separate (and non-concurrent) job becomes redundant and now will be
executed concurently with other images.
Based on the dynamically picked multi-workers mode, provide the global
view as a graf with its MP/MT state synchronization as the following:
* use globally shared locking info also containing global layers view
for MP-workers. With the shared global view state we can no longer
use local locking objects individual for each task.
* if cannot use multi-process workers, like when executing it via
Mistral by monkey patched eventlet greenthreads, choose threadinglock
and multi-threads-safe standard dictionary in the shared class
namespace to store the global view there
* if it can do MP, pick processlock also containing a safe from data
races Manager().dict() as the global view shared among cooperating OS
processes.
* use that global view in a transparent fashion, provided by a special
classmethod proxying access to the internal state shared for workers.
Ultimately, all that optimizes:
* completion time
* re-fetching of the already processed layers
* local deduplication of layers
* the amount of outbound HTTP requests to registries
* if-layer-exists and other internal logic check executed against the
in-memory cache firstly.
As layers locking and unlocking becomes a popular action, reduce the
noise of the debug messages it produces.
Closes-bug: #1847225
Related-bug: #1844446
Change-Id: Ie5ef4045b7e22c
Signed-off-by: Bogdan Dobrelya <email address hidden>
(cherry picked from commit 46f81298948865a
tags: | added: in-stable-train |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/stein) | #55 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 7b46f4d70b3e36b
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 15 11:33:16 2019 +0200
Make upload workers faster on processing layers
Make upload workers processing image layers only once (as the best
effort). This also reworks and simplifies locks management for
individual tasks now managed for the PythonImageUploader class
namespace only.
When fetching source layer, cross-link it for the target
local image, whenever that source is already exists. When pushing a
layer to a target registry, do not repeat transfering the same data,
if already pushed earlier for another image.
The 1st time a layer gets uploaded/fetched for an image, that image and
its known path (local or remote) becomes a reference for future
cross-
Store such information about already processed layers in global view
shared for all workers to speed-up data transfering jobs they execute.
Having that global view, uploading the 1st image in the tasks list as a
separate (and non-concurrent) job becomes redundant and now will be
executed concurently with other images.
Based on the dynamically picked multi-workers mode, provide the global
view as a graf with its MP/MT state synchronization as the following:
* use globally shared locking info also containing global layers view
for MP-workers. With the shared global view state we can no longer
use local locking objects individual for each task.
* if cannot use multi-process workers, like when executing it via
Mistral by monkey patched eventlet greenthreads, choose threadinglock
and multi-threads-safe standard dictionary in the shared class
namespace to store the global view there
* if it can do MP, pick processlock also containing a safe from data
races Manager().dict() as the global view shared among cooperating OS
processes.
* use that global view in a transparent fashion, provided by a special
classmethod proxying access to the internal state shared for workers.
Ultimately, all that optimizes:
* completion time
* re-fetching of the already processed layers
* local deduplication of layers
* the amount of outbound HTTP requests to registries
* if-layer-exists and other internal logic check executed against the
in-memory cache firstly.
As layers locking and unlocking becomes a popular action, reduce the
noise of the debug messages it produces.
Closes-bug: #1847225
Related-bug: #1844446
Change-Id: Ie5ef4045b7e22c
Signed-off-by: Bogdan Dobrelya <email address hidden>
(cherry picked from commit 46f81298948865a
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (master) | #56 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 3ca472c24fde83b
Author: Alex Schultz <email address hidden>
Date: Tue Oct 22 13:41:05 2019 -0600
Switch to use process executor
We switched the tripleo-
tripleo-common, but a user will likely run the 'openstack tripleo
container image prepare' command. Currently it uses the default which is
the threading executor.
Poke tripleo common lower/requirements as well.
Change-Id: Ifc5b46633a1f9f
Related-Bug: #1844446
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (stable/train) | #57 |
Related fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (stable/train) | #58 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/train
commit 51d589b766f9854
Author: Alex Schultz <email address hidden>
Date: Tue Oct 22 13:41:05 2019 -0600
Switch to use process executor
We switched the tripleo-
tripleo-common, but a user will likely run the 'openstack tripleo
container image prepare' command. Currently it uses the default which is
the threading executor.
Poke tripleo common lower/requirements as well.
Change-Id: Ifc5b46633a1f9f
Related-Bug: #1844446
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (stable/stein) | #59 |
Related fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to python-tripleoclient (stable/stein) | #60 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit 9206ff9a853ae62
Author: Alex Schultz <email address hidden>
Date: Tue Oct 22 13:41:05 2019 -0600
Switch to use process executor
We switched the tripleo-
tripleo-common, but a user will likely run the 'openstack tripleo
container image prepare' command. Currently it uses the default which is
the threading executor.
Poke tripleo common lower/requirements as well.
Change-Id: Ifc5b46633a1f9f
Related-Bug: #1844446
Related fix proposed to branch: master /review. opendev. org/682905
Review: https:/