Juniper/contrail reset_db*.sh scripts can hang

Bug #1765506 reported by Krzysztof Klimonda on 2018-04-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Committed
High
Michał Błotniak
Trunk
Fix Committed
Undecided
Michał Błotniak
OpenContrail
Fix Committed
Undecided
Unassigned

Bug Description

tools/reset_db.sh and tools/reset_db_psql.sh have a potential for hanging on the command that redirects stdin into `docker exec` call.

When that happens, quick look through the affected VM shows that `docker exec` is waiting for the spawned process to finish, and mysql/psql calls wait on readv(0, indefinitely (fd 0 is stdin). This could be docker version specific, or could be caused by piping large .sql files.

Review in progress for https://review.opencontrail.org/42182
Submitter: Krzysztof Klimonda (<email address hidden>)

OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/42260
Submitter: Michal Blotniak (<email address hidden>)

Jeba Paulaiyan (jebap) on 2018-04-20
Changed in opencontrail:
importance: Undecided → High
assignee: nobody → Michał Błotniak (buoto)
assignee: Michał Błotniak (buoto) → nobody
importance: High → Undecided
tags: added: provisioning
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/42260
Submitter: Krzysztof Klimonda (<email address hidden>)

Reviewed: https://review.opencontrail.org/42182
Committed: http://github.com/Juniper/contrail/commit/2292ba68fbeaadf3944d44a7f69b360440832c7e
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 2292ba68fbeaadf3944d44a7f69b360440832c7e
Author: Krzysztof Klimonda <email address hidden>
Date: Thu Apr 19 15:57:10 2018 +0200

Lower job timeout to 90 minutes

We're hitting some random deadlocks on redirecting stdin to docker exec
calls. This review lowers a timeout for the job to 90 minutes, so that
reviews stuck on that fail faster.

Partial-Bug: #1765506
Change-Id: Ie4b31bf29fe1652f13ee458919ae2014afb796d1

Review in progress for https://review.opencontrail.org/42260
Submitter: Michal Blotniak (<email address hidden>)

Reviewed: https://review.opencontrail.org/42260
Committed: http://github.com/Juniper/contrail/commit/27f029ffea0906941544bb9661891b1c119ce4ef
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 27f029ffea0906941544bb9661891b1c119ce4ef
Author: Michał Błotniak <email address hidden>
Date: Fri Apr 20 10:33:14 2018 +0200

Mount sourcedir in DB containers

Passing huge init SQL files via stdin to docker exec caused random
deadlocks. To avoid this files are mounted and read by DB clients inside
containers.

Note that containers in dev environments should be rebuilt
with `make testenv`.

Change-Id: Ia7688c7048da88cb016c28682754145dc37c547b
Partial-Bug: #1765506

Changed in opencontrail:
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers