Cinder

Creating multiple simultaneous backups results in many backup failures

Bug #1719580 reported by Gorka Eguileor
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
Gorka Eguileor

Bug Description

When running 10 simultaneous backups, with any backup driver inheriting from chuncked driver, simultaneously most of them will end up in "error" status.

Gorka Eguileor (gorka)
Changed in cinder:
assignee: nobody → Gorka Eguileor (gorka)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/507510

Changed in cinder:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/507510
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=af0f00bc52f79d9395adfe0575b0dbe353e18bbe
Submitter: Zuul
Branch: master

commit af0f00bc52f79d9395adfe0575b0dbe353e18bbe
Author: Gorka Eguileor <email address hidden>
Date: Wed Sep 13 19:46:17 2017 +0200

    Run backup compression on native thread

    Backup data compression is a CPU bound operation that will not yield to
    other greenthreads, so given enough simultaneous backup operations they
    will lead to other threads' starvation.

    This is really problematic for DB connections, since starvation will
    lead to connections getting dropped with errors such as "Lost connection
    to MySQL server during query".

    Detailed information on why these connections get dropped can be found
    in comment "[31 Aug 2007 9:21] Magnus Blåudd" on this MySQL bug [1].

    These DB issues may result in backups unnecessary ending in an "error"
    state.

    This patch fixes this by moving the compression to a native thread so
    the cooperative multitasking in Cinder Backup can continue switching
    threads.

    [1] https://bugs.mysql.com/bug.php?id=28359

    Closes-Bug: #1692775
    Closes-Bug: #1719580
    Change-Id: I1946dc0ad9cb7a68072a39816fa9fa224c2eb6a5

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/513016

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/pike)

Reviewed: https://review.openstack.org/513016
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=439f90da8e4c1cfa1acfe36b5ea8a6c5bae6139f
Submitter: Zuul
Branch: stable/pike

commit 439f90da8e4c1cfa1acfe36b5ea8a6c5bae6139f
Author: Gorka Eguileor <email address hidden>
Date: Wed Sep 13 19:46:17 2017 +0200

    Run backup compression on native thread

    Backup data compression is a CPU bound operation that will not yield to
    other greenthreads, so given enough simultaneous backup operations they
    will lead to other threads' starvation.

    This is really problematic for DB connections, since starvation will
    lead to connections getting dropped with errors such as "Lost connection
    to MySQL server during query".

    Detailed information on why these connections get dropped can be found
    in comment "[31 Aug 2007 9:21] Magnus Blåudd" on this MySQL bug [1].

    These DB issues may result in backups unnecessary ending in an "error"
    state.

    This patch fixes this by moving the compression to a native thread so
    the cooperative multitasking in Cinder Backup can continue switching
    threads.

    [1] https://bugs.mysql.com/bug.php?id=28359

    Closes-Bug: #1692775
    Closes-Bug: #1719580
    Change-Id: I1946dc0ad9cb7a68072a39816fa9fa224c2eb6a5
    (cherry picked from commit af0f00bc52f79d9395adfe0575b0dbe353e18bbe)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 12.0.0.0b1

This issue was fixed in the openstack/cinder 12.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 11.0.1

This issue was fixed in the openstack/cinder 11.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (driverfixes/ocata)

Fix proposed to branch: driverfixes/ocata
Review: https://review.openstack.org/518306

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (driverfixes/mitaka)

Fix proposed to branch: driverfixes/mitaka
Review: https://review.openstack.org/518309

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (driverfixes/newton)

Fix proposed to branch: driverfixes/newton
Review: https://review.openstack.org/518311

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (driverfixes/newton)

Reviewed: https://review.openstack.org/518311
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=41754fd57f27bba646d1d6e26388c7f5f4c2dc4e
Submitter: Zuul
Branch: driverfixes/newton

commit 41754fd57f27bba646d1d6e26388c7f5f4c2dc4e
Author: Gorka Eguileor <email address hidden>
Date: Wed Sep 13 19:46:17 2017 +0200

    Run backup compression on native thread

    Backup data compression is a CPU bound operation that will not yield to
    other greenthreads, so given enough simultaneous backup operations they
    will lead to other threads' starvation.

    This is really problematic for DB connections, since starvation will
    lead to connections getting dropped with errors such as "Lost connection
    to MySQL server during query".

    Detailed information on why these connections get dropped can be found
    in comment "[31 Aug 2007 9:21] Magnus Blåudd" on this MySQL bug [1].

    These DB issues may result in backups unnecessary ending in an "error"
    state.

    This patch fixes this by moving the compression to a native thread so
    the cooperative multitasking in Cinder Backup can continue switching
    threads.

    [1] https://bugs.mysql.com/bug.php?id=28359

    Closes-Bug: #1692775
    Closes-Bug: #1719580
    Change-Id: I1946dc0ad9cb7a68072a39816fa9fa224c2eb6a5
    (cherry picked from commit af0f00bc52f79d9395adfe0575b0dbe353e18bbe)
    (cherry picked from commit 439f90da8e4c1cfa1acfe36b5ea8a6c5bae6139f)
    (cherry picked from commit b241f93267646a6501e3a5fb14521f05e38a2ae9)

tags: added: in-driverfixes-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (driverfixes/ocata)

Reviewed: https://review.openstack.org/518306
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b241f93267646a6501e3a5fb14521f05e38a2ae9
Submitter: Zuul
Branch: driverfixes/ocata

commit b241f93267646a6501e3a5fb14521f05e38a2ae9
Author: Gorka Eguileor <email address hidden>
Date: Wed Sep 13 19:46:17 2017 +0200

    Run backup compression on native thread

    Backup data compression is a CPU bound operation that will not yield to
    other greenthreads, so given enough simultaneous backup operations they
    will lead to other threads' starvation.

    This is really problematic for DB connections, since starvation will
    lead to connections getting dropped with errors such as "Lost connection
    to MySQL server during query".

    Detailed information on why these connections get dropped can be found
    in comment "[31 Aug 2007 9:21] Magnus Blåudd" on this MySQL bug [1].

    These DB issues may result in backups unnecessary ending in an "error"
    state.

    This patch fixes this by moving the compression to a native thread so
    the cooperative multitasking in Cinder Backup can continue switching
    threads.

    [1] https://bugs.mysql.com/bug.php?id=28359

    Closes-Bug: #1692775
    Closes-Bug: #1719580
    Change-Id: I1946dc0ad9cb7a68072a39816fa9fa224c2eb6a5
    (cherry picked from commit af0f00bc52f79d9395adfe0575b0dbe353e18bbe)
    (cherry picked from commit 439f90da8e4c1cfa1acfe36b5ea8a6c5bae6139f)

tags: added: in-driverfixes-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/560605

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (driverfixes/mitaka)

Reviewed: https://review.openstack.org/518309
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0137bc6b0c0e87504253918d5a4572c6ac14e31c
Submitter: Zuul
Branch: driverfixes/mitaka

commit 0137bc6b0c0e87504253918d5a4572c6ac14e31c
Author: Gorka Eguileor <email address hidden>
Date: Wed Sep 13 19:46:17 2017 +0200

    Run backup compression on native thread

    Backup data compression is a CPU bound operation that will not yield to
    other greenthreads, so given enough simultaneous backup operations they
    will lead to other threads' starvation.

    This is really problematic for DB connections, since starvation will
    lead to connections getting dropped with errors such as "Lost connection
    to MySQL server during query".

    Detailed information on why these connections get dropped can be found
    in comment "[31 Aug 2007 9:21] Magnus Blåudd" on this MySQL bug [1].

    These DB issues may result in backups unnecessary ending in an "error"
    state.

    This patch fixes this by moving the compression to a native thread so
    the cooperative multitasking in Cinder Backup can continue switching
    threads.

    [1] https://bugs.mysql.com/bug.php?id=28359

    Closes-Bug: #1692775
    Closes-Bug: #1719580
    Change-Id: I1946dc0ad9cb7a68072a39816fa9fa224c2eb6a5
    (cherry picked from commit af0f00bc52f79d9395adfe0575b0dbe353e18bbe)
    (cherry picked from commit 439f90da8e4c1cfa1acfe36b5ea8a6c5bae6139f)
    (cherry picked from commit b241f93267646a6501e3a5fb14521f05e38a2ae9)
    (cherry picked from commit 41754fd57f27bba646d1d6e26388c7f5f4c2dc4e)

tags: added: in-driverfixes-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/ocata)

Reviewed: https://review.openstack.org/560605
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=173d4d0a4686a409f9578538018aa808bb437ca9
Submitter: Zuul
Branch: stable/ocata

commit 173d4d0a4686a409f9578538018aa808bb437ca9
Author: Gorka Eguileor <email address hidden>
Date: Wed Sep 13 19:46:17 2017 +0200

    Run backup compression on native thread

    Backup data compression is a CPU bound operation that will not yield to
    other greenthreads, so given enough simultaneous backup operations they
    will lead to other threads' starvation.

    This is really problematic for DB connections, since starvation will
    lead to connections getting dropped with errors such as "Lost connection
    to MySQL server during query".

    Detailed information on why these connections get dropped can be found
    in comment "[31 Aug 2007 9:21] Magnus Blåudd" on this MySQL bug [1].

    These DB issues may result in backups unnecessary ending in an "error"
    state.

    This patch fixes this by moving the compression to a native thread so
    the cooperative multitasking in Cinder Backup can continue switching
    threads.

    [1] https://bugs.mysql.com/bug.php?id=28359

    Closes-Bug: #1692775
    Closes-Bug: #1719580
    Change-Id: I1946dc0ad9cb7a68072a39816fa9fa224c2eb6a5
    (cherry picked from commit af0f00bc52f79d9395adfe0575b0dbe353e18bbe)
    (cherry picked from commit 439f90da8e4c1cfa1acfe36b5ea8a6c5bae6139f)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.7

This issue was fixed in the openstack/cinder 10.0.7 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.