[SRU] borgbackup version 1.1.15 and 1.0.13

[Steve Dodd]

[ Impact ]
* Data is corrupted on some specific conditions, upstream asked to backport 1.1.15 that fixes some high ram usage and this corruption bug (not for focal, already fixed)

[ Regression Potential ]
* Really low, the package has an autopkgtestsuite that runs thousand of tests, and the fix is since one year part of upstream code

[ Test Case ]
* Backup some TB of data, and try to read the index, it should be not corrupted

[ Other info ]
Per the "important notes" section of the borg docs:

https://borgbackup.readthedocs.io/en/stable/changes.html

"Pre-1.1.11 potential index corruption / data loss issue
A bug was discovered in our hashtable code, see issue #4829. The code is used for the client-side chunks cache and the server-side repo index.

Although borg uses the hashtables very heavily, the index corruption did not happen too frequently, because it needed specific conditions to happen.

Data loss required even more specific conditions, so it should be rare (and also detectable via borg check). [..]"

Theoretically affects all Ubuntu releases before 20.04 (focal.)

I know this is a universe package, but if anyone was up for an SRU that would be fantastic..

[Steve Dodd (anarchetic)]

Upstream bug: https://github.com/borgbackup/borg/issues/4829

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Gianfranco Costamagna (costamagnagianfranco)]

Unfortunately upgrading from 1.1.11 to 1.1.14 requires xxhash being installed on the system, because I removed the embedded version being used in borgbackup

[Robie Basak (racb)]

SRU review

There's something wrong going on here, but I'm not sure exactly what you intend so I'm not sure what to suggest. The upload in the Bionic queue is 1.1.14-2ubuntu1.20.04.1 but Bionic is 18.04. Focal is on 1.1.11-5 which is lower than the Bionic upload, so that can't work.

Please could you clarify exactly what versions you're trying to put where, and which are intended to be cherry-picks and which are intended to be new upstream versions? The Bionic queue upload seems to suggest that it's both a new upstream version and a cherry-pick, which is why I'm confused.

[Gianfranco Costamagna (costamagnagianfranco)]

xenial 16.04: upgrade from the current 1.0.12 to 1.0.13-0ubuntu1.16.04.1 (a.k.a. stock 1.0.13 + this fix cherry-picked from 1.0 maint branch)
bionic 18.04: upgrade to 1.1.14-2ubuntu18.04.1 (sorry, just the versioning in changelog was wrong!)
focal 20.04: upgrade to 1.1.14-2ubuntu20.04.1

I fixed the changelog, changed from 20 to 18 and reuploaded to bionic
(and uploaded the focal version too)

[Brian Murray (brian-murray)]

It looks like there may be a test case in the upstream bug but it'd be helpful if one was added to the Launchpad one.

[Gianfranco Costamagna (costamagnagianfranco)]

Hello Brian,
I'll ask upstream to test it once the new version is in proposed pocket. Is it ok for you?

[Brian Murray (brian-murray)]

What I'm saying is there is not a documented "Test Case" in the bug report which is an important part of the SRU process. Please add one so that we can review this SRU.

[Gianfranco Costamagna (costamagnagianfranco)]

Brian, the testcase is to backup some TB of data, and try to read them, the index should not be corrupted.
This is why I'm asking upstream to test the package once its in proposed pocket, because I lack the data to test it properly

[Gianfranco Costamagna (costamagnagianfranco)]

I added the testcase to the bug

[Steve Dodd (anarchetic)]

Given this is a *known data corrupting bug* declared by upstream, it would seem really odd to hold up the release for bionic and focal, where the solution is just an upgrade to the upstream version containing the fix! (I can possibly see the argument for being more careful with cherrypicked patches.)

Unless upstream has published a detailed test case there may not be one accessible to us.

[Steve Dodd (anarchetic)]

I've gone through the upstream bug https://github.com/borgbackup/borg/issues/4829 and not found any data which could be used as a test case. While the description of the bug there is quite detailed, I think one would have to be incredibly familiar with borg internals & code to create a reproducer.

[Chris Halse Rogers (raof)]

It looks like this has been noticed on the borgbackup GitHub, resulting in https://github.com/ThomasWaldmann/borg/commit/aa3938a7a53c29d35df8567f4ebba4c2c76e7342 .

As it's just a test, this might be a sensible thing to cherry-pick? It'd turn the test case from “back up lots of data, and hope you hit the edge case” to “the tests, run as a part of the build, complete without errors”.

[Gianfranco Costamagna (costamagnagianfranco)]

Hello, yes, I requested that test and I will cherry-pick.

I tried right now a build with that added test in my ppa (but no fix)
and it failed
https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/costamagnagianfranco-ppa/+build/20359561

the very same build (so no patch for the corruption issue) was good without the added test
https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/costamagnagianfranco-ppa/+build/20359652

Now, I tested the fixed package with that added testcase and it was good.
https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/locutusofborg-ppa/+build/20359659

So, reuploading in unapproved queue with the added testcase looks really useful. If the build is good, we can assume the runtime is good too, since the code triggered is exactly the same.

[Gianfranco Costamagna (costamagnagianfranco)]

and reuploaded!

[Gianfranco Costamagna (costamagnagianfranco)]

I uploaded 1.1.15 which contains the upstream fix and new regression test for this particular bug.
If it builds, we can assume we are fine

[Timo Aaltonen (tjaalton)]

Hello Steve, or anyone else affected,

Accepted borgbackup into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/borgbackup/1.0.13-0ubuntu1.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Steve, or anyone else affected,

Accepted borgbackup into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/borgbackup/1.1.15-1~ubuntu1.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Steve, or anyone else affected,

Accepted borgbackup into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/borgbackup/1.1.15-1~ubuntu1.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Steve, or anyone else affected,

Accepted borgbackup into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/borgbackup/1.1.15-1~ubuntu1.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Steve, or anyone else affected,

Accepted xxhash into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xxhash/0.7.3-1~ubuntu18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Steve, or anyone else affected,

Accepted borgbackup into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/borgbackup/1.1.15-1~ubuntu1.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Gianfranco Costamagna (costamagnagianfranco)]

I checked all of them, they install correctly, they upgrade correctly, they run correctly, the testsuite showing the regression in index corruption works correctly, and the regression is now shown anymore. The autopkgtest was good for every architecture, so I can say that this is so far so good. Marking as verification done then.

If anybody else wants to do some different and additional testing, please go ahead!

[Gianfranco Costamagna (costamagnagianfranco)]

btw there is a bug where the embedded liblzf is still used, so the runtime dependency on liblzf is not yet added. This is not a problem, and on the next update the full system liblzf will be used
If you are interested in the fix, its available there:
http://launchpadlibrarian.net/516662518/borgbackup_1.1.15-1_1.1.15-2.diff.gz

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for borgbackup has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package borgbackup - 1.1.15-1~ubuntu1.20.10.1

---------------
borgbackup (1.1.15-1~ubuntu1.20.10.1) groovy; urgency=medium

  * New upstream release
    - cherry pick upstream patch and release to fix hashindex corruption bug
      (LP: #1877844)

 -- Gianfranco Costamagna <email address hidden> Fri, 08 Jan 2021 14:59:26 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package borgbackup - 1.1.15-1~ubuntu1.20.04.1

---------------
borgbackup (1.1.15-1~ubuntu1.20.04.1) focal; urgency=medium

  * New upstream release
    - cherry pick upstream patch and release to fix hashindex corruption bug
      (LP: #1877844)

 -- Gianfranco Costamagna <email address hidden> Fri, 08 Jan 2021 14:59:26 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package borgbackup - 1.1.15-1~ubuntu1.18.04.2

---------------
borgbackup (1.1.15-1~ubuntu1.18.04.2) bionic; urgency=medium

  * New upstream release
    - cherry pick upstream patch and release to fix hashindex corruption bug
      (LP: #1877844)
    - Lower compat level to 11

 -- Gianfranco Costamagna <email address hidden> Sat, 09 Jan 2021 09:22:14 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package xxhash - 0.7.3-1~ubuntu18.04.2

---------------
xxhash (0.7.3-1~ubuntu18.04.2) bionic; urgency=medium

  * Lower compat level to 11 to let it build with older debhelper
    - LP: #1877844

 -- Gianfranco Costamagna <email address hidden> Fri, 08 Jan 2021 19:03:49 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package borgbackup - 1.0.13-0ubuntu1.16.04.1

---------------
borgbackup (1.0.13-0ubuntu1.16.04.1) xenial; urgency=medium

  * New upstream release
  * debian/patches/4968.patch:
  * debian/patches/5533.patch:
    - cherry pick upstream patch and release to fix hashindex corruption bug
      (LP: #1877844)
    - cherry pick also un upstream patch to add a regression test for the
      corruption bug

 -- Gianfranco Costamagna <email address hidden> Mon, 12 Oct 2020 20:37:56 +0200