Timeshift 21.09.1-1 broken after Rsync upgrade to 3.2.7-0ubuntu0.22.04.2

[Rogério Satriani Rosa]

[ Impact ]

Timeshift is broken after upgrade to 21.09.1-1. This is because of
a change in behaviour by rsync; rsync 3.2.4 changed how the locale's
worked out. From https://download.samba.org/pub/rsync/NEWS#3.2.4

"A long-standing bug was preventing rsync from figuring out the current locale's decimal point character, which made rsync always output numbers using the "C" locale. Since this is now fixed in 3.2.4, a script that parses rsync's decimal numbers (e.g. from the verbose footer) may want to setup the environment in a way that the output continues to be in the C locale. For instance, one of the following should work fine:

    export LC_ALL=C.UTF-8
"

This broke timeshift and the workaround mentioned in rsync release
notes needed to be applied to fix the locale.

 * justification for backporting the fix to the stable release.

While the behaviour change is external to timeshift (it's in rsync),
for anyone using newer rsync it broke timeshift completely and worse,
users aren't even aware that their existing snapshots and backups are no longer
working.

The said workaround has been applied in upstream timeshift
https://github.com/teejee2008/timeshift/pull/904

But we haven't got this fix in Jammy.

[ Test Plan ]

It's readily reproducible on Jammy:
1. Install Ubuntu 22.04 LTS
2. Install the latest Timeshift
3. Change locale to any language with a (,) decimal seperator (e.g. German)
4. (a) Launch Timeshift, start rsync-backup, log message said that rsync failed to create backup.
   (b) Can use CLI too with `sudo timeshift --create` and see it fail.
5. Then use the timshift package from PPA https://launchpad.net/~pponnuvel/+archive/ubuntu/jammy-timeshift
   which contains the fix and `sudo timeshift --create` will succeed.

[ Where problems could occur ]

This changes the locale to "C.UTF-8". If anyone relies existing broken behaviour
that won't work anymore. Similarly, if rsync doesn't changes behaviour again based
on locale, timeshift might start failing again.

[ Other Info ]

I've looked into Focal, Jammy, Mantic, and Noble for this issue.

Focal is using older rsync (before the locale change [0]), so it's unaffected.
Both Mantic, and Noble have the upstream fix [1] incorporated (fixed through new releases).
So they don't have this issue either.

Thus this is a Jammy-only backport of the fix [1].

[0] https://download.samba.org/pub/rsync/NEWS#3.2.4
[1] https://github.com/teejee2008/timeshift/pull/904

[Ralf W. (daphcnual)]

I can confirm that behavior for these versions: ubuntu 22.04.2, rsync 3.2.7-0ubuntu0.22.04.2, timeshift 21.09.1-1.

On my system the last successful timeshift snapshot was performed manually directly before upgrading rsync via software updater. Since then all automated snapshots (boot + all scheduled ones) fail to generate a snapshot. Triggering an "ondemand"-snapshot DOES perform a snapshot, but it does never appear in GUI list.

There is NO indication that automated snapshot are not performed if one does not startup the timeshift GUI or look into the logfiles - therefore I consider this bug is quite problematic.

The logfile does not contain any useful information for me:
[09:08:18] ------------------------------------------------------------------------------
[09:08:18] Creating new snapshot...(RSYNC)
[09:08:18] Saving to device: /dev/nvme0n1p2, mounted at path: /run/timeshift/backup
[09:08:18] Linking from snapshot: 2023-03-07_09-05-09
[09:08:18] Main: save_exclude_list_for_backup()
[09:08:18] Main: create_exclude_list_for_backup()
[09:08:18] Main: create_exclude_list_for_backup(): exit
[09:08:18] Synching files with rsync...
[09:08:18] RsyncTask:execute()
[09:08:18] rsync -aii --recursive --verbose --delete --force --stats --sparse --delete-excluded --link-dest='/run/timeshift/backup/timeshift/snapshots/2023-03-07_09-05-09/localhost/' --log-file='/run/timeshift/backup/timeshift/snapshots/2023-03-10_09-08-18/rsync-log' --exclude-from='/run/timeshift/backup/timeshift/snapshots/2023-03-10_09-08-18/exclude.list' --delete-excluded '/' '/run/timeshift/backup/timeshift/snapshots/2023-03-10_09-08-18/localhost/'
[09:08:18] RsyncTask:prepare(): saved: /tmp/q1JzC4r1/2023-03-10_09-08-18/script.sh
[09:08:18] AsyncTask: child_pid: 10234
[09:08:23] AsyncTask: finish(): enter
[09:08:23] exit_code: 0
[09:08:24] E: rsync returned an error
[09:08:24] E: Failed to create new snapshot
[09:08:24] Failed to create snapshot
[09:08:24] ------------------------------------------------------------------------------

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ronald Huetter (user-0209)]

Same here. Again to stress: it SILENTLY stopped creating the snapshots. This is VERY problematic.

[José Marinho (jmarinho)]

This bug was reported on the Linux Mint site on Github: https://github.com/linuxmint/timeshift/issues/152

As suggested on this comment: https://github.com/linuxmint/timeshift/issues/152#issuecomment-1470141113 downgrading rsync version fixes the issue too.

[vasilis34 (vmanolop)]

Same here on ubuntu 22.04.2. Feels like upgrading to Timeshift version 22.11.2 from Mint PPA works fine for backup and it is a preferable workaround since you don't need to revert to an old security patched rsync. I didn't have the chance though to test restore. Has anybody test it with this version?

[Gabriele (prodeguerriero)]

Just adding that triggering an on-demand snapshot via the command line works as expected.

On top of it, it cleans up all incomplete snapshots that it finds along the way.

[Andre Ruiz (andre-ruiz)]

According to:

https://github.com/linuxmint/timeshift/issues/218

The new version available at the PPA fixes the issue. Maybe ubuntu should consider a backport?

Silently failing backups is very dangerous. I just realized they were failing, after probably many weeks of having no backups.

[Andre Ruiz (andre-ruiz)]

In any case, reverting rsync from 3.2.7-0ubuntu0.22.04.2 to 3.2.3-8ubuntu3 and then holding it also works as a temporary solution.

[Zoran (zoran-ubuntu)]

Timeshift from the Mint GitHub repo works fine on Ubuntu. Any update on this? This issue is almost a year old now. There is a working, regularly updated codebase and yet we still have this problem on Ubuntu.

[Rogério Satriani Rosa (maelstrom2000)]

Zoran,

For me, at least, it's been fixed for a while, with regular Ubuntu
updates. Current installed version on my PC, which I believe is in
Ubuntu 22.04 repo, is: Timeshift 23.07.1-0~202307081612~ubuntu22.04.1

This version is working fine. I also have removed the Mint PPA, so this
is coming from Ubuntu repo.

Hope this helps

Em 18/02/2024 14:02, Zoran escreveu:
> Timeshift from the Mint GitHub repo works fine on Ubuntu. Any update on
> this? This issue is almost a year old now. There is a working, regularly
> updated codebase and yet we still have this problem on Ubuntu.
>

[Ralf W. (daphcnual)]

On my Ubuntu 22.04 timeshift is still version 21.09.1-1 (02 Oct 2021).
No newer version is available from repository.

For the average user timeshift-snapshots stopped working ONE YEAR AGO - afaik w/o any notification (apart from entries in log files). For this reason, I consider this error to be rather critical and its importance should really be increased.

[Rudra Trivedi (rudratrivedi)]

This is due to a change in rsync 3.2.4, related to decimal separators in European language locales vs English locales.

The upstream fix for this is here, and I am escalating it internally to get in included in the Ubuntu releases:
https://github.com/teejee2008/timeshift/pull/904

@maelstrom2000: the version you currently have installed is still from the PPA, you would have to uninstall and reinstall after removing the PPA to get the version from the Ubuntu repos.

[Rogério Satriani Rosa (maelstrom2000)]

Rudra, you are correct. The version installed for me is still from Mint
PPA. It shows as "local install" in Synaptic as I had already removed
the Mint PPA some time ago. Clearly it never got updated from Ubuntu
repo since it was replace by Mint package. Thanks for looking into this
and alert me.

Em 28/03/2024 12:58, Rudra Trivedi escreveu:
> This is due to a change in rsync 3.2.4, related to decimal separators in
> European language locales vs English locales.
>
> The upstream fix for this is here, and I am escalating it internally to get in included in the Ubuntu releases:
> https://github.com/teejee2008/timeshift/pull/904
>
>
> @maelstrom2000: the version you currently have installed is still from the PPA, you would have to uninstall and reinstall after removing the PPA to get the version from the Ubuntu repos.
>

[Ponnuvel Palaniyappan (pponnuvel)]

Mantic and Noble have newer packages of timeshift which contain the upstream fix [0].

Focal is at 3.1.3-8ubuntu0.7 which predates the rsync change (3.2.4). So this issue affects just Jammy.

I've created a PPA with this fix for Jammy [1] to test. I'll do an SRU for Jammy after confirmation.

[0] https://github.com/teejee2008/timeshift/pull/904
[1] https://launchpad.net/~pponnuvel/+archive/ubuntu/jammy-timeshift

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "timeshift_jammy_diff.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Dave Jones (waveform)]

Patch looks mostly good. Needed a couple of trivial changes (run update-maintainer to deal with maintainer address in d/control, and add bug-ref in d/changelog), but I've sponsored with those changes, thanks!

[Timo Aaltonen (tjaalton)]

Hello Rogério, or anyone else affected,

Accepted timeshift into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/timeshift/21.09.1-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ponnuvel Palaniyappan (pponnuvel)]

Set the locale variables to de_DE.UTF-8.

Then testing timeshift:

ubuntu@bronzor:~$ dpkg -l | grep timeshift
ii timeshift 21.09.1-1 amd64 System restore utility
ubuntu@bronzor:~$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=
ubuntu@bronzor:~$ sudo timeshift --create
Mounted '/dev/sda1' at '/run/timeshift/backup'
------------------------------------------------------------------------------
Creating new snapshot...(RSYNC)
Saving to device: /dev/sda1, mounted at path: /run/timeshift/backup
Linking from snapshot: 2024-04-26_12-40-26
Synching files with rsync...
E: rsync returned an error
E: Failed to create new snapshot
Failed to create snapshot
------------------------------------------------------------------------------
Removing snapshots (incomplete):
------------------------------------------------------------------------------
Removing '2024-04-26_13-01-21'...
 73.52% complete (00:00:00 remaining)
(process:1524): GLib-GIO-CRITICAL **: 13:01:33.226: g_output_stream_clear_pending: assertion 'G_IS_OUTPUT_STREAM (stream)' failed

(process:1524): GLib-GIO-CRITICAL **: 13:01:33.226: g_output_stream_clear_pending: assertion 'G_IS_OUTPUT_STREAM (stream)' failed
Removed '2024-04-26_13-01-21'
------------------------------------------------------------------------------

====

After installing timeshift from jammy-prposed, the same command works:

ubuntu@bronzor:~$ dpkg -l | grep timeshift
ii timeshift 21.09.1-1ubuntu1 amd64 System restore utility
ubuntu@bronzor:~$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=
ubuntu@bronzor:~$ sudo timeshift --create

/dev/sda1 is mounted at: /run/timeshift/backup, options: rw,relatime,stripe=64

------------------------------------------------------------------------------
Creating new snapshot...(RSYNC)
Saving to device: /dev/sda1, mounted at path: /run/timeshift/backup
Linking from snapshot: 2024-04-26_12-40-26
Synching files with rsync...
Created control file: /run/timeshift/backup/timeshift/snapshots/2024-04-26_14-50-25/info.json
RSYNC Snapshot saved successfully (4s)
Tagged snapshot '2024-04-26_14-50-25': ondemand
------------------------------------------------------------------------------
ubuntu@bronzor:~$

[Ralf W. (daphcnual)]

Hello Timo,

after allowing packages from 'proposed' I manually upgraded these packages:

    rsync (version 3.2.3-8ubuntu3) ... upgraded to version 3.2.7-0ubuntu0.22.04.2
    timeshift (version 21.09.1-1) ... upgraded to version 21.09.1-1ubuntu1

(rsync had previously been held back)

Then I successfully ran
- an on-demand snapshot from timeshift gui
- an automated boot-snapshot

Both worked w/o problems recognized and show up in timeshift gui as expected.

Thank you all for fixing that issue! :-)

[Rogério Satriani Rosa (maelstrom2000)]

As Ralf mentioned below, I too confirm it works now as intended. Thanks
for fixing it. All best.

Em 27/04/2024 04:28, Ralf W. escreveu:
> Hello Timo,
>
> after allowing packages from 'proposed' I manually upgraded these
> packages:
>
> rsync (version 3.2.3-8ubuntu3) ... upgraded to version 3.2.7-0ubuntu0.22.04.2
> timeshift (version 21.09.1-1) ... upgraded to version 21.09.1-1ubuntu1
>
> (rsync had previously been held back)
>
> Then I successfully ran
> - an on-demand snapshot from timeshift gui
> - an automated boot-snapshot
>
> Both worked w/o problems recognized and show up in timeshift gui as
> expected.
>
> Thank you all for fixing that issue! :-)
>

[Ponnuvel Palaniyappan (pponnuvel)]

@SRU team, is this good to move to jammy from jammy-proposed?

[Launchpad Janitor (janitor)]

This bug was fixed in the package timeshift - 21.09.1-1ubuntu1

---------------
timeshift (21.09.1-1ubuntu1) jammy; urgency=medium

  * d/patches: force rsync to output decimals in C locale. (LP: #2009885)

 -- Ponnuvel Palaniyappan <email address hidden> Thu, 28 Mar 2024 18:29:41 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for timeshift has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.