dejadup excessive memory use resulting in machine lock!

[cat]

as of 8-10 days ago, dejadup is suddenly using all memory (32gb) and then ends up locking up the machine, requiring a hard reboot.

DejaDup: 44.0
Ubuntu 23.04

$ dconf dump /org/gnome/deja-dup/
[/]
backend='drive'
delete-after=0
last-backup='2023-05-26T02:49:58.562918Z'
last-restore='2023-05-18T19:29:20.209321Z'
last-run='2023-06-03T01:34:11.755126Z'
nag-check='2023-04-11T05:58:10.614182Z'
periodic=false
periodic-period=7
prompt-check='2023-04-11T05:21:02.728403Z'
tool='duplicity'
window-height=775
window-width=992

[drive]
folder='BFG'
icon='. GThemedIcon drive-harddisk-usb drive-harddisk drive drive-harddisk-usb-symbolic drive-harddisk-symbolic drive-symbolic'
name='BACKUP'
uuid='004AA50E4AA50210'

---
Operating System: Kubuntu 23.04
KDE Plasma Version: 5.27.4
KDE Frameworks Version: 5.104.0
Qt Version: 5.15.8
Kernel Version: 6.2.0-20-generic (64-bit)
Graphics Platform: X11
Processors: 12 × AMD Ryzen 5 5600X 6-Core Processor
Memory: 31.0 GiB of RAM
Graphics Processor: NVIDIA GeForce GTX 1080/PCIe/SSE2

[Michael Terry (mterry)]

Fascinating, thanks for the report! Sorry Deja Dup is giving you hassle.

Have you noticed it happening again since then? Was deja-dup active before this happened?

[cat (custodiet)]

Yes its happening every time i start it. Even after i apt uninstall,
purge it and re-install it.

Used to work fine, a couple of months ago.

On 6/15/23 20:03, Michael Terry wrote:
> Fascinating, thanks for the report! Sorry Deja Dup is giving you hassle.
>
> Have you noticed it happening again since then? Was deja-dup active
> before this happened?
>
> ** Changed in: deja-dup
> Status: New => Incomplete
>

[Michael Terry (mterry)]

That's troubling, but I'm not sure how to debug this.. hmm.. I don't often run under Kubuntu - I wonder if we're hitting some error flow (because some piece of GNOME isn't present) and we are leaking memory in that flow.

[cat (custodiet)]

I’m Happy to debug for you guys, if you tell me what to do. Lemme know.
I’ll try running under Gnome vs Plasma on Dane machine.

[ajeshp (ajesh-prabhu)]

I am having the same issue;

DejaDup: 44.0
Ubuntu 23.04

Operating System: Ubuntu 23.04
Window manager: i3
Kernel Version: 6.2.0-24-generic (64-bit)
Graphics Platform: X11
Processors: Intel i7-2600 (8) @ 3.800GHz
Memory: 19.5.0 GiB of RAM
Graphics Processor: AMD ATI Radeon HD 6450/7450/8450 / R5 230 OEM

Was able to resolve the issue by changing the backup location to a new folder. The memory leak didn't occur when the new backup was progressing.

[Michael Terry (mterry)]

That's odd that changing folders would change things... It might just be a flaky error state?

Cat - I appreciate the offer to help debug, but memory issues are famously hard to debug, especially for Gtk apps with their reference counting. (Which you would think would help, but figuring out *why* the references are wrong...)

The easiest way to help would be to get reliable reproduction steps -- if I can reproduce this, I can do the hard memory debugging, where knowledge of the source code would come in handy.

[J-Paul BERARD (arverne)]

Same issue today with Ubuntu 22.04 LTS (classic version : GNOME).
We are using Déjà-dup for years without problems.
The backup is made on an external hard drive (USB).
The memory used increases continually until the system freezes (8 Go Ram) after about 10 min. The Déjà-dup window displays “Preparing…”
I have deleted the cache folder : no improvement.

[J-Paul BERARD (arverne)]

New important information :
Like I explained above, I had the same issue.
I found the problem ! The file system on the external disk was corrupted ! (external SSD disk Transcend, formated as NTFS).
Without using Deja-Dup, when I tried to open the backup folder on this external disk, I got the same issue : memory used increasing until the computer been frozen.
The Ubuntu disk utilities said "no problem on the disk".

To solve the problem :
1 - connect the external drive to a Windows© computer and use the command : chkdsk /f E: (if your disk is identified as E:). The errors are detected and corrected. (Maybe Deja-Dup will work again like usual ; not tested). The backup folder can now be deleted (or be used again).
2- To backup : create a new backup folder on the external disk with a DIFFERENT name. Run the backup (new complete backup ; it's what I have done first). Not necessary of course, if the backup works again after the 1.

[zozio32 (remy-pascal)]

same here, duplicity is eating all the memory, then the swap, then it crashes the computer. I am on Ubuntu 22.04, quite an old computer (8 GB ram max).
It never did that before.

I'll try to solution above a let you know...

[cat (custodiet)]

As J-Paul pointed out and I can confirm the problem is if you have an NTFS drive thats gets "dirty"/file system gets corrupted, due to computer crash, unsafe mounting etc, Deja Dup starts using all memory (32gb) and then ends up locking up the machine, requiring a hard reboot.

J-Paul's solution works and I can confirm everything is back to normal after using Windows to chkdsk /f.

[Michael Terry (mterry)]

Oh that's exciting news! I will try to reproduce. Thanks all for your reports.

[Michael Terry (mterry)]

OK initial attempts at causing a corrupted file system aren't successful in triggering this bug. I tried by making a bunch of file copies to an external drive and unplugging it mid-copy.

It either seems fine or nautilus can't mount it at all. In neither case does my memory explode.

I will try again later, but if anyone has suggestions, let me know.

[cat (custodiet)]

The corrupted fs needs to be NTFS.

[Launchpad Janitor (janitor)]

[Expired for Déjà Dup because there has been no activity for 60 days.]

[Cory Fisher (coryjamesfisher)]

I had the same issue as Paul. NTFS corrupt drive crashes Ubuntu by running it out of memory. Thanks for documenting the fix here. My backup kept doing this to me when it would start. Typical oom situation. Unable to even drop to a shell.

[Hugo Beltrán (jiug)]

Tried J-Paul's solution running Windows on VirtualBox and it correctly identified the orphaned duplicity files from a previously failed backup.

Back in Ubuntu Studio 22.04 the old backup is running normally as I type this.

Huge thanks to everyone commenting on this post.