Duplicity leaks gigabytes of memory while using high CPU

[Ross Gammon]

This machine is being used a lot to cast to Chromecast using Chromium. Some days Chromium is regularly freezing every 10 minutes or so. It is necessary to restart the computer to get casting again. Regularly, the freeze causes the "force close" or "wait" pop up to appear. If I am quick to react, I can quit Chromium and a reboot gets things going again. But if I am too slow, sometimes the whole desktop freezes including the mouse and I need to Alt-SysRq+B.

If I am casting and during a boring bit I want to check my emails, then running Thunderbird causes the whole system to slow down, and the casting to reduce resolution and only update the screen every 10 seconds or so.

Please let me know if there is anything I can do to help diagnose the problem.

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: gnome-shell 45.0-1ubuntu2
ProcVersionSignature: Ubuntu 6.5.0-10.10.1-lowlatency 6.5.3
Uname: Linux 6.5.0-10-lowlatency x86_64
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Thu Nov 16 17:52:42 2023
DisplayManager: gdm3
RelatedPackageVersions: mutter-common 45.0-3ubuntu3.1
SourcePackage: gnome-shell
UpgradeStatus: Upgraded to mantic on 2023-11-06 (10 days ago)

[Daniel van Vugt (vanvugt)]

Thanks for the bug report.

When the system is slow please run:

  free -h > free.txt
  ps aux --sort -rss > ps.txt

and attach the resulting text files here.

[Ross Gammon (rosco2)]

It took a while for the slowdown to happen today, even with my daily backup, Thunderbord, firefox and Chromium all running.
I include two versions of the requested files. One where the desktop was running OK, and a second version where I only just managed to type in the commands in the terminal before the mouse froze. Firefox was the first to show the "wait or force close" dialog, but the terminal was also slow to register each keypress.

[Ross Gammon (rosco2)]

Thanks Daniel.

The reason I assigned gnome-shell is because I run Ubuntu Studio on 3 other machines, but they have the default KDE/plasma desktop, and they all work fine. This machine has standard Ubuntu (Gnome) with the Ubuntu Studio packages installed on top.

Of course it could also be hardware related.

[Ross Gammon (rosco2)]

It looks like it is duplicity that is using up all of my memory and CPU cycles. If I stop the daily backup after I turn the computer on (after it automatically starts up and before I open any other apps), then I have problem free use of the computer for the rest of the day.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[smelis (stefanmelis)]

Some more info that might help solve it. I'm seeing this too on Ubuntu 22.04.4 LTS
Duplicity slowly started eating away at memory recently, until it consumes all and OOM's.

ps aux:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
user 274550 99.7 69.7 140278804 137900288 ? Rsl 10:53 41:10 /usr/bin/python3.10 /usr/bin/duplicity collection-status --include=/home/user/.cache/deja-dup/metadata --exclude=/home/user/snap/*/*/.cache --exclude=/home/user/.var/app/*/cache --include=/data/dig/docker_volume --exclude=/home/user/.local/share/Trash --exclude=/home/user/.cache/deja-dup --exclude=/home/user/.cache --include=/home/user --exclude=/sys --exclude=/run --exclude=/proc --exclude=/dev --exclude=/var/tmp --exclude=/tmp --exclude=/media/user/backup/gargantua --exclude=** --exclude-if-present=CACHEDIR.TAG --exclude-if-present=.deja-dup-ignore gio+file:///media/user/backup/gargantua --no-encryption --verbosity=9 --timeout=120 --archive-dir=/home/user/.cache/deja-dup --tempdir=/tmp --log-fd=16

After 20 minutes more, all RAM is consumed:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
user 274550 99.3 96.8 194902916 191371264 ? Rsl 10:53 63:42 /usr/bin/python3.10 /usr/bin/duplicity collection-status ...

free:
               total used free shared buff/cache available
Mem: 188Gi 185Gi 1,2Gi 751Mi 1,3Gi 345Mi
Swap: 0B 0B 0B

/var/log/kern.log OOM, monitor kills process:

Jun 4 11:59:41 gargantua kernel: [245789.662663] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[e-mail]/app.slice/app-gnome-org.gnome.DejaDup.Monitor-6739.scope,task=duplicity,pid=274550,uid=1000
Jun 4 11:59:41 gargantua kernel: [245789.662676] Out of memory: Killed process 274550 (duplicity) total-vm:196347780kB, anon-rss:192909824kB, file-rss:1792kB, shmem-rss:0kB, UID:1000 pgtables:382104kB oom_score_adj:0

What might be a hint as to what's going on are the repeated:

Jun 4 12:22:53 gargantua kernel: [247182.016664] ntfs3: 121981 callbacks suppressed
Jun 4 12:22:53 gargantua kernel: [247182.016667] ntfs3: sda1: ino=774bf, "gargantua" Looks like your dir is corrupt

that fill up /var/log/kern.log

The media the backup is written to is an external Seagate with NTFS fs (which seems to have a corrupt "gargantua" directory, which is where the backup should go).