/etc/cron.daily/dpkg tries to tar corrupt (file system corruption?) file in /var/lib/dpkg/alternatives w/o checking if these are good

[Roy Reshef]

Hi there,

For a while I have noticed that every once in a while, I have a CPU spike "out of the blue". I noticed on gnome-system-monitor it is a gzip process taking 100% of one core for minutes on end, but didn't catch which file it was processing.
Today I caught it red-handed, it was gzip -f -9 .//alternatives.tar.0 . It took it 16 minutes to complete!
Reading more I understood it is part of /etc/cron.daily/dpkg backup script.
I then run the following command:
$ ll /var/backups/
total 136063888
drwxr-xr-x 2 root root 4096 Jun 17 07:58 ./
drwxr-xr-x 15 root root 4096 Apr 13 13:11 ../
-rw-r--r-- 1 root root 138512814080 Jun 17 07:46 alternatives.tar.0
-rw-r--r-- 1 root root 134867846 Jun 11 07:44 alternatives.tar.1.gz
-rw-r--r-- 1 root root 134867841 Jun 10 07:46 alternatives.tar.2.gz
-rw-r--r-- 1 root root 134867846 Jun 4 07:47 alternatives.tar.3.gz
-rw-r--r-- 1 root root 134867841 Jun 2 07:45 alternatives.tar.4.gz
-rw-r--r-- 1 root root 134867848 May 28 07:47 alternatives.tar.5.gz
-rw-r--r-- 1 root root 134867840 May 26 07:46 alternatives.tar.6.gz
...
Yes, that's almost 140GB! (and while the script is running the previous alternatives.tar.0 is kept too, so double the size).
I then examined the files the script is tarring in /var/lib/dpkg/alternatives/ . Most files were normal, up to 1300 bytes or so. But two massive exceptions:
$ find /var/lib/dpkg/alternatives -type f -size +1300c -exec ls -l {} \;
-rw-r--r-- 1 root root 137438954095 Apr 17 2294 ex
-rw-r--r-- 1 16384 32 1073742481 Feb 2 06:20 view

I do not know how these two got corrupted and ended up with these humongous files, weird date and/or owner/group, I recall a file system corruption a few months ago but I don't think it happened during apt/dpkg. No idea.
In any case, these are all related to vim package.

Trying to run now
$ sudo update-alternatives --remove-all ex
update-alternatives: error: /var/lib/dpkg/alternatives/ex corrupt: line not terminated while trying to read status
$ sudo update-alternatives --remove-all view
update-alternatives: error: /var/lib/dpkg/alternatives/view corrupt: invalid status
So dpkg knows these files are corrupt.

I have solved the problem easily by running
$ sudo rm -f /etc/alternatives/ex /var/lib/dpkg/alternatives/ex
$ sudo rm -f /var/lib/dpkg/alternatives/view
$ sudo rm -f /var/backups/alternatives.tar.0 /var/backups/alternatives.tar.*.gz
$ sudo apt install --reinstall vim

Now I run for test
$ sudo /etc/cron.daily/dpkg
which took about 1-2 seconds, and then
$ ll /var/backups/
total 7312
drwxr-xr-x 2 root root 4096 Jun 17 08:32 ./
drwxr-xr-x 15 root root 4096 Apr 13 13:11 ../
-rw-r--r-- 1 root root 122880 Jun 17 08:32 alternatives.tar.0
...

My only request from you is - as dpkg can see easily that files are corrupt, if these are - DO NOT add them to the tar file (or do not do that if their size is unreasonable). These are supposed to be tiny files, but you can see what a corrupt one (or two) enormous files in /var/lib/dpkg/alternatives can do to the system. Thank you!

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: dpkg 1.19.7ubuntu3.2
ProcVersionSignature: Ubuntu 5.4.0-120.136-generic 5.4.189
Uname: Linux 5.4.0-120-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu27.24
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Fri Jun 17 18:57:41 2022
InstallationDate: Installed on 2019-08-24 (1028 days ago)
InstallationMedia: Ubuntu 18.04.3 LTS "Bionic Beaver" - Release amd64 (20190805)
SourcePackage: dpkg
UpgradeStatus: No upgrade log present (probably fresh install)