Comment 5 for bug 1966499

Revision history for this message
Eric Anopolsky (erpo41) wrote :

I arrived at work this morning to find a frozen workstation with a full disk. This bug was one of two relevant search results, so I'm documenting the recovery steps I took here in case it will be helpful to others:

1. Held down the power button until the PC turned off.
2. Waited a few seconds and powered the PC back on.
3. On the grub menu, chose Advanced options for Ubuntu -> Ubuntu, with Linux 5.13.0-35-generic (recovery mode) and waited for the system to boot.
4. On the Recovery Menu, chose the root option and hit enter.
5. Ran the following:

rm /var/log/kern.log # freed up about 800GB
head -n 100000 /var/log/syslog > /var/log/syslog.bak # saved a few copies of the error just in case
rm /var/log/syslog # freed up another 800GB
apt remove linux-image-5.13.0-37-generic linux-image-unsigned-5.13.0-37-generic # it was necessary to remove the second package because that's what apt tried to install when only the first one was specified
reboot # temporary fix is complete

6. Allowed the system to boot normally without intervention.
7. Logged in, opened a terminal, and confirmed that the fix worked:

uname -a
# Expected output: Linux IS01209 5.13.0-35-generic #40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

virsh list
# Expected output:
# Id Name State
#---------------------------------
# 1 my-windows-vm running
#
# Note: My Windows VM is configured to start on boot. For affected users who have a Windows VM that is not configured to start on boot, it may be necessary to start the VM manually before moving on to the next step.

watch du -h /var/log/syslog /var/log/kern.log
# Expected output:
# Every 2.0s: du -h /var/... (hostname and date/time)
#
# 320K /var/log/syslog
# 100K /var/log/kern.log
#
# Note: If those two files stay the same size or grow slowly, the fix worked. If those two files grow rapidly, the fix did not work.

Note that the above steps will also uninstall the linux-generic-hwe-20.04 package, which will prevent future kernel updates. When this bug is resolved, I plan to run (and recommend that others run) `sudo apt install linux-generic-hwe-20.04` to resume kernel updates.