Comment 53 for bug 1977919

Revision history for this message
Bill B. (n1sni) wrote :

So this was painful for us. AWS hosted server running Ubuntu 20.04.4 LTS. Just for others, here are the steps we took thanks to the other comments here:

We had to force shut down the machine and wait (aws console). Then we got this script running, and started the machine back up:

while true; do
  ssh -o ConnectTimeout=2 -i <.pem file> ubuntu@<host> "sudo systemctl disable docker.service; sudo systemctl disable containerd.service"
done

---
Now, ssh into machine normally, and this is what we ran in our case:
sudo -i
cd /boot/grub
grep -Ei 'submenu|menuentry ' /boot/grub/grub.cfg | sed -re "s/(.? )'([^']+)'.*/\1 \2/"
The result on mine was:
---
menuentry Ubuntu
submenu Advanced options for Ubuntu
        menuentry Ubuntu, with Linux 5.13.0-1028-aws # <-- BAD BAD BAD
        menuentry Ubuntu, with Linux 5.13.0-1028-aws (recovery mode)
        menuentry Ubuntu, with Linux 5.13.0-1025-aws # <-- what we want
        menuentry Ubuntu, with Linux 5.13.0-1025-aws (recovery mode)
        menuentry Ubuntu, with Linux 5.4.0-1029-aws
        menuentry Ubuntu, with Linux 5.4.0-1029-aws (recovery mode)
menuentry Ubuntu 20.04.4 LTS (20.04) (on /dev/nvme0n1p1)
submenu Advanced options for Ubuntu 20.04.4 LTS (20.04) (on /dev/nvme0n1p1)
        menuentry Ubuntu (on /dev/nvme0n1p1)
        menuentry Ubuntu, with Linux 5.13.0-1028-aws (on /dev/nvme0n1p1)
        menuentry Ubuntu, with Linux 5.13.0-1028-aws (recovery mode) (on /dev/nvme0n1p1)
        menuentry Ubuntu, with Linux 5.13.0-1025-aws (on /dev/nvme0n1p1)
        menuentry Ubuntu, with Linux 5.13.0-1025-aws (recovery mode) (on /dev/nvme0n1p1)
        menuentry Ubuntu, with Linux 5.4.0-1029-aws (on /dev/nvme0n1p1)
        menuentry Ubuntu, with Linux 5.4.0-1029-aws (recovery mode) (on /dev/nvme0n1p1)
---
So we wanted off 1028, and back to 1025. Edited:
vi /etc/default/grub
changed:
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.13.0-1025-aws"

NOTE: the first half is the "MENUENTRY" from above, then ">" and then the submenu.
Saved

run:
grub-mkconfig -o /boot/grub/grub.cfg
sudo reboot

then, after reboot:

sudo systemctl enable docker.service; sudo systemctl enable containerd.service
sudo reboot

Should be back up! It seems like upgrading to the newest kernel also helps based on the above, but will try that later.

Hope this helps someone like you all helped us!!