So this was painful for us. AWS hosted server running Ubuntu 20.04.4 LTS. Just for others, here are the steps we took thanks to the other comments here:
We had to force shut down the machine and wait (aws console). Then we got this script running, and started the machine back up:
while true; do
ssh -o ConnectTimeout=2 -i <.pem file> ubuntu@<host> "sudo systemctl disable docker.service; sudo systemctl disable containerd.service"
done
---
Now, ssh into machine normally, and this is what we ran in our case:
sudo -i
cd /boot/grub
grep -Ei 'submenu|menuentry ' /boot/grub/grub.cfg | sed -re "s/(.? )'([^']+)'.*/\1 \2/"
The result on mine was:
---
menuentry Ubuntu
submenu Advanced options for Ubuntu
menuentry Ubuntu, with Linux 5.13.0-1028-aws # <-- BAD BAD BAD
menuentry Ubuntu, with Linux 5.13.0-1028-aws (recovery mode)
menuentry Ubuntu, with Linux 5.13.0-1025-aws # <-- what we want
menuentry Ubuntu, with Linux 5.13.0-1025-aws (recovery mode)
menuentry Ubuntu, with Linux 5.4.0-1029-aws
menuentry Ubuntu, with Linux 5.4.0-1029-aws (recovery mode)
menuentry Ubuntu 20.04.4 LTS (20.04) (on /dev/nvme0n1p1)
submenu Advanced options for Ubuntu 20.04.4 LTS (20.04) (on /dev/nvme0n1p1)
menuentry Ubuntu (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.13.0-1028-aws (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.13.0-1028-aws (recovery mode) (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.13.0-1025-aws (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.13.0-1025-aws (recovery mode) (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.4.0-1029-aws (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.4.0-1029-aws (recovery mode) (on /dev/nvme0n1p1)
---
So we wanted off 1028, and back to 1025. Edited:
vi /etc/default/grub
changed:
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 5.13.0-1025-aws"
NOTE: the first half is the "MENUENTRY" from above, then ">" and then the submenu.
Saved
So this was painful for us. AWS hosted server running Ubuntu 20.04.4 LTS. Just for others, here are the steps we took thanks to the other comments here:
We had to force shut down the machine and wait (aws console). Then we got this script running, and started the machine back up:
while true; do
ssh -o ConnectTimeout=2 -i <.pem file> ubuntu@<host> "sudo systemctl disable docker.service; sudo systemctl disable containerd.service"
done
--- "Advanced options for Ubuntu>Ubuntu, with Linux 5.13.0-1025-aws"
Now, ssh into machine normally, and this is what we ran in our case:
sudo -i
cd /boot/grub
grep -Ei 'submenu|menuentry ' /boot/grub/grub.cfg | sed -re "s/(.? )'([^']+)'.*/\1 \2/"
The result on mine was:
---
menuentry Ubuntu
submenu Advanced options for Ubuntu
menuentry Ubuntu, with Linux 5.13.0-1028-aws # <-- BAD BAD BAD
menuentry Ubuntu, with Linux 5.13.0-1028-aws (recovery mode)
menuentry Ubuntu, with Linux 5.13.0-1025-aws # <-- what we want
menuentry Ubuntu, with Linux 5.13.0-1025-aws (recovery mode)
menuentry Ubuntu, with Linux 5.4.0-1029-aws
menuentry Ubuntu, with Linux 5.4.0-1029-aws (recovery mode)
menuentry Ubuntu 20.04.4 LTS (20.04) (on /dev/nvme0n1p1)
submenu Advanced options for Ubuntu 20.04.4 LTS (20.04) (on /dev/nvme0n1p1)
menuentry Ubuntu (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.13.0-1028-aws (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.13.0-1028-aws (recovery mode) (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.13.0-1025-aws (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.13.0-1025-aws (recovery mode) (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.4.0-1029-aws (on /dev/nvme0n1p1)
menuentry Ubuntu, with Linux 5.4.0-1029-aws (recovery mode) (on /dev/nvme0n1p1)
---
So we wanted off 1028, and back to 1025. Edited:
vi /etc/default/grub
changed:
GRUB_DEFAULT=
NOTE: the first half is the "MENUENTRY" from above, then ">" and then the submenu.
Saved
run:
grub-mkconfig -o /boot/grub/grub.cfg
sudo reboot
then, after reboot:
sudo systemctl enable docker.service; sudo systemctl enable containerd.service
sudo reboot
Should be back up! It seems like upgrading to the newest kernel also helps based on the above, but will try that later.
Hope this helps someone like you all helped us!!