| 2022-04-13 00:55:42 |
Matthew Ruffell |
bug |
|
|
added bug |
| 2022-04-13 00:55:49 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Focal |
|
| 2022-04-13 00:55:49 |
Matthew Ruffell |
bug task added |
|
ec2-hibinit-agent (Ubuntu Focal) |
|
| 2022-04-13 00:55:49 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Impish |
|
| 2022-04-13 00:55:49 |
Matthew Ruffell |
bug task added |
|
ec2-hibinit-agent (Ubuntu Impish) |
|
| 2022-04-13 00:55:49 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-04-13 00:55:49 |
Matthew Ruffell |
bug task added |
|
ec2-hibinit-agent (Ubuntu Jammy) |
|
| 2022-04-13 00:56:15 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Focal): status |
New |
In Progress |
|
| 2022-04-13 00:56:17 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Impish): status |
New |
In Progress |
|
| 2022-04-13 00:56:19 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Jammy): status |
New |
In Progress |
|
| 2022-04-13 00:56:22 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2022-04-13 00:56:24 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Impish): importance |
Undecided |
Medium |
|
| 2022-04-13 00:56:25 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Jammy): importance |
Undecided |
Medium |
|
| 2022-04-13 00:56:27 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Focal): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2022-04-13 00:56:29 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Impish): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2022-04-13 00:56:31 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Jammy): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2022-04-13 00:57:58 |
Matthew Ruffell |
description |
[Impact]
It is not uncommon for users to add a swapfile to their AWS instance, in case they run short of memory. For users that optionally enable Hibernation support, the swapfile generated by ec2-hibinit-agent, /swap-hibinit, needs to always be the highest priority when it comes to suspend the system, since ec2-hibinit-agent sets up /swap-hibinit to be the correct way to suspend and resume via the resume=UUID=<uuid> and resume_offeset=<offset> kernel command line parameters.
ec2-hibinit-agent keeps /swap-hibinit swapoff during normal instance use, and right before Hibernation occurs, /etc/acpi/actions/sleep.sh swapon /swap-hibinit, and calls systemctl hibernate:
do_hibernate() {
if [ -d /run/systemd/system ]; then
systemctl hibernate
case "$2" in
SBTN)
swapon /swap-hibinit && do_hibernate
Something changed between 18.04 and 20.04, such that new swapfiles are added with a lower priority than the previous swapfile when they are swapon:
On Focal and later, we see behaviour like if we simply swapon /swap-hibinit generated by ec2-hibinit-agent, we
see it is -2:
$ sudo swapon /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
/swap-hibinit file 3.9G 0B -2
Turning it off:
$ sudo swapoff /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
Lets add /swapfile in:
$ sudo swapon /swapfile
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
Now we enable /swap-hibinit again, and see it is -3:
$ sudo swapon /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
/swap-hibinit file 3.9G 0B -3
Lets add in another swapfile, /swapfile-second, and we see -2, -3, -4:
$ sudo swapon /swapfile-second
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
/swap-hibinit file 3.9G 0B -3
/swapfile-second file 4G 0B -4
What happens is that if we have a swapfile, say, /swapfile at default priority -2, when we go to hibernate, the swapon in /etc/acpi/actions/sleep.sh will set the priority of /swap-hibinit to -3. systemd / the kernel will then select the highest priority swapfile to hibernate to, in this case /swapfile, which is NOT setup for resume= or resume_offset= on the kernel command line, and hibernation will fail.
Apr 11 21:08:15 ip-172-31-84-225 kernel: [ 240.990073] Adding 4095996k swap on /swap-hibinit. Priority:-3 extents:6 across:4644860k SSFS
This leaves the instance in the "Stopping" state on the EC2 console until it hits the 20 minute timeout, at which point it is force stopped.
The fix is to set the priority when we swapon /swap-hibinit to something higher than any other swapfile, to ensure we hibernate to /swap-hibinit.
[Testcase]
From the EC2 console, select "Launch Instance".
Create a:
- t2.medium
- Ubuntu 20.04, 21.04 or 22.04
- 20gb storage space, advanced > enable encryption > yes.
- Advanced settings > Stop State (Hibernation) Support > Enabled
On boot wait for ec2-hibinit-agent to complete hibinit-agent.service, and see that /swap-hibinit is created, and swapoff.
$ ll /swap-hibinit
Add a swapfile, and switch it on:
$ sudo fallocate -l 4G /swapfile
$ sudo dd if=/dev/zero of=/swapfile bs=1024 count=4194304
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ echo "/swapfile swap swap defaults 0 0" | sudo tee -a /etc/fstab
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
Go back to EC2 console, "Instance State" > "Hibernate".
You will see this in journalctl:
Mar 15 11:41:54 ip-172-31-27-108 kernel: [ 520.121761] Adding 16095656k swap on /swap-hibinit. Priority:-3 extents:13 across:17611176k SSFS
Mar 15 11:41:54 ip-172-31-27-108 root: ACPI action undefined: LNXSLPBN:00
and the instance will not hibernate. EC2 console will report "Stopping" for 20 minutes until it times out and is force stopped.
If you enable the following ppa and install the test ec2-hibinit-agent package:
Hibernation should succeed within a minute or two.
[Where problems could occur]
This change will only affect users of instances where Hibernation has been explicitly enabled, either from the EC2 instance launch advanced settings, or via the "--hibernation-options Configured=true" parameter to the "aws ec2" command. For all other users, including those with swapfiles enabled, this change will have no effect.
We are changing the /swap-hibinit file to be maximum priority right before we hibernate, to ensure it is the swapfile selected to hibernate to. Since we swapoff /swap-hibinit as soon as we resume, /swap-hibinit is used solely for hibernation, and not for regular swap space, so it is unlikely to cause any regressions to users with their own swapfiles configured with various priorities.
A potential risk is users that do not use /swap-hibinit, and use their own swapfile for hibernation, and overwrite the changes ec2-hibinit-agent makes to grub files to set the resume=UUID<uuid> and resume_offset=<offset> values. I believe such users would likely remove or purge the ec2-hibinit-agent package, since hibinit-agent.service runs at startup and re-adds the grub configuration for /swap-hibinit whether you like it or not, and having /swap-hibinit around would waste disk space that you would be paying for. Because of this, I believe that this change will not break users who hibernate to their own swapfiles, because they would have removed ec2-hibinit-agent on instance creation.
[Other info]
Chris Newcomer came across the above upstream bug, which seems to be the same issue:
https://github.com/aws/amazon-ec2-hibinit-agent/issues/20
The reporter, Ben Mares, suggests a patch to /etc/acpi/actions/sleep.sh to either read the value of a bash environment variable swap_priority, or default to 10.
https://github.com/aws/amazon-ec2-hibinit-agent/pull/21
I'm not exactly on board with the environment variable, or the default magic number of 10, as we don't know how our users are setting up swapfiles, and what priorities they set them to. I think we should instead just set the priority to the maximum, 32767 instead. |
[Impact]
It is not uncommon for users to add a swapfile to their AWS instance, in case they run short of memory. For users that optionally enable Hibernation support, the swapfile generated by ec2-hibinit-agent, /swap-hibinit, needs to always be the highest priority when it comes to suspend the system, since ec2-hibinit-agent sets up /swap-hibinit to be the correct way to suspend and resume via the resume=UUID=<uuid> and resume_offeset=<offset> kernel command line parameters.
ec2-hibinit-agent keeps /swap-hibinit swapoff during normal instance use, and right before Hibernation occurs, /etc/acpi/actions/sleep.sh swapon /swap-hibinit, and calls systemctl hibernate:
do_hibernate() {
if [ -d /run/systemd/system ]; then
systemctl hibernate
case "$2" in
SBTN)
swapon /swap-hibinit && do_hibernate
Something changed between 18.04 and 20.04, such that new swapfiles are added with a lower priority than the previous swapfile when they are swapon:
On Focal and later, we see behaviour like if we simply swapon /swap-hibinit generated by ec2-hibinit-agent, we
see it is -2:
$ sudo swapon /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
/swap-hibinit file 3.9G 0B -2
Turning it off:
$ sudo swapoff /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
Lets add /swapfile in:
$ sudo swapon /swapfile
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
Now we enable /swap-hibinit again, and see it is -3:
$ sudo swapon /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
/swap-hibinit file 3.9G 0B -3
Lets add in another swapfile, /swapfile-second, and we see -2, -3, -4:
$ sudo swapon /swapfile-second
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
/swap-hibinit file 3.9G 0B -3
/swapfile-second file 4G 0B -4
What happens is that if we have a swapfile, say, /swapfile at default priority -2, when we go to hibernate, the swapon in /etc/acpi/actions/sleep.sh will set the priority of /swap-hibinit to -3. systemd / the kernel will then select the highest priority swapfile to hibernate to, in this case /swapfile, which is NOT setup for resume= or resume_offset= on the kernel command line, and hibernation will fail.
Apr 11 21:08:15 ip-172-31-84-225 kernel: [ 240.990073] Adding 4095996k swap on /swap-hibinit. Priority:-3 extents:6 across:4644860k SSFS
This leaves the instance in the "Stopping" state on the EC2 console until it hits the 20 minute timeout, at which point it is force stopped.
The fix is to set the priority when we swapon /swap-hibinit to something higher than any other swapfile, to ensure we hibernate to /swap-hibinit.
[Testcase]
From the EC2 console, select "Launch Instance".
Create a:
- t2.medium
- Ubuntu 20.04, 21.04 or 22.04
- 20gb storage space, advanced > enable encryption > yes.
- Advanced settings > Stop State (Hibernation) Support > Enabled
On boot wait for ec2-hibinit-agent to complete hibinit-agent.service, and see that /swap-hibinit is created, and swapoff.
$ ll /swap-hibinit
Add a swapfile, and switch it on:
$ sudo fallocate -l 4G /swapfile
$ sudo dd if=/dev/zero of=/swapfile bs=1024 count=4194304
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ echo "/swapfile swap swap defaults 0 0" | sudo tee -a /etc/fstab
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
Go back to EC2 console, "Instance State" > "Hibernate".
You will see this in journalctl:
Mar 15 11:41:54 ip-172-31-27-108 kernel: [ 520.121761] Adding 16095656k swap on /swap-hibinit. Priority:-3 extents:13 across:17611176k SSFS
Mar 15 11:41:54 ip-172-31-27-108 root: ACPI action undefined: LNXSLPBN:00
and the instance will not hibernate. EC2 console will report "Stopping" for 20 minutes until it times out and is force stopped.
If you enable the following ppa and install the test ec2-hibinit-agent package:
https://launchpad.net/~mruffell/+archive/ubuntu/sf331069-test
Hibernation should succeed within a minute or two.
[Where problems could occur]
This change will only affect users of instances where Hibernation has been explicitly enabled, either from the EC2 instance launch advanced settings, or via the "--hibernation-options Configured=true" parameter to the "aws ec2" command. For all other users, including those with swapfiles enabled, this change will have no effect.
We are changing the /swap-hibinit file to be maximum priority right before we hibernate, to ensure it is the swapfile selected to hibernate to. Since we swapoff /swap-hibinit as soon as we resume, /swap-hibinit is used solely for hibernation, and not for regular swap space, so it is unlikely to cause any regressions to users with their own swapfiles configured with various priorities.
A potential risk is users that do not use /swap-hibinit, and use their own swapfile for hibernation, and overwrite the changes ec2-hibinit-agent makes to grub files to set the resume=UUID<uuid> and resume_offset=<offset> values. I believe such users would likely remove or purge the ec2-hibinit-agent package, since hibinit-agent.service runs at startup and re-adds the grub configuration for /swap-hibinit whether you like it or not, and having /swap-hibinit around would waste disk space that you would be paying for. Because of this, I believe that this change will not break users who hibernate to their own swapfiles, because they would have removed ec2-hibinit-agent on instance creation.
[Other info]
Chris Newcomer came across the above upstream bug, which seems to be the same issue:
https://github.com/aws/amazon-ec2-hibinit-agent/issues/20
The reporter, Ben Mares, suggests a patch to /etc/acpi/actions/sleep.sh to either read the value of a bash environment variable swap_priority, or default to 10.
https://github.com/aws/amazon-ec2-hibinit-agent/pull/21
I'm not exactly on board with the environment variable, or the default magic number of 10, as we don't know how our users are setting up swapfiles, and what priorities they set them to. I think we should instead just set the priority to the maximum, 32767 instead. |
|
| 2022-04-13 00:58:30 |
Matthew Ruffell |
tags |
|
sts |
|
| 2022-04-13 04:48:25 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for focal https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5580292/+files/lp1968805_focal.debdiff |
|
| 2022-04-13 04:49:18 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for impish https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5580293/+files/lp1968805_impish.debdiff |
|
| 2022-04-13 04:50:07 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for jammy https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5580294/+files/lp1968805_jammy.debdiff |
|
| 2022-04-13 08:22:00 |
Ubuntu Foundations Team Bug Bot |
tags |
sts |
patch sts |
|
| 2022-04-13 08:22:08 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2022-04-18 15:32:32 |
Dan Streetman |
bug |
|
|
added subscriber Dan Streetman |
| 2022-04-21 05:24:24 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for bionic https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5581947/+files/lp1968805_bionic.debdiff |
|
| 2022-04-21 05:54:52 |
Matthew Ruffell |
tags |
patch sts |
patch sts sts-sponsor |
|
| 2022-05-11 03:30:41 |
Matthew Ruffell |
description |
[Impact]
It is not uncommon for users to add a swapfile to their AWS instance, in case they run short of memory. For users that optionally enable Hibernation support, the swapfile generated by ec2-hibinit-agent, /swap-hibinit, needs to always be the highest priority when it comes to suspend the system, since ec2-hibinit-agent sets up /swap-hibinit to be the correct way to suspend and resume via the resume=UUID=<uuid> and resume_offeset=<offset> kernel command line parameters.
ec2-hibinit-agent keeps /swap-hibinit swapoff during normal instance use, and right before Hibernation occurs, /etc/acpi/actions/sleep.sh swapon /swap-hibinit, and calls systemctl hibernate:
do_hibernate() {
if [ -d /run/systemd/system ]; then
systemctl hibernate
case "$2" in
SBTN)
swapon /swap-hibinit && do_hibernate
Something changed between 18.04 and 20.04, such that new swapfiles are added with a lower priority than the previous swapfile when they are swapon:
On Focal and later, we see behaviour like if we simply swapon /swap-hibinit generated by ec2-hibinit-agent, we
see it is -2:
$ sudo swapon /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
/swap-hibinit file 3.9G 0B -2
Turning it off:
$ sudo swapoff /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
Lets add /swapfile in:
$ sudo swapon /swapfile
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
Now we enable /swap-hibinit again, and see it is -3:
$ sudo swapon /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
/swap-hibinit file 3.9G 0B -3
Lets add in another swapfile, /swapfile-second, and we see -2, -3, -4:
$ sudo swapon /swapfile-second
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
/swap-hibinit file 3.9G 0B -3
/swapfile-second file 4G 0B -4
What happens is that if we have a swapfile, say, /swapfile at default priority -2, when we go to hibernate, the swapon in /etc/acpi/actions/sleep.sh will set the priority of /swap-hibinit to -3. systemd / the kernel will then select the highest priority swapfile to hibernate to, in this case /swapfile, which is NOT setup for resume= or resume_offset= on the kernel command line, and hibernation will fail.
Apr 11 21:08:15 ip-172-31-84-225 kernel: [ 240.990073] Adding 4095996k swap on /swap-hibinit. Priority:-3 extents:6 across:4644860k SSFS
This leaves the instance in the "Stopping" state on the EC2 console until it hits the 20 minute timeout, at which point it is force stopped.
The fix is to set the priority when we swapon /swap-hibinit to something higher than any other swapfile, to ensure we hibernate to /swap-hibinit.
[Testcase]
From the EC2 console, select "Launch Instance".
Create a:
- t2.medium
- Ubuntu 20.04, 21.04 or 22.04
- 20gb storage space, advanced > enable encryption > yes.
- Advanced settings > Stop State (Hibernation) Support > Enabled
On boot wait for ec2-hibinit-agent to complete hibinit-agent.service, and see that /swap-hibinit is created, and swapoff.
$ ll /swap-hibinit
Add a swapfile, and switch it on:
$ sudo fallocate -l 4G /swapfile
$ sudo dd if=/dev/zero of=/swapfile bs=1024 count=4194304
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ echo "/swapfile swap swap defaults 0 0" | sudo tee -a /etc/fstab
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
Go back to EC2 console, "Instance State" > "Hibernate".
You will see this in journalctl:
Mar 15 11:41:54 ip-172-31-27-108 kernel: [ 520.121761] Adding 16095656k swap on /swap-hibinit. Priority:-3 extents:13 across:17611176k SSFS
Mar 15 11:41:54 ip-172-31-27-108 root: ACPI action undefined: LNXSLPBN:00
and the instance will not hibernate. EC2 console will report "Stopping" for 20 minutes until it times out and is force stopped.
If you enable the following ppa and install the test ec2-hibinit-agent package:
https://launchpad.net/~mruffell/+archive/ubuntu/sf331069-test
Hibernation should succeed within a minute or two.
[Where problems could occur]
This change will only affect users of instances where Hibernation has been explicitly enabled, either from the EC2 instance launch advanced settings, or via the "--hibernation-options Configured=true" parameter to the "aws ec2" command. For all other users, including those with swapfiles enabled, this change will have no effect.
We are changing the /swap-hibinit file to be maximum priority right before we hibernate, to ensure it is the swapfile selected to hibernate to. Since we swapoff /swap-hibinit as soon as we resume, /swap-hibinit is used solely for hibernation, and not for regular swap space, so it is unlikely to cause any regressions to users with their own swapfiles configured with various priorities.
A potential risk is users that do not use /swap-hibinit, and use their own swapfile for hibernation, and overwrite the changes ec2-hibinit-agent makes to grub files to set the resume=UUID<uuid> and resume_offset=<offset> values. I believe such users would likely remove or purge the ec2-hibinit-agent package, since hibinit-agent.service runs at startup and re-adds the grub configuration for /swap-hibinit whether you like it or not, and having /swap-hibinit around would waste disk space that you would be paying for. Because of this, I believe that this change will not break users who hibernate to their own swapfiles, because they would have removed ec2-hibinit-agent on instance creation.
[Other info]
Chris Newcomer came across the above upstream bug, which seems to be the same issue:
https://github.com/aws/amazon-ec2-hibinit-agent/issues/20
The reporter, Ben Mares, suggests a patch to /etc/acpi/actions/sleep.sh to either read the value of a bash environment variable swap_priority, or default to 10.
https://github.com/aws/amazon-ec2-hibinit-agent/pull/21
I'm not exactly on board with the environment variable, or the default magic number of 10, as we don't know how our users are setting up swapfiles, and what priorities they set them to. I think we should instead just set the priority to the maximum, 32767 instead. |
[Impact]
It is not uncommon for users to add a swapfile to their AWS instance, in case they run short of memory. For users that optionally enable Hibernation support, the swapfile generated by ec2-hibinit-agent, /swap-hibinit, needs to always be the highest priority when it comes to suspend the system, since ec2-hibinit-agent sets up /swap-hibinit to be the correct way to suspend and resume via the resume=UUID=<uuid> and resume_offeset=<offset> kernel command line parameters.
ec2-hibinit-agent keeps /swap-hibinit swapoff during normal instance use, and right before Hibernation occurs, /etc/acpi/actions/sleep.sh swapon /swap-hibinit, and calls systemctl hibernate:
do_hibernate() {
if [ -d /run/systemd/system ]; then
systemctl hibernate
case "$2" in
SBTN)
swapon /swap-hibinit && do_hibernate
Something changed between 18.04 and 20.04, such that new swapfiles are added with a lower priority than the previous swapfile when they are swapon:
On Focal and later, we see behaviour like if we simply swapon /swap-hibinit generated by ec2-hibinit-agent, we
see it is -2:
$ sudo swapon /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
/swap-hibinit file 3.9G 0B -2
Turning it off:
$ sudo swapoff /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
Lets add /swapfile in:
$ sudo swapon /swapfile
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
Now we enable /swap-hibinit again, and see it is -3:
$ sudo swapon /swap-hibinit
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
/swap-hibinit file 3.9G 0B -3
Lets add in another swapfile, /swapfile-second, and we see -2, -3, -4:
$ sudo swapon /swapfile-second
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
/swap-hibinit file 3.9G 0B -3
/swapfile-second file 4G 0B -4
What happens is that if we have a swapfile, say, /swapfile at default priority -2, when we go to hibernate, the swapon in /etc/acpi/actions/sleep.sh will set the priority of /swap-hibinit to -3. systemd / the kernel will then select the highest priority swapfile to hibernate to, in this case /swapfile, which is NOT setup for resume= or resume_offset= on the kernel command line, and hibernation will fail.
Apr 11 21:08:15 ip-172-31-84-225 kernel: [ 240.990073] Adding 4095996k swap on /swap-hibinit. Priority:-3 extents:6 across:4644860k SSFS
This leaves the instance in the "Stopping" state on the EC2 console until it hits the 20 minute timeout, at which point it is force stopped.
The fix is to set the priority when we swapon /swap-hibinit to something higher than any other swapfile, to ensure we hibernate to /swap-hibinit.
[Testcase]
From the EC2 console, select "Launch Instance".
Create a:
- t2.medium
- Ubuntu 20.04, 21.04 or 22.04
- 20gb storage space, advanced > enable encryption > yes.
- Advanced settings > Stop State (Hibernation) Support > Enabled
On boot wait for ec2-hibinit-agent to complete hibinit-agent.service, and see that /swap-hibinit is created, and swapoff.
$ ll /swap-hibinit
Add a swapfile, and switch it on:
$ sudo fallocate -l 4G /swapfile
$ sudo dd if=/dev/zero of=/swapfile bs=1024 count=4194304
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ echo "/swapfile swap swap defaults 0 0" | sudo tee -a /etc/fstab
$ swapon --show
NAME TYPE SIZE USED PRIO
/swapfile file 4G 0B -2
Go back to EC2 console, "Instance State" > "Hibernate".
You will see this in journalctl:
Mar 15 11:41:54 ip-172-31-27-108 kernel: [ 520.121761] Adding 16095656k swap on /swap-hibinit. Priority:-3 extents:13 across:17611176k SSFS
Mar 15 11:41:54 ip-172-31-27-108 root: ACPI action undefined: LNXSLPBN:00
and the instance will not hibernate. EC2 console will report "Stopping" for 20 minutes until it times out and is force stopped.
If you enable the following ppa and install the test ec2-hibinit-agent package:
https://launchpad.net/~mruffell/+archive/ubuntu/sf331069-test
Hibernation should succeed within a minute or two.
[Where problems could occur]
This change will only affect users of instances where Hibernation has been explicitly enabled, either from the EC2 instance launch advanced settings, or via the "--hibernation-options Configured=true" parameter to the "aws ec2" command. For all other users, including those with swapfiles enabled, this change will have no effect.
We are changing the /swap-hibinit file to be maximum priority right before we hibernate, to ensure it is the swapfile selected to hibernate to. Since we swapoff /swap-hibinit as soon as we resume, /swap-hibinit is used solely for hibernation, and not for regular swap space, so it is unlikely to cause any regressions to users with their own swapfiles configured with various priorities.
A potential risk is users that do not use /swap-hibinit, and use their own swapfile for hibernation, and overwrite the changes ec2-hibinit-agent makes to grub files to set the resume=UUID<uuid> and resume_offset=<offset> values. I believe such users would likely remove or purge the ec2-hibinit-agent package, since hibinit-agent.service runs at startup and re-adds the grub configuration for /swap-hibinit whether you like it or not, and having /swap-hibinit around would waste disk space that you would be paying for. Because of this, I believe that this change will not break users who hibernate to their own swapfiles, because they would have removed ec2-hibinit-agent on instance creation.
[Other info]
Chris Newcomer came across the above upstream bug, which seems to be the same issue:
https://github.com/aws/amazon-ec2-hibinit-agent/issues/20
The reporter, Ben Mares, suggests a patch to /etc/acpi/actions/sleep.sh to either read the value of a bash environment variable swap_priority, or default to 10.
https://github.com/aws/amazon-ec2-hibinit-agent/pull/21
I'm not exactly on board with the environment variable, or the default magic number of 10, as we don't know how our users are setting up swapfiles, and what priorities they set them to. I think we should instead just set the priority to the maximum, 32767 instead.
I opened a pull request, which has been reviewed by the AWS EC2 team:
https://github.com/aws/amazon-ec2-hibinit-agent/pull/22
The fix was merged upstream with:
commit a2303d269610a6e7415c5045766da605eaa7e30f
From: Matthew Ruffell <matthew.ruffell@canonical.com>
Date: Wed, 20 Apr 2022 15:59:25 +1200
Subject: Swapon with maximum priority before hibernation
Link: https://github.com/aws/amazon-ec2-hibinit-agent/commit/a2303d269610a6e7415c5045766da605eaa7e30f |
|
| 2022-05-11 03:38:46 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Bionic |
|
| 2022-05-11 03:38:46 |
Matthew Ruffell |
bug task added |
|
ec2-hibinit-agent (Ubuntu Bionic) |
|
| 2022-05-11 03:38:46 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Kinetic |
|
| 2022-05-11 03:38:46 |
Matthew Ruffell |
bug task added |
|
ec2-hibinit-agent (Ubuntu Kinetic) |
|
| 2022-05-11 03:38:54 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Bionic): status |
New |
In Progress |
|
| 2022-05-11 03:38:56 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2022-05-11 03:38:58 |
Matthew Ruffell |
ec2-hibinit-agent (Ubuntu Bionic): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2022-05-11 03:39:12 |
Matthew Ruffell |
attachment removed |
debdiff for ec2-hibinit-agent for impish https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5580293/+files/lp1968805_impish.debdiff |
|
|
| 2022-05-11 03:39:22 |
Matthew Ruffell |
attachment removed |
debdiff for ec2-hibinit-agent for jammy https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5580294/+files/lp1968805_jammy.debdiff |
|
|
| 2022-05-11 03:43:57 |
Matthew Ruffell |
attachment removed |
debdiff for ec2-hibinit-agent for bionic https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5581947/+files/lp1968805_bionic.debdiff |
|
|
| 2022-05-11 03:44:06 |
Matthew Ruffell |
attachment removed |
debdiff for ec2-hibinit-agent for focal https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5580292/+files/lp1968805_focal.debdiff |
|
|
| 2022-05-11 05:27:04 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for kinetic https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5588354/+files/lp1968805_kinetic_v2.debdiff |
|
| 2022-05-11 05:27:46 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for jammy https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5588355/+files/lp1968805_jammy_v2.debdiff |
|
| 2022-05-11 05:28:20 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for impish https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5588356/+files/lp1968805_impish_v2.debdiff |
|
| 2022-05-11 05:28:51 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for focal https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5588357/+files/lp1968805_focal_v2.debdiff |
|
| 2022-05-11 05:29:14 |
Matthew Ruffell |
attachment added |
|
debdiff for ec2-hibinit-agent for bionic https://bugs.launchpad.net/ubuntu/+source/ec2-hibinit-agent/+bug/1968805/+attachment/5588358/+files/lp1968805_bionic_v2.debdiff |
|
| 2022-05-18 13:42:58 |
Heitor Alves de Siqueira |
bug |
|
|
added subscriber Heitor Alves de Siqueira |
| 2022-05-18 13:43:07 |
Heitor Alves de Siqueira |
tags |
patch sts sts-sponsor |
patch sts sts-sponsor-halves |
|
| 2022-05-19 17:06:29 |
Launchpad Janitor |
ec2-hibinit-agent (Ubuntu Kinetic): status |
In Progress |
Fix Released |
|
| 2022-05-25 11:32:48 |
Robie Basak |
ec2-hibinit-agent (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2022-05-25 11:32:50 |
Robie Basak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-05-25 11:32:52 |
Robie Basak |
bug |
|
|
added subscriber SRU Verification |
| 2022-05-25 11:32:54 |
Robie Basak |
tags |
patch sts sts-sponsor-halves |
patch sts sts-sponsor-halves verification-needed verification-needed-jammy |
|
| 2022-05-25 11:33:05 |
Robie Basak |
ec2-hibinit-agent (Ubuntu Impish): status |
In Progress |
Fix Committed |
|
| 2022-05-25 11:33:09 |
Robie Basak |
tags |
patch sts sts-sponsor-halves verification-needed verification-needed-jammy |
patch sts sts-sponsor-halves verification-needed verification-needed-impish verification-needed-jammy |
|
| 2022-05-25 11:33:20 |
Robie Basak |
ec2-hibinit-agent (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2022-05-25 11:33:25 |
Robie Basak |
tags |
patch sts sts-sponsor-halves verification-needed verification-needed-impish verification-needed-jammy |
patch sts sts-sponsor-halves verification-needed verification-needed-focal verification-needed-impish verification-needed-jammy |
|
| 2022-05-25 11:33:36 |
Robie Basak |
ec2-hibinit-agent (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2022-05-25 11:33:40 |
Robie Basak |
tags |
patch sts sts-sponsor-halves verification-needed verification-needed-focal verification-needed-impish verification-needed-jammy |
patch sts sts-sponsor-halves verification-needed verification-needed-bionic verification-needed-focal verification-needed-impish verification-needed-jammy |
|
| 2022-05-25 11:34:04 |
Robie Basak |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2022-05-26 04:29:19 |
Matthew Ruffell |
tags |
patch sts sts-sponsor-halves verification-needed verification-needed-bionic verification-needed-focal verification-needed-impish verification-needed-jammy |
patch sts sts-sponsor-halves verification-done-focal verification-needed verification-needed-bionic verification-needed-impish verification-needed-jammy |
|
| 2022-05-26 05:15:26 |
Matthew Ruffell |
tags |
patch sts sts-sponsor-halves verification-done-focal verification-needed verification-needed-bionic verification-needed-impish verification-needed-jammy |
patch sts sts-sponsor-halves verification-done-focal verification-done-impish verification-needed verification-needed-bionic verification-needed-jammy |
|
| 2022-05-26 06:00:41 |
Matthew Ruffell |
tags |
patch sts sts-sponsor-halves verification-done-focal verification-done-impish verification-needed verification-needed-bionic verification-needed-jammy |
patch sts sts-sponsor-halves verification-done-focal verification-done-impish verification-done-jammy verification-needed verification-needed-bionic |
|
| 2022-06-03 06:07:59 |
Matthew Ruffell |
tags |
patch sts sts-sponsor-halves verification-done-focal verification-done-impish verification-done-jammy verification-needed verification-needed-bionic |
patch sts sts-sponsor-halves verification-done-bionic verification-done-focal verification-done-impish verification-done-jammy |
|
| 2022-06-06 14:15:10 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2022-06-06 14:19:11 |
Launchpad Janitor |
ec2-hibinit-agent (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2022-06-06 14:22:06 |
Launchpad Janitor |
ec2-hibinit-agent (Ubuntu Impish): status |
Fix Committed |
Fix Released |
|
| 2022-06-06 14:37:21 |
Launchpad Janitor |
ec2-hibinit-agent (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2022-06-06 14:59:16 |
Launchpad Janitor |
ec2-hibinit-agent (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
