ec2-hibinit-agent needs to properly initialize swap file

Bug #1892728 reported by Andrea Righi on 2020-08-24
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
High
Seth Forshee
Groovy
High
Seth Forshee

Bug Description

SRU Justification

Impact: Running swapon on ext4 with a file with holes fails in the Groovy 5.8 kernel. This prevents using swap files created using fallocate(), which prevents hibernation in EC2.

Fix: Commit 424de74af0d0679e99660904a5f472acd85bdb73 from linux-next implements swap_activate aops to fix this issue.

Test Case: Run swapon with a file allocated using fallocate() in an ext4 filesystem. With the fix this should be successful, and it should be possible to hibernate/resume an EC2 instance using such a swapfile.

Regression Potential: The code changes only impact activating swap on ext4, so regressions are possible for this operation. All other operations should not be impacted.

---

With commit 30460e1ea3e6 ("fs: Enable bmap() function to properly return errors"), applied to v5.6-rc1, running swapon on a file that has holes returns an error (EINVAL); hibinit-agent is creating the swap file just using fallocate(), without touching the whole file by deafult, so there might be holes, swapon fails and hibernation cannot be performed.

To prevent this problem we need to touch the swap file before using it and this can be done setting "touch-swap = True" in /etc/hibinit-config.cfg.

The downside of this setting is that a new instance requires more time to be properly "initialized" the first time it boots due to the extra I/O required to touch the swap file (this additional I/O can be quite consistent in large instances, like c5.18xlarge for example), but I don't see a better way to make sure the file is all allocated without holes, so I guess we need to turn on this option if we want to use hibernation w/ kernels >= 5.6.

Andrea Righi (arighi) on 2020-08-24
no longer affects: linux-aws (Ubuntu)
tags: added: rls-gg-incoming
Balint Reczey (rbalint) wrote :

Previously created files should also be updated to work with newer kernels.

Changed in ec2-hibinit-agent (Ubuntu):
status: New → Confirmed
Dimitri John Ledkov (xnox) wrote :

Can creation of such swapfile should be done as a systemd unit, which wouldn't block boot per-se and perform that operation in-parallel to the rest of the system booting/starting?

(aka simply systemd service unit)

tags: removed: rls-gg-incoming
Balint Reczey (rbalint) wrote :

@xnox This would be too much divergence from upstream and timeout issues related to this package already caused complications in the past.

tags: added: id-5f47d0f8b8ce963bb1e25973
Francis Ginther (fginther) wrote :

We also need to set "AllowSuspend=no" in /etc/systemd/sleep.conf (or a .d file) as CONFIG_SUSPEND has been disabled in the kernel to improve the robustness of hibernation.

Balint Reczey (rbalint) on 2020-09-22
Changed in ec2-hibinit-agent (Ubuntu Groovy):
importance: Undecided → Critical
importance: Critical → High
Balint Reczey (rbalint) wrote :

@fginther I've opened LP: #1898087 to disable suspend.

Changed in ec2-hibinit-agent (Ubuntu Groovy):
assignee: nobody → Balint Reczey (rbalint)
status: Confirmed → In Progress
Balint Reczey (rbalint) wrote :

How can I observe the problem?
I've installed linux-image-5.6.0-1028-oem and the created swapfile works, and when I create a file with fallocate that also works:

ubuntu@ip-172-31-4-85:~$ fallocate -l 4194304000 swap2
ubuntu@ip-172-31-4-85:~$ du -sh swap2
4.0G swap2
ubuntu@ip-172-31-4-85:~$ sudo mkswap swap2
mkswap: swap2: insecure permissions 0664, 0600 suggested.
mkswap: swap2: insecure file owner 1000, 0 (root) suggested.
Setting up swapspace version 1, size = 3.9 GiB (4194299904 bytes)
no label, UUID=8d0c81ae-ae7e-4d92-a40f-31812468cd84
ubuntu@ip-172-31-4-85:~$ sudo swapon swap2
swapon: /home/ubuntu/swap2: insecure permissions 0664, 0600 suggested.
swapon: /home/ubuntu/swap2: insecure file owner 1000, 0 (root) suggested.
ubuntu@ip-172-31-4-85:~$ echo $?
0

OTOH when I use truncate, the sparse file is created and it indeed fails to set the file up as swap:

ubuntu@ip-172-31-4-85:~$ sudo swapoff swap2
ubuntu@ip-172-31-4-85:~$ truncate -s 0 swap2
ubuntu@ip-172-31-4-85:~$ truncate -s 4G swap2
ubuntu@ip-172-31-4-85:~$ du -sh swap2
0 swap2
ubuntu@ip-172-31-4-85:~$ sudo mkswap swap2
mkswap: swap2: insecure permissions 0664, 0600 suggested.
mkswap: swap2: insecure file owner 1000, 0 (root) suggested.
Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
no label, UUID=02e6ad30-2a03-4cb6-8f02-e64fd98d8bf9
ubuntu@ip-172-31-4-85:~$ du -sh swap2
4.0K swap2
ubuntu@ip-172-31-4-85:~$ sudo swapon swap2
swapon: /home/ubuntu/swap2: insecure permissions 0664, 0600 suggested.
swapon: /home/ubuntu/swap2: insecure file owner 1000, 0 (root) suggested.
swapon: /home/ubuntu/swap2: skipping - it appears to have holes.
ubuntu@ip-172-31-4-85:~$

Changed in ec2-hibinit-agent (Ubuntu Groovy):
status: In Progress → Incomplete
tags: added: fr-205
Balint Reczey (rbalint) wrote :

It seems the kernel needs to be fixed, because fallocate changed in the kernel according to Andrea:
https://lkml.org/lkml/2020/9/4/259

affects: ec2-hibinit-agent (Ubuntu Groovy) → linux (Ubuntu Groovy)
Changed in linux (Ubuntu Groovy):
assignee: Balint Reczey (rbalint) → nobody
Seth Forshee (sforshee) on 2020-10-14
description: updated
Changed in linux (Ubuntu Groovy):
assignee: nobody → Seth Forshee (sforshee)
status: Incomplete → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.8.0-25.26

---------------
linux (5.8.0-25.26) groovy; urgency=medium

  * groovy/linux: 5.8.0-25.26 -proposed tracker (LP: #1899940)

  * CVE-2020-12351
    - Bluetooth: L2CAP: Fix calling sk_filter on non-socket based channel

  * CVE-2020-12352
    - Bluetooth: A2MP: Fix not initializing all members

  * CVE-2020-12351 // CVE-2020-12352 // CVE-2020-24490
    - Bluetooth: Disable High Speed by default
    - Bluetooth: MGMT: Fix not checking if BT_HS is enabled
    - [Config] Disable BlueZ highspeed support

  * ec2-hibinit-agent needs to properly initialize swap file (LP: #1892728)
    - ext4: implement swap_activate aops using iomap

 -- Andrea Righi <email address hidden> Thu, 15 Oct 2020 12:09:24 +0200

Changed in linux (Ubuntu Groovy):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers