When using swap in ZFS, system stops when you start using swap

Bug #1847628 reported by Sung Jeho on 2019-10-10
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Native ZFS for Linux
Unknown
Unknown
Release Notes for Ubuntu
Undecided
Unassigned
ubiquity (Ubuntu)
High
Unassigned
zfs-linux (Ubuntu)
High
Jean-Baptiste Lallement

Bug Description

# Problem

When using swap in ZFS, system stops when you start using swap.

> stress --vm 100

if you doing swapoff will only occur OOM and the system will not stop.

# Environment

jehos@MacBuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Eoan Ermine (development branch)
Release: 19.10
Codename: eoan

jehos@MacBuntu:~$ dpkg -l | grep zfs
ii libzfs2linux 0.8.1-1ubuntu13 amd64 OpenZFS filesystem library for Linux
ii zfs-initramfs 0.8.1-1ubuntu13 amd64 OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 0.8.1-1ubuntu13 amd64 OpenZFS Event Daemon
ii zfsutils-linux 0.8.1-1ubuntu13 amd64 command-line tools to manage OpenZFS filesystems

jehos@MacBuntu:~$ uname -a
Linux MacBuntu 5.3.0-13-generic #14-Ubuntu SMP Tue Sep 24 02:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

jehos@MacBuntu:~$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 66.1M 1.81G - - - 3% 1.00x ONLINE -
rpool 230G 124G 106G - - 9% 53% 1.00x ONLINE -

jehos@MacBuntu:~$ zfs get all rpool/swap
NAME PROPERTY VALUE SOURCE
rpool/swap type volume -
rpool/swap creation 목 10월 10 15:56 2019 -
rpool/swap used 2.13G -
rpool/swap available 98.9G -
rpool/swap referenced 72K -
rpool/swap compressratio 1.11x -
rpool/swap reservation none default
rpool/swap volsize 2G local
rpool/swap volblocksize 4K -
rpool/swap checksum on default
rpool/swap compression zle local
rpool/swap readonly off default
rpool/swap createtxg 34 -
rpool/swap copies 1 default
rpool/swap refreservation 2.13G local
rpool/swap guid 18209330213704683244 -
rpool/swap primarycache metadata local
rpool/swap secondarycache none local
rpool/swap usedbysnapshots 0B -
rpool/swap usedbydataset 72K -
rpool/swap usedbychildren 0B -
rpool/swap usedbyrefreservation 2.13G -
rpool/swap logbias throughput local
rpool/swap objsetid 393 -
rpool/swap dedup off default
rpool/swap mlslabel none default
rpool/swap sync always local
rpool/swap refcompressratio 1.11x -
rpool/swap written 72K -
rpool/swap logicalused 40K -
rpool/swap logicalreferenced 40K -
rpool/swap volmode default default
rpool/swap snapshot_limit none default
rpool/swap snapshot_count none default
rpool/swap snapdev hidden default
rpool/swap context none default
rpool/swap fscontext none default
rpool/swap defcontext none default
rpool/swap rootcontext none default
rpool/swap redundant_metadata all default
rpool/swap encryption off default
rpool/swap keylocation none default
rpool/swap keyformat none default
rpool/swap pbkdf2iters 0 default

Colin Ian King (colin-king) wrote :

A swapfile on ZFS is a bad idea. Swapped out pages get pushed through the vfs into zfs and each page of swap will be magnified in the number of free pages required to get this page out to disk.

Colin Ian King (colin-king) wrote :

https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-use-a-zvol-as-a-swap-device

..there are known issues with swap on ZFS not working well on heavily memory loaded systems.

Sung Jeho (jehos) wrote :

@colin-king

> A swapfile on ZFS is a bad idea. Swapped out pages get pushed through the vfs into zfs and each page of swap will be magnified in the number of free pages required to get this page out to disk.

currently, swap uses zvol in default. not a swapfile. (actually I also thinking swapfile is mistake of bad architecture..)

> ..there are known issues with swap on ZFS not working well on heavily memory loaded systems.

and also, i also see that article, and check it.
It was already optimised.

Sung Jeho (jehos) wrote :

I did the same test for USB flash memory and it works normally.
try create a swap on zvol, it's definitely not working well..

===================================================================

jehos@MacBuntu:~$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 7.46 GiB, 8002732032 bytes, 15630336 sectors
Disk model: Cruzer Blade
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 91BE0613-2C00-8944-BEFF-12CCBF5DE2B8
jehos@MacBuntu:~$ sudo mkswap -f /dev/sdc
Setting up swapspace version 1, size = 7.5 GiB (8002727936 bytes)
no label, UUID=14bb8b5c-ded5-444e-8a62-a21cbc469d08

jehos@MacBuntu:~$ sudo swapon /dev/sdc

jehos@MacBuntu:~$ free -m
              total used free shared buff/cache available
Mem: 7870 3106 3843 257 920 4260
Swap: 7631 0 7631

jehos@MacBuntu:~$ time stress --vm 100 -t 10
stress: info: [6247] dispatching hogs: 0 cpu, 0 io, 100 vm, 0 hdd
stress: info: [6247] successful run completed in 15s

real 0m15.124s
user 0m0.736s
sys 0m3.536s

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in zfs-linux (Ubuntu):
status: New → Confirmed

Confirmed this affects me. 16gig of ram, and a swap ZVOL. When memory gets low the system freezes for several minutes.

Also reported this bug here: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1848102

As I believe ubiquity shouldn't be creating a ZVOL for swap when this is a known issue.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubiquity (Ubuntu):
status: New → Confirmed
Changed in ubiquity (Ubuntu):
importance: Undecided → High
tags: added: rls-ee-incoming
Dimitri John Ledkov (xnox) wrote :

partman-swapfile does not create swapfiles on neither btrfs nor zfs.

when using desktop installer and choosing zfs a zfs zvol is created for swap.

it uses all the settings seen in the guide, but the "com.sun:auto-snapshot=false" do we need to add that or does our zfs not support it?

Or are you saying, that zfs zvol swap shouldn't be created either, since that is known to cause lockups?

Changed in zfs-linux (Ubuntu):
status: Confirmed → Incomplete
Changed in ubiquity (Ubuntu):
status: Confirmed → Incomplete
Richard Laager (rlaager) wrote :

> "com.sun:auto-snapshot=false" do we need to add that or does our zfs not support it?

You do not need that. That is used by some snapshot tools, but Ubuntu is doing its own zsys thing.

> Or are you saying, that zfs zvol swap shouldn't be created either,
> since that is known to cause lockups?

That is correct. As per the thread at the attached github zfsonlinux bug report, zfs zvol swap causes system freezes for minutes at a time, and under enough memory pressure will lock up the system completely.

Ubiquity installer should put swap on it's own partition, not a zvol.

Changed in ubuntu-release-notes:
status: New → Fix Released
Changed in ubiquity (Ubuntu):
status: Incomplete → Triaged
Changed in zfs-linux (Ubuntu):
status: Incomplete → Triaged
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubiquity - 19.10.21

---------------
ubiquity (19.10.21) eoan; urgency=medium

  [ Dimitri John Ledkov ]
  * scripts/plugininstall.py: skip installing extras with multiarch
    dependencies in the first pass.

  [ Mathieu Trudel-Lapierre ]
  * Updated translations from Launchpad.

  [ Adam Conrad ]
  * Rewrite zsys-setup swap handling to use a swap partition (LP: #1847628)

 -- Adam Conrad <email address hidden> Wed, 16 Oct 2019 17:36:10 +0100

Changed in ubiquity (Ubuntu):
status: Triaged → Fix Released
Changed in zfs-linux (Ubuntu):
importance: Undecided → Critical
tags: added: rls-ff-incoming
removed: rls-ee-incoming
Changed in zfs-linux (Ubuntu):
importance: Critical → High

upstream proposal is to force everything through ZIL by creating the swap like this:

zfs create -o sync=always -o primarycache=metadata -o secondarycache=none -o logbias=throughput -b 4K -V 8G rpool/swap

I'll test with our setup on Ubuntu.

Changed in zfs-linux (Ubuntu):
assignee: nobody → Jean-Baptiste Lallement (jibel)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.