zram-config fails to start/install on a system with more then 30 processors

[Marcus Pereira]

I have an dual processor "Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz".

Its an octo core with hyper-thread so the number of virtual processors are 32:
# grep -c ^processor /proc/cpuinfo
32

zram fails to install and start. from dmesg:

[20817.608907] zram: Creating 32 devices ...
[20817.621309] Adding 514360k swap on /dev/zram0. Priority:5 extents:1 across:514360k SS
[20817.624251] Adding 514360k swap on /dev/zram1. Priority:5 extents:1 across:514360k SS
[20817.627307] Adding 514360k swap on /dev/zram2. Priority:5 extents:1 across:514360k SS
[20817.630403] Adding 514360k swap on /dev/zram3. Priority:5 extents:1 across:514360k SS
[20817.633359] Adding 514360k swap on /dev/zram4. Priority:5 extents:1 across:514360k SS
[20817.636437] Adding 514360k swap on /dev/zram5. Priority:5 extents:1 across:514360k SS
[20817.639563] Adding 514360k swap on /dev/zram6. Priority:5 extents:1 across:514360k SS
[20817.642648] Adding 514360k swap on /dev/zram7. Priority:5 extents:1 across:514360k SS
[20817.645693] Adding 514360k swap on /dev/zram8. Priority:5 extents:1 across:514360k SS
[20817.648791] Adding 514360k swap on /dev/zram9. Priority:5 extents:1 across:514360k SS
[20817.651829] Adding 514360k swap on /dev/zram10. Priority:5 extents:1 across:514360k SS
[20817.654847] Adding 514360k swap on /dev/zram11. Priority:5 extents:1 across:514360k SS
[20817.657859] Adding 514360k swap on /dev/zram12. Priority:5 extents:1 across:514360k SS
[20817.660907] Adding 514360k swap on /dev/zram13. Priority:5 extents:1 across:514360k SS
[20817.663990] Adding 514360k swap on /dev/zram14. Priority:5 extents:1 across:514360k SS
[20817.666961] Adding 514360k swap on /dev/zram15. Priority:5 extents:1 across:514360k SS
[20817.669740] Adding 514360k swap on /dev/zram16. Priority:5 extents:1 across:514360k SS
[20817.672784] Adding 514360k swap on /dev/zram17. Priority:5 extents:1 across:514360k SS
[20817.675825] Adding 514360k swap on /dev/zram18. Priority:5 extents:1 across:514360k SS
[20817.678807] Adding 514360k swap on /dev/zram19. Priority:5 extents:1 across:514360k SS
[20817.681866] Adding 514360k swap on /dev/zram20. Priority:5 extents:1 across:514360k SS
[20817.684797] Adding 514360k swap on /dev/zram21. Priority:5 extents:1 across:514360k SS
[20817.687597] Adding 514360k swap on /dev/zram22. Priority:5 extents:1 across:514360k SS
[20817.690546] Adding 514360k swap on /dev/zram23. Priority:5 extents:1 across:514360k SS
[20817.693530] Adding 514360k swap on /dev/zram24. Priority:5 extents:1 across:514360k SS
[20817.696615] Adding 514360k swap on /dev/zram25. Priority:5 extents:1 across:514360k SS
[20817.699563] Adding 514360k swap on /dev/zram26. Priority:5 extents:1 across:514360k SS
[20817.702580] Adding 514360k swap on /dev/zram27. Priority:5 extents:1 across:514360k SS
[20817.705844] init: zram-config pre-start process (7513) terminated with status 255

From my tests I see that the maximum number of swaps linux accepts to create is 29. I already had 1 swap partition so zram-config fails to start trying to create the 28th extent.

I made a patch to zram-config.conf that, before starting zram-config, counts how many swap spaces are already configured and reduce them from the number of extents to create:

patch:
# diff -u zram-config.conf.dpkg-dist zram-config.conf
--- zram-config.conf.dpkg-dist 2012-01-09 11:29:00.000000000 -0200
+++ zram-config.conf 2012-11-13 02:10:30.122396348 -0200
@@ -6,6 +6,8 @@
 pre-start script
   # load dependency modules
   NRDEVICES=$(grep -c ^processor /proc/cpuinfo | sed 's/^0$/1/')
+ NRSWAPS=$(($(cat /proc/swaps | wc -l)+$NRDEVICES))
+ if [ $NRSWAPS>=30 ]; then NRDEVICES=$((30+$NRDEVICES-$NRSWAPS)); fi
   if modinfo zram | grep -q ' zram_num_devices:' 2>/dev/null; then
     MODPROBE_ARGS="zram_num_devices=${NRDEVICES}"
   elif modinfo zram | grep -q ' num_devices:' 2>/dev/null; then

I have tested on a 32 core and 4 core machines with 1 swap partition and no error at start.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "zram-config.conf.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Oibaf (oibaf)]

Note that with kernel 3.15, this should be revised. 3.15 indeed support multi-stream functionality:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=beca3ec71fe5490ee9237dc42400f50402baf83e

It is no longer needed to set up many devices, just use 1 device and the use something like:
echo `nproc` > /sys/block/zram0/max_comp_streams

(I still haven't tried it myself if it scales better or worse)

For more info see: http://kernelnewbies.org/Linux_3.15#head-52af9ef123b7c0792b09a1a0222fdc8c21ab5d4c

[Lenin (gagarin)]

I have 40 core machines, but I only get /dev/zram0 .. 25 used.

# lsb_release -d
Description: Ubuntu 18.04.1 LTS

# systemctl --all --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
Ï zram-config.service loaded failed failed Initializes zram swaping

# systemctl status zram-config.service
Ï zram-config.service - Initializes zram swaping
   Loaded: loaded (/lib/systemd/system/zram-config.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-12-19 14:07:29 CET; 3s ago
  Process: 41212 ExecStart=/usr/bin/init-zram-swapping (code=exited, status=255)
 Main PID: 41212 (code=exited, status=255)

Dec 19 14:07:29 host init-zram-swapping[41212]: no label, UUID=49e2e5a0-af6f-46e9-9cff-4df9be04acc5
Dec 19 14:07:29 host init-zram-swapping[41212]: swapon: /dev/zram38: swapon failed: Operation not permitted
Dec 19 14:07:29 host init-zram-swapping[41212]: sh: echo: I/O error
Dec 19 14:07:29 host init-zram-swapping[41212]: mkswap: /dev/zram39: warning: wiping old swap signature.
Dec 19 14:07:29 host init-zram-swapping[41212]: Setting up swapspace version 1, size = 400.8 MiB (420233216 byte
Dec 19 14:07:29 host init-zram-swapping[41212]: no label, UUID=e6b6cd37-8036-4466-a98d-df3e15af5626
Dec 19 14:07:29 host init-zram-swapping[41212]: swapon: /dev/zram39: swapon failed: Operation not permitted
Dec 19 14:07:29 host systemd[1]: zram-config.service: Main process exited, code=exited, status=255/n/a
Dec 19 14:07:29 host systemd[1]: zram-config.service: Failed with result 'exit-code'.
Dec 19 14:07:29 host systemd[1]: Failed to start Initializes zram swaping.

[Chase (chase9)]

It's worth noting the multiple streams Oibaf mentioned are currently the default. On my system, there are the max number of swap devices each with 32 streams. I propose we only have one device and use the default behavior of max_comp_streams = number of CPUs.

It's my first time creating an apt patch, hopefully I've done it correctly.

[Oibaf (oibaf)]

Before using multiple streams it would be nice to test the performance (IIRC I did some test many years ago, and it scaled worse than having multiple devices, but not really sure if my test was actually reliable and eventually if that still applies with a recent kernel).

[Chase (chase9)]

I can test this tonight on my Ubuntu server. Otherwise, I can test now on a Fedora workstation. Do you have any tips for benchmarking swap?

[Oibaf (oibaf)]

Unfortunately I only have a vague idea that I tested multi-stream, but then reverted to multiple devices because of the poor performance.
Probably I just tested one of my scripts I use for managing my own stuff, something that used a lot of CPU and RAM.
Don't remember anything else, sorry. :(

[Chase (chase9)]

Running benchmarks now. I'll upload as they get done. Here's the baseline (stock Ubuntu Server install):

https://openbenchmarking.org/result/2009148-FI-SWAP7694110

[Chase (chase9)]

Here is a benchmark with the default zram-config

https://openbenchmarking.org/result/2009144-FI-SWAP6511210

[Chase (chase9)]

Here it is with the customized zram-config
https://openbenchmarking.org/result/2009143-FI-SWAP4713310

[Chase (chase9)]

Be sure to click the last link, as it will show you all three tests side-by-side.

Overall, I would say that the custom zram-config doesn't much matter to the performance.

[Oibaf (oibaf)]

How were you able to enable "default zram-config" if it has this bug?
What do you mean exactly with "custom zram-config"? Just adding your 99_fixcores.patch?

[Chase (chase9)]

This was tested in a VM running Ubuntu Server 20.04 with fewer than 29 cores dedicated to it, so I didn't run into the bug. Unfortunately the production server wasn't available for testing (especially since each benchmark took over an hour!).

Yes, custom zram-config means the sram-config but with the patch applied.

[Steve Langasek (vorlon)]

ubuntu-release has been subscribed to this bug, but I don't see why. This is not a well-formed feature freeze exception request, and it looks like the issue here is a bug, not a new feature.

[Oibaf (oibaf)]

@chase9 : thanks for testing!
Would you like to also test performance with different compression algorithms? See bug 1856043.

[Oibaf (oibaf)]

@chase9 : your tests compared default zram-config vs. your customized zram-config.
However, as you noticed, default zram-config already enabled multiple streams for every device. Probably this is by accident, since multiple streams were added by default in the kernel after zram-config package already enabled multiple devices, and no one ever noticed.
So it would also be useful to test the multiple devices *without multiple streams* vs. single device *with multiple streams*.
Thanks.

[Oibaf (oibaf)]

I updated zram-config package with chase9 patch and uploaded it to my test PPA:
https://launchpad.net/~oibaf/+archive/ubuntu/test/+packages

[Launchpad Janitor (janitor)]

[Expired for zram-config (Ubuntu) because there has been no activity for 60 days.]

[Oibaf (oibaf)]

It looks like this was finally properly addressed in 0.7, available since Ubuntu 22.04/jammy, see https://launchpad.net/ubuntu/+source/zram-config/+changelog