| 2016-04-07 16:59:39 |
Stéphane Graber |
bug |
|
|
added bug |
| 2017-04-20 09:15:11 |
Colin Ian King |
zfs-linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2017-04-20 09:15:13 |
Colin Ian King |
zfs-linux (Ubuntu): assignee |
|
Colin Ian King (colin-king) |
|
| 2017-04-20 09:15:15 |
Colin Ian King |
zfs-linux (Ubuntu): status |
New |
In Progress |
|
| 2017-05-31 14:10:58 |
Colin Ian King |
zfs-linux (Ubuntu): status |
In Progress |
Incomplete |
|
| 2017-07-05 14:02:17 |
Colin Ian King |
attachment added |
|
lxd-zfs-btrfs-comparisons.ods https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1567557/+attachment/4909746/+files/lxd-zfs-btrfs-comparisons.ods |
|
| 2017-07-05 14:05:33 |
Colin Ian King |
bug task added |
|
lxd (Ubuntu) |
|
| 2017-07-05 14:05:43 |
Colin Ian King |
zfs-linux (Ubuntu): status |
Incomplete |
In Progress |
|
| 2017-07-05 14:23:58 |
Colin Ian King |
attachment added |
|
output from health-check when examining lxd https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1567557/+attachment/4909747/+files/health-check-lxd-with-zfs.log |
|
| 2017-07-05 14:40:42 |
Colin Ian King |
lxd (Ubuntu): importance |
Undecided |
High |
|
| 2017-07-05 14:40:44 |
Colin Ian King |
zfs-linux (Ubuntu): importance |
Medium |
Low |
|
| 2017-07-05 14:40:49 |
Colin Ian King |
lxd (Ubuntu): status |
New |
Confirmed |
|
| 2017-07-05 14:40:57 |
Colin Ian King |
zfs-linux (Ubuntu): status |
In Progress |
Confirmed |
|
| 2017-07-05 14:40:59 |
Colin Ian King |
zfs-linux (Ubuntu): status |
Confirmed |
Triaged |
|
| 2017-07-19 15:12:39 |
Colin Ian King |
bug watch added |
|
https://github.com/zfsonlinux/zfs/issues/6372 |
|
| 2017-07-19 15:12:39 |
Colin Ian King |
bug task added |
|
zfs |
|
| 2017-07-19 15:25:47 |
Colin Ian King |
attachment added |
|
Spreadsheet of clone times based on number of clones https://bugs.launchpad.net/zfs/+bug/1567557/+attachment/4917705/+files/zfs-clone-performance.ods |
|
| 2017-07-19 16:24:35 |
Bug Watch Updater |
zfs: status |
Unknown |
New |
|
| 2017-07-20 16:43:31 |
Colin Ian King |
summary |
Performance degradation of "zfs clone" when under load |
Performance degradation of "zfs clone" |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
nominated for series |
|
Ubuntu Artful |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
bug task added |
|
zfs-linux (Ubuntu Artful) |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
bug task added |
|
lxd (Ubuntu Artful) |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
bug task added |
|
zfs-linux (Ubuntu Xenial) |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
bug task added |
|
lxd (Ubuntu Xenial) |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
nominated for series |
|
Ubuntu Zesty |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
bug task added |
|
zfs-linux (Ubuntu Zesty) |
|
| 2017-08-08 16:27:06 |
Colin Ian King |
bug task added |
|
lxd (Ubuntu Zesty) |
|
| 2017-08-08 17:36:00 |
Colin Ian King |
attachment added |
|
Updated benchmarks comparing old and new performance stats for large numbers of clones https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1567557/+attachment/4928925/+files/zfs-clone-performance.ods |
|
| 2017-08-08 19:35:40 |
Launchpad Janitor |
zfs-linux (Ubuntu Artful): status |
Triaged |
Fix Released |
|
| 2017-08-10 12:43:28 |
Colin Ian King |
zfs-linux (Ubuntu Xenial): assignee |
|
Colin Ian King (colin-king) |
|
| 2017-08-10 12:43:30 |
Colin Ian King |
zfs-linux (Ubuntu Zesty): assignee |
|
Colin Ian King (colin-king) |
|
| 2017-08-10 12:43:36 |
Colin Ian King |
zfs-linux (Ubuntu Zesty): status |
New |
In Progress |
|
| 2017-08-10 12:43:39 |
Colin Ian King |
zfs-linux (Ubuntu Xenial): status |
New |
In Progress |
|
| 2017-08-10 12:43:41 |
Colin Ian King |
zfs-linux (Ubuntu Xenial): importance |
Undecided |
Low |
|
| 2017-08-10 12:43:43 |
Colin Ian King |
zfs-linux (Ubuntu Zesty): importance |
Undecided |
Low |
|
| 2017-08-10 17:12:22 |
Colin Ian King |
description |
I've been running some scale tests for LXD and what I've noticed is that "zfs clone" gets slower and slower as the zfs filesystem is getting busier.
It feels like "zfs clone" requires some kind of pool-wide lock or something and so needs for all operations to complete before it can clone a new filesystem.
A basic LXD scale test with btrfs vs zfs shows what I mean, see below for the reports.
The test is run on a completely dedicated physical server with the pool on a dedicated SSD, the exact same machine and SSD was used for the btrfs test.
The zfs filesystem is configured with those settings:
- relatime=on
- sync=disabled
- xattr=sa
So it shouldn't be related to pending sync() calls...
The workload in this case is ultimately 1024 containers running busybox as their init system and udhcpc grabbing an IP.
The problem gets significantly worse if spawning busier containers, say a full Ubuntu system.
=== zfs ===
root@edfu:~# /home/ubuntu/lxd-benchmark spawn --count=1024 --image=images:alpine/edge/amd64 --privileged=true
Test environment:
Server backend: lxd
Server version: 2.0.0.rc8
Kernel: Linux
Kernel architecture: x86_64
Kernel version: 4.4.0-16-generic
Storage backend: zfs
Storage version: 5
Container backend: lxc
Container version: 2.0.0.rc15
Test variables:
Container count: 1024
Container mode: privileged
Image: images:alpine/edge/amd64
Batches: 128
Batch size: 8
Remainder: 0
[Apr 3 06:42:51.170] Importing image into local store: 64192037277800298d8c19473c055868e0288b039349b1c6579971fe99fdbac7
[Apr 3 06:42:52.657] Starting the test
[Apr 3 06:42:53.994] Started 8 containers in 1.336s
[Apr 3 06:42:55.521] Started 16 containers in 2.864s
[Apr 3 06:42:58.632] Started 32 containers in 5.975s
[Apr 3 06:43:05.399] Started 64 containers in 12.742s
[Apr 3 06:43:20.343] Started 128 containers in 27.686s
[Apr 3 06:43:57.269] Started 256 containers in 64.612s
[Apr 3 06:46:09.112] Started 512 containers in 196.455s
[Apr 3 06:58:19.309] Started 1024 containers in 926.652s
[Apr 3 06:58:19.309] Test completed in 926.652s
=== btrfs ===
Test environment:
Server backend: lxd
Server version: 2.0.0.rc8
Kernel: Linux
Kernel architecture: x86_64
Kernel version: 4.4.0-16-generic
Storage backend: btrfs
Storage version: 4.4
Container backend: lxc
Container version: 2.0.0.rc15
Test variables:
Container count: 1024
Container mode: privileged
Image: images:alpine/edge/amd64
Batches: 128
Batch size: 8
Remainder: 0
[Apr 3 07:42:12.053] Importing image into local store: 64192037277800298d8c19473c055868e0288b039349b1c6579971fe99fdbac7
[Apr 3 07:42:13.351] Starting the test
[Apr 3 07:42:14.793] Started 8 containers in 1.442s
[Apr 3 07:42:16.495] Started 16 containers in 3.144s
[Apr 3 07:42:19.881] Started 32 containers in 6.530s
[Apr 3 07:42:26.798] Started 64 containers in 13.447s
[Apr 3 07:42:42.048] Started 128 containers in 28.697s
[Apr 3 07:43:13.210] Started 256 containers in 59.859s
[Apr 3 07:44:26.238] Started 512 containers in 132.887s
[Apr 3 07:47:30.708] Started 1024 containers in 317.357s
[Apr 3 07:47:30.708] Test completed in 317.357s |
[SRU Justification]
Creating tens of hundreds of clones can be prohibitively slow. The underlying mechanism to gather clone information is using a 16K buffer which limits performance. Also, the initial assumption is to pass in zero sized buffer to the underlying ioctl() to get an idea of the size of the buffer required to fetch information back to userspace. If we bump the initial buffer to a larger size then we reduce the need for two ioctl calls which improves performance.
[Fix]
Bump initial buffer size from 16K to 256K
[Regression Potential]
This is minimal as this is just a tweak in the initial buffer size and larger sizes are handled correctly by ZFS since they are normally used on the second ioctl() call once we have established the size of the buffer required from the first ioctl() call. Larger initial buffers just remove the need for the initial size estimation for most cases where the number of clones is less than ~5000. There is a risk that a larger buffer size could lead to a ENOMEM issue when allocating the buffer, but the size of buffer used is still trivial for modern large 64 bit servers running ZFS.
[Test case]
Create 4000 clones. With the fix this takes 35-40% less time than without the fix. See the example test.sh script as an example of how to create this many clones.
------------------------------------------
I've been running some scale tests for LXD and what I've noticed is that "zfs clone" gets slower and slower as the zfs filesystem is getting busier.
It feels like "zfs clone" requires some kind of pool-wide lock or something and so needs for all operations to complete before it can clone a new filesystem.
A basic LXD scale test with btrfs vs zfs shows what I mean, see below for the reports.
The test is run on a completely dedicated physical server with the pool on a dedicated SSD, the exact same machine and SSD was used for the btrfs test.
The zfs filesystem is configured with those settings:
- relatime=on
- sync=disabled
- xattr=sa
So it shouldn't be related to pending sync() calls...
The workload in this case is ultimately 1024 containers running busybox as their init system and udhcpc grabbing an IP.
The problem gets significantly worse if spawning busier containers, say a full Ubuntu system.
=== zfs ===
root@edfu:~# /home/ubuntu/lxd-benchmark spawn --count=1024 --image=images:alpine/edge/amd64 --privileged=true
Test environment:
Server backend: lxd
Server version: 2.0.0.rc8
Kernel: Linux
Kernel architecture: x86_64
Kernel version: 4.4.0-16-generic
Storage backend: zfs
Storage version: 5
Container backend: lxc
Container version: 2.0.0.rc15
Test variables:
Container count: 1024
Container mode: privileged
Image: images:alpine/edge/amd64
Batches: 128
Batch size: 8
Remainder: 0
[Apr 3 06:42:51.170] Importing image into local store: 64192037277800298d8c19473c055868e0288b039349b1c6579971fe99fdbac7
[Apr 3 06:42:52.657] Starting the test
[Apr 3 06:42:53.994] Started 8 containers in 1.336s
[Apr 3 06:42:55.521] Started 16 containers in 2.864s
[Apr 3 06:42:58.632] Started 32 containers in 5.975s
[Apr 3 06:43:05.399] Started 64 containers in 12.742s
[Apr 3 06:43:20.343] Started 128 containers in 27.686s
[Apr 3 06:43:57.269] Started 256 containers in 64.612s
[Apr 3 06:46:09.112] Started 512 containers in 196.455s
[Apr 3 06:58:19.309] Started 1024 containers in 926.652s
[Apr 3 06:58:19.309] Test completed in 926.652s
=== btrfs ===
Test environment:
Server backend: lxd
Server version: 2.0.0.rc8
Kernel: Linux
Kernel architecture: x86_64
Kernel version: 4.4.0-16-generic
Storage backend: btrfs
Storage version: 4.4
Container backend: lxc
Container version: 2.0.0.rc15
Test variables:
Container count: 1024
Container mode: privileged
Image: images:alpine/edge/amd64
Batches: 128
Batch size: 8
Remainder: 0
[Apr 3 07:42:12.053] Importing image into local store: 64192037277800298d8c19473c055868e0288b039349b1c6579971fe99fdbac7
[Apr 3 07:42:13.351] Starting the test
[Apr 3 07:42:14.793] Started 8 containers in 1.442s
[Apr 3 07:42:16.495] Started 16 containers in 3.144s
[Apr 3 07:42:19.881] Started 32 containers in 6.530s
[Apr 3 07:42:26.798] Started 64 containers in 13.447s
[Apr 3 07:42:42.048] Started 128 containers in 28.697s
[Apr 3 07:43:13.210] Started 256 containers in 59.859s
[Apr 3 07:44:26.238] Started 512 containers in 132.887s
[Apr 3 07:47:30.708] Started 1024 containers in 317.357s
[Apr 3 07:47:30.708] Test completed in 317.357s |
|
| 2017-08-10 17:54:15 |
Brian Murray |
zfs-linux (Ubuntu Zesty): status |
In Progress |
Fix Committed |
|
| 2017-08-10 17:54:17 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-08-10 17:54:21 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2017-08-17 23:07:04 |
Brian Murray |
zfs-linux (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2017-08-18 14:21:08 |
Colin Ian King |
tags |
|
verification-done-xenial |
|
| 2017-08-22 16:20:30 |
Stéphane Graber |
bug task deleted |
lxd (Ubuntu) |
|
|
| 2017-08-22 19:53:06 |
Stéphane Graber |
bug task deleted |
lxd (Ubuntu Xenial) |
|
|
| 2017-08-22 19:53:08 |
Stéphane Graber |
bug task deleted |
lxd (Ubuntu Zesty) |
|
|
| 2017-08-22 19:53:11 |
Stéphane Graber |
bug task deleted |
lxd (Ubuntu Artful) |
|
|
| 2017-09-04 15:47:15 |
Łukasz Zemczak |
tags |
verification-done-xenial |
verification-done-xenial verification-done-zesty |
|
| 2017-09-04 15:47:45 |
Launchpad Janitor |
zfs-linux (Ubuntu Zesty): status |
Fix Committed |
Fix Released |
|
| 2017-09-04 15:47:51 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-09-04 15:49:37 |
Launchpad Janitor |
zfs-linux (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2019-09-15 09:16:34 |
Bug Watch Updater |
zfs: status |
New |
Fix Released |
|
