Bug #1636847 “unexpectedly large memory usage of mounted snaps” : Bugs : linux package : Ubuntu

Revision history for this message

Brad Figg (brad-figg) wrote on 2016-10-26: Missing required logs.

#1

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1636847

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status:	New → Incomplete

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2016-10-26:

#2

Marking as 'confirmed' so the bot doesn't auto-close it.

Changed in linux (Ubuntu):
status:	Incomplete → Confirmed

Revision history for this message

Andy Whitcroft (apw) wrote on 2016-10-26:

#3

Evidence suggests this is triggered by the parallel decompressor. Will spin a kernel with that switched out:

CONFIG_SQUASHFS_DECOMP_SINGLE=y

Revision history for this message

Andy Whitcroft (apw) wrote on 2016-10-26:

#4

Ok test kernels at the URL below, please report testing results back here:

people.canonical.com/~apw/lp1636847-xenial/

Thanks.

Revision history for this message

Zygmunt Krynicki (zyga) wrote on 2016-10-26:

#5

After using the CONFING_SQUASHFS_DECOMP_SINGLE=y option in an experimental kernel the memory usage dropped significantly. The traces below are for -45 and the experimental -46 kernel. The 1 and 4 are the number of CPUs on the system.

A single CPU system used to consumer 131MB per mounted snap, this is now reduced to just 4MB/snap. A four-CPU system has less dramatic improvement where the numbers are 7MB -> 4MB.

This suggests some kind of bug in the CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU option perhaps

+ echo Ubuntu 16.04
Ubuntu 16.04
+ ./analyze.py ubuntu 16.04 4.4.0-45-generic 1 size-1m.squashfs.xz.heavy
# num-mounted extra-memory delta
0: 142.60MB
1: 274.71MB (delta: 132.11MB)
2: 406.55MB (delta: 131.84MB)
3: 538.36MB (delta: 131.81MB)
4: 670.19MB (delta: 131.82MB)
+ ./analyze.py ubuntu 16.04 4.4.0-46-generic 1 size-1m.squashfs.xz.heavy
# num-mounted extra-memory delta
0: 62.28MB
1: 66.99MB (delta: 4.71MB)
2: 71.05MB (delta: 4.07MB)
3: 75.12MB (delta: 4.06MB)
4: 79.19MB (delta: 4.07MB)
+ ./analyze.py ubuntu 16.04 4.4.0-45-generic 4 size-1m.squashfs.xz.heavy
# num-mounted extra-memory delta
0: 235.43MB
1: 242.38MB (delta: 6.96MB)
2: 249.38MB (delta: 7.00MB)
3: 256.45MB (delta: 7.06MB)
4: 263.42MB (delta: 6.97MB)
+ ./analyze.py ubuntu 16.04 4.4.0-46-generic 4 size-1m.squashfs.xz.heavy
# num-mounted extra-memory delta
0: 72.79MB
1: 75.90MB (delta: 3.11MB)
2: 79.96MB (delta: 4.06MB)
3: 83.93MB (delta: 3.97MB)
4: 88.00MB (delta: 4.07MB)

Andy Whitcroft (apw) on 2016-10-26

Changed in linux (Ubuntu Xenial):
status:	New → Confirmed
Changed in linux (Ubuntu Yakkety):
status:	New → Confirmed
Changed in linux (Ubuntu):
assignee:	nobody → Colin Ian King (colin-king)
importance:	Undecided → Medium
Changed in linux (Ubuntu Xenial):
importance:	Undecided → Critical
Changed in linux (Ubuntu Yakkety):
importance:	Undecided → Critical
Changed in linux (Ubuntu Xenial):
assignee:	nobody → Andy Whitcroft (apw)
Changed in linux (Ubuntu Yakkety):
assignee:	nobody → Andy Whitcroft (apw)

Tim Gardner (timg-tpi) on 2016-10-26

Changed in linux (Ubuntu Xenial):
status:	Confirmed → Fix Committed
Changed in linux (Ubuntu Yakkety):
status:	Confirmed → Fix Committed
Changed in linux (Ubuntu):
status:	Confirmed → Fix Committed

Revision history for this message

Seth Forshee (sforshee) wrote on 2016-11-07:

#6

@zyga: I'm honestly very surprised that the config change had that drastic an impact on the single-CPU system. Can you tell me what 'cat /sys/devices/system/cpu/possible' says on that system?

Can I assume that the "size-1M" implies a 1MB block size?

Let's start with the simplest options for reducing RAM usage. The most obvious place for gains is the buffers squashfs uses for decompression and caching. squashfs has three caches:

- The "metadata" cache caches 8 blocks of metadata. The metadata block size is fixed at 8KB and the cache is fixed at 8 blocks, so this consumes 64KB of RAM (plus overhead). So there isn't a lot to be gained here.

- The "data" cache is the one affected by CONFIG_SQUASHFS_DECOMP_SINGLE. squashfs allocates RAM up front for each possible decompression thread, at fs_block_size bytes per thread. The previously used config option allocated one cache per possible system in the CPU (which is why I was suprised at the numbers for the single CPU system; at a 1MB block size that implies the system supports over 100 CPUs). The only simple way to gain more here would be to reduce the block size of the filesystems.

- The "fragment" cache. This caches CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE blocks, which is currently set to 3 in our kernels. So it would seem that this accounts for the bulk of the remaining RAM usage. That means for a 1MB block size it's a fixed size of 3MB. We could reduce this to 1 to save some RAM, and reducing the block size of the squashfs images would again help here.

So if the images do have a 1MB block size there are two simple things that will yield the biggest gains - reducing the fragment cache to 1 block and reducing the block size of the squashfs images. Obviously any reduction in cache sizes may result in a loss of performance, depending on access patterns.

I'll go ahead and build a kernel with CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=1 for you to test. I would suggest taking a look at performance in addition to RAM usage when you're testing.

If 1MB is the block size you're using, would you be open to making this smaller?

@zyga: I'm honestly very surprised that the config change had that drastic an impact on the single-CPU system. Can you tell me what 'cat /sys/devices/system/cpu/possible' says on that system?

Can I assume that the "size-1M" implies a 1MB block size?

Let's start with the simplest options for reducing RAM usage. The most obvious place for gains is the buffers squashfs uses for decompression and caching. squashfs has three caches:

- The "metadata" cache caches 8 blocks of metadata. The metadata block size is fixed at 8KB and the cache is fixed at 8 blocks, so this consumes 64KB of RAM (plus overhead). So there isn't a lot to be gained here.

- The "data" cache is the one affected by CONFIG_SQUASHFS_DECOMP_SINGLE. squashfs allocates RAM up front for each possible decompression thread, at fs_block_size bytes per thread. The previously used config option allocated one cache per possible system in the CPU (which is why I was suprised at the numbers for the single CPU system; at a 1MB block size that implies the system supports over 100 CPUs). The only simple way to gain more here would be to reduce the block size of the filesystems.

- The "fragment" cache. This caches CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE blocks, which is currently set to 3 in our kernels. So it would seem that this accounts for the bulk of the remaining RAM usage. That means for a 1MB block size it's a fixed size of 3MB. We could reduce this to 1 to save some RAM, and reducing the block size of the squashfs images would again help here.

So if the images do have a 1MB block size there are two simple things that will yield the biggest gains - reducing the fragment cache to 1 block and reducing the block size of the squashfs images. Obviously any reduction in cache sizes may result in a loss of performance, depending on access patterns.

I'll go ahead and build a kernel with CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=1 for you to test. I would suggest taking a look at performance in addition to RAM usage when you're testing.

If 1MB is the block size you're using, would you be open to making this smaller?

Revision history for this message

Seth Forshee (sforshee) wrote on 2016-11-07:

#7

Test build with CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=1 is done.

http://people.canonical.com/~sforshee/lp1636847/linux-4.4.0-46.67+lp1636847v201611071220/

Revision history for this message

Seth Forshee (sforshee) wrote on 2016-11-08:

#8

Adding some notes and ideas based on looking more at the squashfs implementation.

One idea I had was to reduce the data cache (or maybe even eliminate it) by making squashfs decompress directly into the page cache. It turns out that squashfs already does this if CONFIG_SQUASHFS_FILE_DIRECT=y, which is the case in our kernel. Rather it tries to read into the page cache directly first, and if that fails it falls back to using the cache, so we can't eliminate the cache entirely. But it does mean that squashfs can probably get by with much less data cache than it currently allocates for SQUASHFS_DECOMP_MULTI and SQUASHFS_DECOMP_MULTI_PERCPU when that option is set. We can't reduce the data cache for CONFIG_SQUASHFS_DECOMP_SINGLE since it always needs enough cache for at least one uncompressed data block.

Also important to note is that although the squashfs data cache size is determined by which decompressor implementation is selected, the cache and decompressor states are managed independently, therefore there's no requirement that they are so tightly coupled. The coupling makes sense if SQUASHFS_FILE_DIRECT is disabled but seems less sensible if it's enabled.

The decompressor implementations come with their own tradeoffs:

- CONFIG_SQUASHFS_DECOMP_SINGLE: Uses less memory. However decompression is serialized, i.e. for a given superblock only one block can be decompressed at a time.

- SQUASHFS_DECOMP_MULTI: Allows for (num_online_cpus() * 2) parallel decompressions, which means the number of parallel decompressions could vary depending on how many CPUs were online when the filesystem was mounted. Also allocates the same number of blocks for the data cache. This implementation has more overhead associated with managing decompressor state memory than the others.

- SQUASHFS_DECOMP_MULTI_PERCPU: This maintains per-cpu decompressor state, therefore it is lockless and there is no overhead for managing the decompressor state memory. That means it uses more RAM though, both for decompressor state memory and for data cache. You also can have as many parallel decompressions going on as there are CPU cores.

Based on this, here are a couple of ideas we could try to strike a good balance between performance and RAM usage:

1. Add a config option for the maximum number of data cache blocks. This would allow us to use one of the implementations which allows for parallel decompression without the data cache size exploding. As long as most blocks get decompressed directly into the page cache (which we'll need to verify) having only one or two blocks in the data cache should not be detrimental to performance.

2. Make a new decompressor implementation which allows for full customization of the number of parallel decompressions and number of blocks in the data cache. In my opinion SQUASHFS_DECOMP_SINGLE is too little parallelization but the others are too much on systems with large numbers of CPUs. Instead we could specify both the maximum number of parallel decompressions in the kernel config (possibly capped at the number of possible CPU cores) and the number of data blocks to cache to suit our needs.