On Mon, Feb 10, 2020 at 07:53:20AM -0000, Andrea Righi wrote:
> OK, so we know that it's not related to the memory cgroup subsystem.
But this is a good instinct. It does seem to happen when eg firefox or git
is in heavy memory use, not the system as a whole.
> Another reason of such unexpected swapping activity could be due to
> memory compaction code that is triggering some direct memory reclaim and
> forcing to swap out pages.
>
> What do you have in /proc/sys/vm/swappiness? Could you try to set that
> to 0 (kernel prefers to drop file-backed pages instead of swapping out
> anonymous pages) and see if the swap out activity is still happening?
$ cat /proc/sys/vm/swappiness
60
I'll try 0.
> Moreover, we can do some more advanced tracing of the direct memory reclaim code, for example:
> $ sudo kprobe-perf -s 'p:shrink_node'
>
This doesn't work:
# kprobe-perf -s 'p:shrink_node'
ERROR: func shrink_node not in /sys/kernel/debug/tracing/available_filter_functions.
Either it doesn't exist, or, it might be unsafe to kprobe. Exiting. Use -F to override.
# ls -l /sys/kernel/debug/tracing/available_filter_functions
ls: cannot access '/sys/kernel/debug/tracing/available_filter_functions': No such file or directory
# find /sys/kernel/debug/tracing
/sys/kernel/debug/tracing
/sys/kernel/debug/tracing/instances
/sys/kernel/debug/tracing/trace_stat
/sys/kernel/debug/tracing/per_cpu
/sys/kernel/debug/tracing/per_cpu/cpu7
/sys/kernel/debug/tracing/per_cpu/cpu6
/sys/kernel/debug/tracing/per_cpu/cpu5
/sys/kernel/debug/tracing/per_cpu/cpu4
/sys/kernel/debug/tracing/per_cpu/cpu3
/sys/kernel/debug/tracing/per_cpu/cpu2
/sys/kernel/debug/tracing/per_cpu/cpu1
/sys/kernel/debug/tracing/per_cpu/cpu0
/sys/kernel/debug/tracing/options
#
On Mon, Feb 10, 2020 at 07:53:20AM -0000, Andrea Righi wrote:
> OK, so we know that it's not related to the memory cgroup subsystem.
But this is a good instinct. It does seem to happen when eg firefox or git
is in heavy memory use, not the system as a whole.
> Another reason of such unexpected swapping activity could be due to vm/swappiness? Could you try to set that
> memory compaction code that is triggering some direct memory reclaim and
> forcing to swap out pages.
>
> What do you have in /proc/sys/
> to 0 (kernel prefers to drop file-backed pages instead of swapping out
> anonymous pages) and see if the swap out activity is still happening?
$ cat /proc/sys/ vm/swappiness
60
I'll try 0.
> Moreover, we can do some more advanced tracing of the direct memory reclaim code, for example:
> $ sudo kprobe-perf -s 'p:shrink_node'
>
This doesn't work:
# kprobe-perf -s 'p:shrink_node' debug/tracing/ available_ filter_ functions. debug/tracing/ available_ filter_ functions debug/tracing/ available_ filter_ functions' : No such file or directory debug/tracing debug/tracing debug/tracing/ instances debug/tracing/ trace_stat debug/tracing/ per_cpu debug/tracing/ per_cpu/ cpu7 debug/tracing/ per_cpu/ cpu6 debug/tracing/ per_cpu/ cpu5 debug/tracing/ per_cpu/ cpu4 debug/tracing/ per_cpu/ cpu3 debug/tracing/ per_cpu/ cpu2 debug/tracing/ per_cpu/ cpu1 debug/tracing/ per_cpu/ cpu0 debug/tracing/ options
ERROR: func shrink_node not in /sys/kernel/
Either it doesn't exist, or, it might be unsafe to kprobe. Exiting. Use -F to override.
# ls -l /sys/kernel/
ls: cannot access '/sys/kernel/
# find /sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
/sys/kernel/
#
Thanks