| 2015-02-11 02:01:26 |
Patrick Crews |
description |
Running a randomized snapshot + restore workload on a single heat stack (single vm) with 2 concurrent users causes significant memory consumption that continues even after all tests are completed.
KiB Mem: 37066812 total, 6883732 used, 30183080 free, 244800 buffers
KiB Mem: 37066812 total, 35717836 used, 1348976 free, 29860 buffers <- post workload
Tests:
clone https://github.com/pcrews/rannsaka
cd rannsaka/rannsaka
python rannsaka.py --host=http://192.168.0.5 --requests 500 -w 2 --test-file=locust_files/heat_basic_stress.py
# PRE STRESS TEST
top - 12:01:42 up 1:04, 2 users, load average: 0.87, 0.87, 0.58
Tasks: 368 total, 2 running, 366 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.3 us, 0.3 sy, 0.3 ni, 96.5 id, 1.6 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 37066812 total, 6883732 used, 30183080 free, 244800 buffers
KiB Swap: 37738492 total, 0 used, 37738492 free. 2516732 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14687 stack 20 0 221808 8092 4596 R 99.3 0.0 0:05.49 qemu-img
2208 rabbitmq 20 0 2349296 66804 2516 S 6.2 0.2 0:28.53 beam.smp
12106 stack 20 0 173852 73228 5084 S 6.2 0.2 0:03.91 glance-api
14705 stack 20 0 25212 1728 1088 R 6.2 0.0 0:00.01 top
# POST STRESS TEST
top - 12:46:17 up 1:49, 3 users, load average: 0.34, 0.53, 1.39
Tasks: 376 total, 2 running, 374 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 0.2 sy, 0.0 ni, 98.5 id, 0.5 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 37066812 total, 35717836 used, 1348976 free, 29860 buffers
KiB Swap: 37738492 total, 251000 used, 37487492 free. 30435868 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21654 libvirt+ 20 0 4812112 491872 9416 S 0.3 1.3 0:15.80 qemu-system-x86
21402 libvirt+ 20 0 4743248 481652 9392 S 0.3 1.3 0:36.78 qemu-system-x86
12281 stack 20 0 340540 145080 3732 S 0.0 0.4 0:12.59 nova-api
12282 stack 20 0 340524 144968 3760 S 0.0 0.4 0:12.53 nova-api
12280 stack 20 0 339660 143952 3736 S 0.0 0.4 0:12.08 nova-api
12279 stack 20 0 337568 141988 3732 S 0.0 0.4 0:13.15 nova-api
9784 mysql 20 0 4099628 121912 7984 S 0.3 0.3 0:34.01 mysqld
12423 stack 20 0 276484 89216 3608 S 0.0 0.2 0:12.66 nova-conductor
12428 stack 20 0 276584 89184 3604 S 0.3 0.2 0:12.86 nova-conductor
12424 stack 20 0 276372 89148 3608 S 0.0 0.2 0:12.27 nova-conductor
12422 stack 20 0 276112 88764 3604 S 0.7 0.2 0:12.51 nova-conductor
12425 stack 20 0 275296 87968 3604 S 0.3 0.2 0:12.79 nova-conductor
12429 stack 20 0 275232 87776 3600 S 0.3 0.2 0:12.76 nova-conductor
12262 stack 20 0 190240 87700 5472 S 0.7 0.2 0:29.42 nova-api
12426 stack 20 0 273992 86596 3608 S 0.0 0.2 0:11.64 nova-conductor
12427 stack 20 0 272976 85724 3592 S 0.3 0.2 0:12.00 nova-conductor
12289 stack 20 0 190240 83728 1568 S 0.0 0.2 0:00.01 nova-api
12290 stack 20 0 190240 83728 1568 S 0.0 0.2 0:00.00 nova-api
12291 stack 20 0 190240 83728 1568 S 0.0 0.2 0:00.01 nova-api
12292 stack 20 0 190240 83728 1568 S 0.0 0.2 0:00.01 nova-api |
Running randomized image creation tests via nova and heat shows significant memory consumption tied to the number of snapshots.
The post-workload value is with ~30 snapshots in the system.
Not sure if this is due to my devstack install / configuration, but this seemed worth writing up for additional scrutiny.
KiB Mem: 37066812 total, 6883732 used, 30183080 free, 244800 buffers
KiB Mem: 37066812 total, 35717836 used, 1348976 free, 29860 buffers <- post workload
Tests:
# Create a vm / heat stack / set of vm's
# repeatedly snapshot them and monitor memory usage as the number of stored images increases.
# My tests:
clone https://github.com/pcrews/rannsaka
cd rannsaka/rannsaka
# expecting at least one stack to snapshot / does not create it in the test
python rannsaka.py --host=http://192.168.0.5 --requests 500 -w 2 --test-file=locust_files/heat_basic_stress.py
Have tried many test variations with varying numbers of servers / stacks, but memory consumption appears tied to the total number of images stored in the system:
[2015-02-10 15:09:34,861] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 1 || 30216604 K free memory
[2015-02-10 15:09:49,724] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 1 || 26717180 K free memory
[2015-02-10 15:09:59,242] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 1 || 27261024 K free memory
[2015-02-10 15:12:35,072] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 5 || 23324884 K free memory
[2015-02-10 15:12:38,312] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 5 || 22932740 K free memory
[2015-02-10 15:12:41,728] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 5 || 22703904 K free memory
[2015-02-10 15:13:00,152] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 5 || 20729056 K free memory
[2015-02-10 15:15:38,137] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 10 || 16715384 K free memory
[2015-02-10 15:15:44,306] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 10 || 16581624 K free memory
[2015-02-10 15:16:05,172] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 10 || 15414400 K free memory
[2015-02-10 15:16:11,367] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 10 || 14560096 K free memory
[2015-02-10 15:19:17,026] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 15 || 10001072 K free memory
[2015-02-10 15:19:25,819] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 15 || 8794308 K free memory
[2015-02-10 15:20:59,007] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 17 || 7489788 K free memory
[2015-02-10 15:21:17,554] testbox/INFO/stdout: 0x7f9c6f819f90: SNAP_COUNT: 17 || 5997816 K free memory
NOTE: comments 2-5 were when I thought this was a heat-bug and are not particularly relevant. |
|
