storing multiple snapshots causing significant memory consumption
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Glance |
Incomplete
|
Undecided
|
Unassigned |
Bug Description
Running randomized image creation tests via nova and heat shows significant memory consumption tied to the number of snapshots.
The post-workload value is with ~30 snapshots in the system.
Not sure if this is due to my devstack install / configuration, but this seemed worth writing up for additional scrutiny.
KiB Mem: 37066812 total, 6883732 used, 30183080 free, 244800 buffers
KiB Mem: 37066812 total, 35717836 used, 1348976 free, 29860 buffers <- post workload
Tests:
# Create a vm / heat stack / set of vm's
# repeatedly snapshot them and monitor memory usage as the number of stored images increases.
# My tests:
clone https:/
cd rannsaka/rannsaka
# expecting at least one stack to snapshot / does not create it in the test
python rannsaka.py --host=http://
Have tried many test variations with varying numbers of servers / stacks, but memory consumption appears tied to the total number of images stored in the system:
[2015-02-10 15:09:34,861] testbox/
[2015-02-10 15:09:49,724] testbox/
[2015-02-10 15:09:59,242] testbox/
[2015-02-10 15:12:35,072] testbox/
[2015-02-10 15:12:38,312] testbox/
[2015-02-10 15:12:41,728] testbox/
[2015-02-10 15:13:00,152] testbox/
[2015-02-10 15:15:38,137] testbox/
[2015-02-10 15:15:44,306] testbox/
[2015-02-10 15:16:05,172] testbox/
[2015-02-10 15:16:11,367] testbox/
[2015-02-10 15:19:17,026] testbox/
[2015-02-10 15:19:25,819] testbox/
[2015-02-10 15:20:59,007] testbox/
[2015-02-10 15:21:17,554] testbox/
NOTE: comments 2-5 were when I thought this was a heat-bug and are not particularly relevant.
affects: | heat → glance |
summary: |
- storing multiple heat snapshots causing significant memory consumption + storing multiple snapshots causing significant memory consumption |
description: | updated |
Changed in glance: | |
status: | Incomplete → New |
Changed in glance: | |
status: | New → Incomplete |
the engine logs have a large number of these messages: python2. 7/dist- packages/ amqp/channel. py:608: DeprecationWarning: auto_delete exchanges has been deprecated
/usr/lib/
'auto_delete exchanges has been deprecated'))
2015-01-20 09:43:21.418 ERROR oslo.messaging. _drivers. impl_rabbit [-] AMQP server on 192.168.0.5:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. _drivers. impl_rabbit [-] Failed to consume message from queue: [Errno 104] Connection reset by peer _drivers. impl_rabbit Traceback (most recent call last): _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 440, in _ensured _drivers. impl_rabbit return fun(*args, **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 512, in __call__ _drivers. impl_rabbit return fun(*args, channel= channels[ 0], **kwargs), channels[0] _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 695, in _consume _drivers. impl_rabbit return self.connection .drain_ events( timeout= 1) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 279, in drain_events _drivers. impl_rabbit return self.transport. drain_events( self.connection , **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/transport /pyamqp. py", line 90, in drain_events _drivers. impl_rabbit return connection. drain_events( **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/connection .py", line 303, in drain_events _drivers. impl_rabbit return amqp_method( channel, args) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/connection .py", line 506, in _close _drivers. impl_rabbit self._x_close_ok() _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/connection .py", line 534, in _x_close_ok _drivers. impl_rabbit self._send_ method( (10, 51)) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/abstract_ channel. py", line 62, in _send_method _drivers. impl_rabbit self.channel_id, method_sig, args, content, _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/method_ framing. py", line 227, in writ...
2015-01-20 09:43:21.419 ERROR oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.
2015-01-20 09:43:21.419 TRACE oslo.messaging.