CI: cinder fails to allocate memory while creating volume for ping test tenant

Bug #1639970 reported by Gabriele Cerami on 2016-11-07
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
High
Unassigned

Bug Description

HA master jobs are failing very often with this error

ERROR cinder.api.middleware.fault [req-05e49ca5-df00-4359-a6fe-b7890b95dc93 dc6bddee7bbe4405ad837ae3394b74d5 824a1cc0ec2a4723be811cbd8a0080d5 - default default] Caught error: <type 'exceptions.OSError'> [Errno [Errno 12] Cannot allocate memory

full log at http://logs.openstack.org/62/336562/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/e68fefe/logs/overcloud-controller-1/var/log/cinder/cinder-api.txt.gz#_2016-11-07_17_27_53_459

We found some general performance issues during a debug session, but most of them impact on CPU, not memory.

Thomas Herve was suggesting to tweak down httpd config to reduce limits for some services, and free some memory.

Tags: ci Edit Tag help
Gabriele Cerami (gcerami) wrote :

A shallow memory utilization analysis showd no particular memory eaters. In each controller, mysqld takes ~6%, http ~3%, and there's a long list of processes at 0-2%. Not easy to understand if any of the process is taking more memory than it should.

Gabriele Cerami (gcerami) wrote :

This is still happening, not very often, but still happening. I think it happens when test env is under heavy load. Downgrading to High since it's not blocking.

Changed in tripleo:
importance: Critical → High
Steven Hardy (shardy) on 2016-11-11
Changed in tripleo:
milestone: ocata-1 → ocata-2
Changed in tripleo:
milestone: ocata-2 → ocata-3
Changed in tripleo:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers