CI: services memory fails

Bug #1611842 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Sagi (Sergey) Shnaidman

Bug Description

There are a lot of puppet memory problems recently, mostly in HA jobs (but not limited to)

Building remotely on tripleo-centos-7-tripleo-test-cloud-rh1-3344848

HA:

Error: Could not prefetch keystone_domain provider 'openstack': Cannot allocate memory - fork(2)
http://logs.openstack.org/07/353307/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/dee6294/logs/postci.txt.gz

Error: Could not set 'present' on ensure: Cannot allocate memory - fork(2)
http://logs.openstack.org/64/352364/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/d6271f0/logs/postci.txt.gz

non-HA:

Error: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Could not evaluate: Cannot allocate memory - fork(2)
http://logs.openstack.org/72/329872/15/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/91d3244/logs/postci.txt.gz

+ free -h
              total used free shared buff/cache available
Mem: 5.8G 5.4G 169M 624K 309M 224M
Swap: 8.0G 62M 7.9G

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  872 heat 20 0 690328 460072 3392 S 0.0 7.5 5:23.60 heat-engine
  873 heat 20 0 527468 297088 3416 S 0.0 4.9 6:24.99 heat-engine
  874 heat 20 0 521608 291536 3580 S 0.0 4.8 5:16.55 heat-engine
  875 heat 20 0 514928 284672 3396 S 0.0 4.7 4:19.51 heat-engine
25204 mysql 20 0 1650920 283484 6144 S 0.0 4.6 1:38.52 mysqld
 1006 nova 20 0 842320 193192 4716 S 0.0 3.2 1:42.27 httpd

Tags: ci
description: updated
Ben Nemec (bnemec)
Changed in tripleo:
importance: Undecided → High
status: New → Triaged
summary: - CI: puppet memory fails
+ CI: services memory fails
Changed in tripleo:
milestone: none → newton-3
Changed in tripleo:
assignee: nobody → Sagi (Sergey) Shnaidman (sshnaidm)
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

With adding memory to overcloud vms it passed
https://review.openstack.org/#/c/353655/

Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

All ovb-ha jobs are failing again with puppet memory problem.

Changed in tripleo:
status: Fix Released → Triaged
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Steven Hardy (shardy)
Changed in tripleo:
milestone: newton-3 → ongoing
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

After increasing memory for undercloud to 8GB it started passing again.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.