[train] ERROR root [ ] Image prepare failed: A process in the process pool was terminated abruptly while the future was running or pending failing on few jobs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Bhagyashri Shewale |
Bug Description
Few c8 train jobs are failing because of ERROR root [ ] Image prepare failed: A process in the process pool was terminated abruptly while the future was running or pending
Jobs affected:
periodic-
periodic-
periodic-
Error logs:
2020-09-11 11:51:18,146 143187 ERROR root [ ] Image prepare failed: A process in the process pool was terminated abruptly while the future was running or pending.
Traceback (most recent call last):
File "/usr/bin/
lock=lock)
File "/usr/lib/
uploader.
File "/usr/lib/
uploader.
File "/usr/lib/
for result in p.map(upload_task, self.upload_tasks):
File "/usr/lib64/
for element in iterable:
File "/usr/lib64/
yield fs.pop().result()
File "/usr/lib64/
return self.__get_result()
File "/usr/lib64/
raise self._exception
concurrent.
Reference link:
Changed in tripleo: | |
assignee: | nobody → Bhagyashri Shewale (bhagyashri-shewale) |
status: | Triaged → Fix Released |
OOM Killer
https:/ /logserver. rdoproject. org/54/ 29254/1/ check/periodic- tripleo- ci-centos- 8-multinode- 1ctlr-featurese t010-train/ bee02d1/ logs/undercloud /var/log/ extra/journal. txt.gz
Sep 11 11:50:54 undercloud. localdomain kernel: httpd invoked oom-killer: gfp_mask= 0x6200ca( GFP_HIGHUSER_ MOVABLE) , nodemask=(null), order=0, oom_score_adj=0 localdomain kernel: httpd cpuset= libpod- d3b833c5ac22b35 fc5873f2fcdde1d 1736aed0ed19159 c1f95b482ea8e23 cc86.scope mems_allowed=0 localdomain kernel: CPU: 4 PID: 128877 Comm: httpd Not tainted 4.18.0- 193.14. 2.el8_2. x86_64 #1 localdomain kernel: Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.11.0-2.el7 04/01/2014 localdomain kernel: Call Trace: localdomain kernel: dump_stack+ 0x5c/0x80 localdomain kernel: dump_header+ 0x6e/0x27a localdomain kernel: ? virtballoon_ oom_notify+ 0x25/0x70 [virtio_balloon] localdomain kernel: oom_kill_ process. cold.28+ 0xb/0x10 localdomain kernel: out_of_ memory+ 0x1ba/0x490 localdomain kernel: __alloc_ pages_slowpath+ 0xc40/0xd60 localdomain kernel: __alloc_ pages_nodemask+ 0x245/0x280 localdomain kernel: filemap_ fault+0x3d0/ 0x860 localdomain kernel: ? alloc_set_ pte+0x203/ 0x480 localdomain kernel: ? filemap_ map_pages+ 0x38d/0x3b0 localdomain kernel: ext4_filemap_ fault+0x2c/ 0x40 [ext4] localdomain kernel: __do_fault+ 0x38/0xc0 localdomain kernel: do_fault+ 0x191/0x3c0 localdomain kernel: __handle_ mm_fault+ 0x441/0x6a0 localdomain kernel: handle_ mm_fault+ 0xda/0x200 localdomain kernel: __do_page_ fault+0x22d/ 0x4e0 localdomain kernel: do_page_ fault+0x32/ 0x110 localdomain kernel: ? async_page_ fault+0x8/ 0x30 localdomain kernel: async_page_ fault+0x1e/ 0x30 localdomain kernel: RIP: 0033:0x7f48a2b5bf42 localdomain kernel: Code: Bad RIP value.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.
Sep 11 11:50:54 undercloud.