[CentOS7] Page allocation failure on master node

Bug #1523484 reported by Dmitry Teselkin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Low
Ivan Suzdal

Bug Description

After successful master node deployment from CentOS7 based ISO there are error messages in /var/log/messages

---
Dec 7 11:22:08 nailgun kernel: kworker/0:1: page allocation failure: order:0, mode:0x204000
Dec 7 11:22:08 nailgun kernel: CPU: 0 PID: 29 Comm: kworker/0:1 Not tainted 3.10.0-229.20.1.el7.x86_64 #1
Dec 7 11:22:08 nailgun kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Dec 7 11:22:08 nailgun kernel: Workqueue: kcopyd do_work [dm_mod]
Dec 7 11:22:08 nailgun kernel: 0000000000204000 000000001fe1b6eb ffff88003d4f34d8 ffffffff816045b6
Dec 7 11:22:08 nailgun kernel: ffff88003d4f3568 ffffffff8115c5c0 ffffea0000d8ee00 0000000000000000
Dec 7 11:22:08 nailgun kernel: ffff88003ffd9800 0000000000000000 ffff88003fc16978 000000001fe1b6eb
Dec 7 11:22:08 nailgun kernel: Call Trace:
Dec 7 11:22:08 nailgun kernel: [<ffffffff816045b6>] dump_stack+0x19/0x1b
Dec 7 11:22:08 nailgun kernel: [<ffffffff8115c5c0>] warn_alloc_failed+0x110/0x180
Dec 7 11:22:08 nailgun kernel: [<ffffffff81160d48>] __alloc_pages_nodemask+0x9a8/0xb90
Dec 7 11:22:08 nailgun kernel: [<ffffffff811ac81e>] ? __kmalloc+0x2e/0x230
Dec 7 11:22:08 nailgun kernel: [<ffffffffa008b41f>] ? __virtblk_add_req+0xdf/0x1c0 [virtio_blk]
Dec 7 11:22:08 nailgun kernel: [<ffffffff8119f3d9>] alloc_pages_current+0xa9/0x170
Dec 7 11:22:08 nailgun kernel: [<ffffffff811a94ac>] new_slab+0x2ec/0x300
Dec 7 11:22:08 nailgun kernel: [<ffffffff8160151b>] __slab_alloc+0x315/0x48f
Dec 7 11:22:08 nailgun kernel: [<ffffffffa012559c>] ? virtqueue_add_sgs+0x31c/0x59c [virtio_ring]
Dec 7 11:22:08 nailgun kernel: [<ffffffff812eda2d>] ? list_del+0xd/0x30
Dec 7 11:22:08 nailgun kernel: [<ffffffff816011eb>] ? get_partial_node.isra.48+0x1ee/0x209
Dec 7 11:22:08 nailgun kernel: [<ffffffff811ac9b8>] __kmalloc+0x1c8/0x230
Dec 7 11:22:08 nailgun kernel: [<ffffffffa012559c>] virtqueue_add_sgs+0x31c/0x59c [virtio_ring]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa008b41f>] __virtblk_add_req+0xdf/0x1c0 [virtio_blk]
Dec 7 11:22:08 nailgun kernel: [<ffffffff812b2fc6>] ? __blk_segment_map_sg+0x56/0x1b0
Dec 7 11:22:08 nailgun kernel: [<ffffffffa008b609>] virtio_queue_rq+0x109/0x280 [virtio_blk]
Dec 7 11:22:08 nailgun kernel: [<ffffffff812b6968>] __blk_mq_run_hw_queue+0x1c8/0x330
Dec 7 11:22:08 nailgun kernel: [<ffffffff812b70f5>] blk_mq_run_hw_queue+0x65/0x90
Dec 7 11:22:08 nailgun kernel: [<ffffffff812b80a8>] blk_sq_make_request+0x258/0x380
Dec 7 11:22:08 nailgun kernel: [<ffffffff812aa972>] generic_make_request+0xe2/0x130
Dec 7 11:22:08 nailgun kernel: [<ffffffff812aaa31>] submit_bio+0x71/0x150
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000ab7e>] dispatch_io+0x18e/0x360 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000a6b0>] ? dm_copy_name_and_uuid+0xc0/0xc0 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000a6e0>] ? list_get_page+0x30/0x30 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000b240>] ? dm_kcopyd_do_callback+0x40/0x40 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000ae5e>] dm_io+0x10e/0x2e0 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000a6b0>] ? dm_copy_name_and_uuid+0xc0/0xc0 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000a6e0>] ? list_get_page+0x30/0x30 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000bfa4>] run_io_job+0x124/0x200 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000b240>] ? dm_kcopyd_do_callback+0x40/0x40 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000b66d>] process_jobs+0x5d/0x100 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000be80>] ? dm_kcopyd_client_create+0x210/0x210 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffffa000b778>] do_work+0x68/0x90 [dm_mod]
Dec 7 11:22:08 nailgun kernel: [<ffffffff8108f0cb>] process_one_work+0x17b/0x470
Dec 7 11:22:08 nailgun kernel: [<ffffffff8108fe9b>] worker_thread+0x11b/0x400
Dec 7 11:22:08 nailgun kernel: [<ffffffff8108fd80>] ? rescuer_thread+0x400/0x400
Dec 7 11:22:08 nailgun kernel: [<ffffffff8109727f>] kthread+0xcf/0xe0
Dec 7 11:22:08 nailgun kernel: [<ffffffff810971b0>] ? kthread_create_on_node+0x140/0x140
Dec 7 11:22:08 nailgun kernel: [<ffffffff81614358>] ret_from_fork+0x58/0x90
Dec 7 11:22:08 nailgun kernel: [<ffffffff810971b0>] ? kthread_create_on_node+0x140/0x140
---

Ivan Suzdal (isuzdal)
Changed in mos:
assignee: nobody → Ivan Suzdal (isuzdal)
Revision history for this message
Ivan Suzdal (isuzdal) wrote :

custom iso fuel-gerrit-8.0-1019-2015-12-07_14-02-04.iso include changes in boostrap_admin_node.sh which set min_free_kbytes to 65535

Revision history for this message
Dmitry Teselkin (teselkin-d) wrote :

According to [0] minimal hardware requirements for HW (production) master node are:
* Quad-core CPU
* 4GB RAM
* 1 gigabit network port
* 128GB SAS Disk
* IPMI access through independent management network

If there is 4 GB of ram the kernel sets vm.min_free_kbytes=67584 which is even more than in the patch provided. So we don't to do anything for production environment.

[0] https://docs.mirantis.com/openstack/fuel/fuel-7.0/planning-guide.html#system-requirements

Changed in mos:
importance: Undecided → Low
status: New → Confirmed
Changed in mos:
milestone: none → 8.0
Ivan Suzdal (isuzdal)
Changed in mos:
status: Confirmed → Fix Committed
Changed in mos:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.