| 2016-01-14 08:27:07 |
Gavin Guo |
bug |
|
|
added bug |
| 2016-01-14 08:28:59 |
Gavin Guo |
description |
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
|
| 2016-01-14 08:30:14 |
Brad Figg |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2016-01-14 08:30:16 |
Brad Figg |
tags |
|
trusty |
|
| 2016-01-14 08:51:35 |
Gema Gomez |
description |
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
Here is the trace from KASAN:
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
|
| 2016-01-14 08:51:42 |
Gema Gomez |
tags |
trusty |
sts trusty |
|
| 2016-01-14 09:04:55 |
Gema Gomez |
description |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
Here is the trace from KASAN:
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
The failing VM was running on a host with kernel 3.13.0-66-generic (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0
Here is the trace from KASAN (from the VM):
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
|
| 2016-01-14 09:07:08 |
Gema Gomez |
description |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
The failing VM was running on a host with kernel 3.13.0-66-generic (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0
Here is the trace from KASAN (from the VM):
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
The failing VM was running on a host with kernel 3.13.0-66-generic (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0
The flavour of this VM is 4 CPUs, 8G RAM, 80G of root disk, 0 G swap and 0 G ephemeral disk.
Here is the trace from KASAN (from the VM):
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
|
| 2016-01-14 09:15:52 |
Gema Gomez |
description |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
The failing VM was running on a host with kernel 3.13.0-66-generic (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0
The flavour of this VM is 4 CPUs, 8G RAM, 80G of root disk, 0 G swap and 0 G ephemeral disk.
Here is the trace from KASAN (from the VM):
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
The failing VM was running on a host with kernel 3.13.0-66-generic (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0. Hosts' seabios: 1.7.5-1ubuntu1~cloud0
The flavour of this VM is 4 CPUs, 8G RAM, 80G of root disk, 0 G swap and 0 G ephemeral disk.
Here is the trace from KASAN (from the VM):
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
|
| 2016-01-14 11:25:56 |
Gavin Guo |
linux (Ubuntu): assignee |
|
Gavin Guo (mimi0213kimo) |
|
| 2016-01-14 12:07:50 |
Gavin Guo |
description |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
The failing VM was running on a host with kernel 3.13.0-66-generic (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0. Hosts' seabios: 1.7.5-1ubuntu1~cloud0
The flavour of this VM is 4 CPUs, 8G RAM, 80G of root disk, 0 G swap and 0 G ephemeral disk.
Here is the trace from KASAN (from the VM):
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled.
==================================================================
BUG: KASan: use after free in blk_mq_register_disk+0x193/0x260 at addr ffff8801ec247400
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Slab 0xffffea0007b091c0 objects=128 used=128 fp=0x (null) flags=0x2ffff0000000080
INFO: Object 0xffff8801ec247400 @offset=1024 fp=0xffff8801ec247420
Bytes b4 ffff8801ec2473f0: 00 ac 71 ef 01 88 ff ff 00 ac 79 ef 01 88 ff ff ..q.......y.....
Object ffff8801ec247400: 20 74 24 ec 01 88 ff ff 2f 76 69 72 74 75 61 6c t$...../virtual
Object ffff8801ec247410: 2f 62 64 69 2f 32 35 33 3a 30 00 00 00 00 00 00 /bdi/253:0......
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007b091c0 ffff8801ec0cb9a8 ffffffff81a6ce35 ffff8801ef001c00
ffff8801ec0cb9d8 ffffffff81244aed ffff8801ef001c00 ffffea0007b091c0
ffff8801ec247400 ffff8801ef79ac98 ffff8801ec0cba00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801ec247300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff8801ec247380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff8801ec247400: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801ec247480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801ec247500: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
================================================================== |
We are trying to debug the kernel using KASAN and we found that when a VM is booting in our cloud, on the virtualised kernel, there is a use-after-free access that should not be there.
The failing VM was running on a host with kernel 3.13.0-66-generic (trusty). Hosts' qemu version: 1:2.2+dfsg-5expubuntu9.3~cloud0. Hosts' seabios: 1.7.5-1ubuntu1~cloud0
The flavour of this VM is 4 CPUs, 8G RAM, 80G of root disk, 0 G swap and 0 G ephemeral disk.
Here is the trace from KASAN (from the VM):
The error message can be observed in the dmesg when the guest VM booted with v3.13.0-65 with KASAN enabled and "slub_debug=PU,kmalloc-32" in kernel command line.
==================================================================
BUG: KASan: out of bounds access in blk_mq_register_disk+0x193/0x260 at addr ffff8801f43f4d90
Read of size 8 by task swapper/0/1
=============================================================================
BUG kmalloc-32 (Not tainted): kasan: bad access detected
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: Allocated in blk_mq_init_hw_queues+0x778/0x920 age=5 cpu=1 pid=1
__slab_alloc+0x4f8/0x560
__kmalloc_node+0xad/0x310
blk_mq_init_hw_queues+0x778/0x920
blk_mq_init_queue+0x5f7/0x6c0
virtblk_probe+0x207/0x980
virtio_dev_probe+0x1be/0x280
driver_probe_device+0xe2/0x5c0
__driver_attach+0xc3/0xd0
bus_for_each_dev+0x95/0xe0
driver_attach+0x2b/0x30
bus_add_driver+0x268/0x360
driver_register+0xd3/0x1a0
register_virtio_driver+0x3c/0x60
init+0x53/0x80
do_one_initcall+0xda/0x1a0
kernel_init_freeable+0x1eb/0x27e
INFO: Freed in kzfree+0x2d/0x40 age=13 cpu=0 pid=8
__slab_free+0x2ab/0x3f0
kfree+0x161/0x170
kzfree+0x2d/0x40
aa_free_task_context+0x5d/0xa0
apparmor_cred_free+0x24/0x40
security_cred_free+0x2b/0x30
put_cred_rcu+0x38/0x140
rcu_nocb_kthread+0x25a/0x410
kthread+0x101/0x120
ret_from_fork+0x58/0x90
INFO: Slab 0xffffea0007d0fd00 objects=23 used=21 fp=0xffff8801f43f52d0 flags=0x2ffff0000004080
INFO: Object 0xffff8801f43f4d70 @offset=3440 fp=0xffff8801f43f5830
Bytes b4 ffff8801f43f4d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff8801f43f4d70: 00 ac 61 f7 01 88 ff ff 00 ac 69 f7 01 88 ff ff ..a.......i.....
Object ffff8801f43f4d80: 00 ac 71 f7 01 88 ff ff 00 ac 79 f7 01 88 ff ff ..q.......y.....
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B 3.13.0-65-generic #105
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.7.5-20150310_111955-batsu 04/01/2014
ffffea0007d0fd00 ffff8801f40cf9a8 ffffffff81a6ce35 ffff8801f7001c00
ffff8801f40cf9d8 ffffffff81244aed ffff8801f7001c00 ffffea0007d0fd00
ffff8801f43f4d70 ffff8801f779ac98 ffff8801f40cfa00 ffffffff8124ac36
Call Trace:
[<ffffffff81a6ce35>] dump_stack+0x45/0x56
[<ffffffff81244aed>] print_trailer+0xfd/0x170
[<ffffffff8124ac36>] object_err+0x36/0x40
[<ffffffff8124cbf9>] kasan_report_error+0x1e9/0x3a0
[<ffffffff81319427>] ? sysfs_get+0x17/0x50
[<ffffffff814dee6b>] ? kobject_add_internal+0x29b/0x4a0
[<ffffffff8124d260>] kasan_report+0x40/0x50
[<ffffffff81696f00>] ? dev_printk_emit+0x20/0x40
[<ffffffff814ae7c3>] ? blk_mq_register_disk+0x193/0x260
[<ffffffff8124bee9>] __asan_load8+0x69/0xa0
[<ffffffff814ae7c3>] blk_mq_register_disk+0x193/0x260
[<ffffffff814a1572>] blk_register_queue+0xd2/0x170
[<ffffffff814b24cf>] add_disk+0x31f/0x720
[<ffffffff816ced9a>] virtblk_probe+0x58a/0x980
[<ffffffff816cd4c0>] ? virtblk_restore+0x100/0x100
[<ffffffff81601b8e>] virtio_dev_probe+0x1be/0x280
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d0d2>] driver_probe_device+0xe2/0x5c0
[<ffffffff8169d620>] ? __device_attach+0x70/0x70
[<ffffffff8169d6e3>] __driver_attach+0xc3/0xd0
[<ffffffff8169a355>] bus_for_each_dev+0x95/0xe0
[<ffffffff8169c89b>] driver_attach+0x2b/0x30
[<ffffffff8169c298>] bus_add_driver+0x268/0x360
[<ffffffff8169dfe3>] driver_register+0xd3/0x1a0
[<ffffffff8218e4b9>] ? loop_init+0x14b/0x14b
[<ffffffff8160213c>] register_virtio_driver+0x3c/0x60
[<ffffffff8218e50c>] init+0x53/0x80
[<ffffffff8100212a>] do_one_initcall+0xda/0x1a0
[<ffffffff8213816b>] kernel_init_freeable+0x1eb/0x27e
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
[<ffffffff81a5bcde>] kernel_init+0xe/0x130
[<ffffffff81a83028>] ret_from_fork+0x58/0x90
[<ffffffff81a5bcd0>] ? rest_init+0x80/0x80
Memory state around the buggy address:
ffff8801f43f4c80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801f43f4d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc 00 00
>ffff8801f43f4d80: 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc fc
^
ffff8801f43f4e00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff8801f43f4e80: fc fc fc fc fc fc fc fc fc 00 00 00 00 fc fc fc
================================================================== |
|
| 2016-01-14 17:47:47 |
Gema Gomez |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2016-01-25 09:22:00 |
penalvch |
linux (Ubuntu): importance |
Undecided |
Medium |
|
