Crash(assert) during reading image from http url through qemu-nbd
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
QEMU |
Expired
|
Undecided
|
Unassigned |
Bug Description
Description:
During reading image from nbd device mounted by qemu-nbd server with url backend I/O error happens
"blk_update_
Steps to reproduce:
1) sudo go run qemu-nbd-
or try directly
1) qemu-nbd -c /dev/nbd0 -r -v --aio=native -f qcow2 json:{"
2) try read whole nbd device while error "blk_update_
Versions:
1) qemu built from sources(/configure --target-
, top commit 5619c179057e241
qemu-nbd -v
qemu-nbd 2.9.90 (v2.10.
2) libcurl(built from sources, top commit 1767adf4399bb3b
curl -V
curl 7.55.0-DEV (Linux) libcurl/7.55.0-DEV OpenSSL/1.0.2g zlib/1.2.8
Backtrace:
(gdb) bt
#0 0x00007f7131426428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/
#1 0x00007f713142802a in __GI_abort () at abort.c:89
#2 0x00007f713141ebd7 in __assert_fail_base (fmt=<optimized out>, assertion=
file=
function=
#3 0x00007f713141ec82 in __GI___assert_fail (assertion=0x54c924 "self != co", file=0x54c871 "util/async.c", line=468,
function=
#4 0x00000000004fe6a2 in aio_co_enter (ctx=0xf0ddb0, co=0xf14650) at util/async.c:468
#5 0x00000000004fe637 in aio_co_wake (co=0xf14650) at util/async.c:456
#6 0x0000000000495c8a in curl_read_cb (ptr=0xf566d9, size=1, nmemb=16135, opaque=0xf1cb90) at block/curl.c:275
#7 0x00007f713242ac24 in Curl_client_
#8 0x00007f713242ae03 in Curl_client_write () from /usr/lib/
#9 0x00007f713244e1cf in readwrite_data () from /usr/lib/
#10 0x00007f713244eb6f in Curl_readwrite () from /usr/lib/
#11 0x00007f713245c1bb in multi_runsingle () from /usr/lib/
#12 0x00007f713245d819 in multi_socket () from /usr/lib/
#13 0x00007f713245e067 in curl_multi_
#14 0x0000000000497555 in curl_setup_preadv (bs=0xf16820, acb=0x7f712d379860) at block/curl.c:918
#15 0x00000000004975fb in curl_co_preadv (bs=0xf16820, offset=6556160, bytes=512, qiov=0x7f712d37
#16 0x000000000047730f in bdrv_driver_preadv (bs=0xf16820, offset=6556160, bytes=512, qiov=0x7f712d37
#17 0x0000000000477c1f in bdrv_aligned_preadv (child=0xf1be20, req=0x7f712d379a60, offset=6556160, bytes=512, align=1,
qiov=
#18 0x0000000000478109 in bdrv_co_preadv (child=0xf1be20, offset=6556160, bytes=512, qiov=0x7f712d37
#19 0x0000000000437498 in qcow2_co_preadv (bs=0xf0fdc0, offset=21563904, bytes=512, qiov=0x7f712d37
at block/qcow2.c:1812
#20 0x000000000047730f in bdrv_driver_preadv (bs=0xf0fdc0, offset=21563904, bytes=512, qiov=0x7f712d37
at block/io.c:836
#21 0x0000000000477c1f in bdrv_aligned_preadv (child=0xf1c0d0, req=0x7f712d379d30, offset=21563904, bytes=512, align=1,
qiov=
#22 0x0000000000478109 in bdrv_co_preadv (child=0xf1c0d0, offset=21563904, bytes=512, qiov=0x7f712d37
at block/io.c:1180
#23 0x00000000004645ad in blk_co_preadv (blk=0xf1be90, offset=21563904, bytes=512, qiov=0x7f712d37
at block/block-
#24 0x00000000004646fa in blk_read_entry (opaque=
#25 0x000000000046481c in blk_prw (blk=0xf1be90, offset=21563904,
---Type <return> to continue, or q <return> to quit---
buf=0xf7f000 "2,NV\241t!
#26 0x0000000000464f81 in blk_pread (blk=0xf1be90, offset=21563904, buf=0xf7f000, count=512) at block/block-
#27 0x00000000004906cb in nbd_trip (opaque=0xf5a940) at nbd/server.c:1380
#28 0x000000000051c0a5 in coroutine_
#29 0x00007f713143b5d0 in ?? () from /lib/x86_
#30 0x00007f712d47a770 in ?? ()
#31 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7f712d37a000
description: | updated |
On 08/03/2017 07:12 AM, Andrey Smetanin wrote: request: I/O error, dev nbd0, sector 42117" dmesg. After some investigation I found that qemu-nbd server aborts in aio_co_enter() assert in util/async.c:468.
> Public bug reported:
>
> Description:
> During reading image from nbd device mounted by qemu-nbd server with url backend I/O error happens
> "blk_update_
>
Based on the backtrace, this looks to be a bug in the block/curl.c
driver, rather than the nbd/ or block/nbd.c code. If I'm right, it
should be possible to reproduce the crash using qemu-io directly on the
curl path, rather than adding the extra layer of an nbd client reading
through qemu-nbd (then again, having the qemu-nbd layer may be what is
allowing multiple parallel requests to hit the curl driver at once,
while qemu-io is not quite as easy to provoke into performing
complicated access patterns).
> bug-report/ qemu-nbd- bug.go (see qemu-nbd-bug- file.driver" :"http" ,"file. url":"http:// localhost: 9666/image","file. readahead" :3276800
> Steps to reproduce:
>
> 1) sudo go run qemu-nbd-
> report.tar.gz)
>
> or try directly
>
> 1) qemu-nbd -c /dev/nbd0 -r -v --aio=native -f qcow2 json:{"
Presumably, you've got something serving the file at port 9666?
> 2) try read whole nbd device while error in dmesg appears x list=x86_ 64-softmmu --disable-user --enable-curl --enable-linux-aio --enable-virtfs --enable-debug --disable-pie 95ff19c8fe6d6a6 cbcb16ed28) : 0-rc0-67- g5619c17) e29121435e1bb1c c2bc05f7bf) : unix/sysv/ linux/raise. c:54 assertion@ entry=0x54c924 "self != co", entry=0x54c871 "util/async.c", line=line@ entry=468, function@ entry=0x54c980 <__PRETTY_ FUNCTION_ _.24766> "aio_co_enter") at assert.c:92 FUNCTION_ _.24766> "aio_co_enter") at assert.c:101 chop_write () from /usr/lib/ x86_64- linux-gnu/ libcurl. so x86_64- linux-gnu/ libcurl. so x86_64- linux-gnu/ libcurl. so x86_64- linux-gnu/ libcurl. so x86_64- linux-gnu/ libcurl. so
>
> Versions:
>
> 1) qemu built from sources(/configure --target-
> , top commit 5619c179057e241
>
> qemu-nbd -v
> qemu-nbd 2.9.90 (v2.10.
>
> 2) libcurl(built from sources, top commit
> 1767adf4399bb3b
>
> curl -V
> curl 7.55.0-DEV (Linux) libcurl/7.55.0-DEV OpenSSL/1.0.2g zlib/1.2.8
>
>
> Backtrace:
> (gdb) bt
> #0 0x00007f7131426428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/
> #1 0x00007f713142802a in __GI_abort () at abort.c:89
> #2 0x00007f713141ebd7 in __assert_fail_base (fmt=<optimized out>, assertion=
> file=file@
> function=
> #3 0x00007f713141ec82 in __GI___assert_fail (assertion=0x54c924 "self != co", file=0x54c871 "util/async.c", line=468,
> function=0x54c980 <__PRETTY_
> #4 0x00000000004fe6a2 in aio_co_enter (ctx=0xf0ddb0, co=0xf14650) at util/async.c:468
> #5 0x00000000004fe637 in aio_co_wake (co=0xf14650) at util/async.c:456
> #6 0x0000000000495c8a in curl_read_cb (ptr=0xf566d9, size=1, nmemb=16135, opaque=0xf1cb90) at block/curl.c:275
> #7 0x00007f713242ac24 in Curl_client_
> #8 0x00007f713242ae03 in Curl_client_write () from /usr/lib/
> #9 0x00007f713244e1cf in readwrite_data () from /usr/lib/
> #10 0x00007f713244eb6f in Curl_readwrite () from /usr/lib/
> #11 0x00007f713245c1bb in multi_runsingle () from /usr/lib/
> #12 0x00007f713245d819 in ...