Comment 5 for bug 1871611

Revision history for this message
Ryan Harper (raharper) wrote :

The current error looks like /target got unmounted ... or there was some
corruption that forced the mount into read-only mode...

Running command ['sh', '-c', 'mkdir -p "$2" && cd "$2" && rsync -aXHAS --one-file-system "$1/" .', '--', '/media/filesystem', '/target'] with allowed return codes [0] (capture=False)
rsync: write failed on "/target/usr/lib/powerpc64le-linux-gnu/liblmdb.so.0.0.0": Read-only file system (30)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]
finish: cmd-install/stage-extract/builtin/cmd-extract: FAIL: acquiring and extracting image from cp:///media/filesystem
finish: cmd-install/stage-extract/builtin/cmd-extract: FAIL: curtin command extract
Traceback (most recent call last):
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/commands/main.py", line 202, in main
    ret = args.func(args)
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/commands/extract.py", line 249, in extract
    copy_to_target(source['uri'], target)
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/commands/extract.py", line 212, in copy_to_target
    '--', source, target])
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/util.py", line 275, in subp
    return _subp(*args, **kwargs)
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/util.py", line 141, in _subp
    cmd=args)
curtin.util.ProcessExecutionError: Unexpected error while running command.

/me looks a CurrentDmesg

[ 712.956086] nvme nvme0: I/O 288 QID 110 timeout, aborting
[ 712.956105] nvme nvme0: I/O 289 QID 110 timeout, aborting
[ 712.956111] nvme nvme0: I/O 290 QID 110 timeout, aborting
[ 712.956118] nvme nvme0: I/O 291 QID 110 timeout, aborting
[ 712.956124] nvme nvme0: I/O 292 QID 110 timeout, aborting
[ 712.956142] nvme nvme0: Abort status: 0x0
[ 712.956146] nvme nvme0: Abort status: 0x0
[ 712.956148] nvme nvme0: Abort status: 0x0
[ 712.956150] nvme nvme0: Abort status: 0x0

And *boom*

[ 744.445233] nvme nvme0: I/O 288 QID 110 timeout, reset controller
[ 805.887142] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[ 846.848434] INFO: task kworker/u257:0:7 blocked for more than 120 seconds.
[ 846.848437] Tainted: P O 5.4.0-21-generic #25-Ubuntu
[ 846.848438] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 846.848440] kworker/u257:0 D 0 7 2 0x00000808
[ 846.848448] Workqueue: writeback wb_workfn (flush-259:0)
[ 846.848450] Call Trace:
[ 846.848453] [c000003ff4ad7020] [c000003ff4ad70d0] 0xc000003ff4ad70d0 (unreliable)
[ 846.848458] [c000003ff4ad7200] [c000000000021a2c] __switch_to+0x2dc/0x450
[ 846.848461] [c000003ff4ad7270] [c000000000ede1fc] __schedule+0x2ec/0x930
[ 846.848464] [c000003ff4ad7350] [c000000000ede898] schedule+0x58/0x130
[ 846.848467] [c000003ff4ad7380] [c0000000005fe67c] wait_transaction_locked+0xac/0x110
[ 846.848470] [c000003ff4ad73f0] [c0000000005fe8c8] add_transaction_credits+0x1e8/0x480
[ 846.848472] [c000003ff4ad7480] [c0000000005fed5c] start_this_handle+0x15c/0x5a0
[ 846.848475] [c000003ff4ad75a0] [c0000000005ff75c] jbd2__journal_start+0x14c/0x2f0
[ 846.848478] [c000003ff4ad7620] [c00000000057d844] __ext4_journal_start_sb+0x84/0x170
[ 846.848481] [c000003ff4ad7680] [c0000000005a7338] ext4_writepages+0x848/0xfb0
[ 846.848483] [c000003ff4ad7830] [c000000000374800] do_writepages+0x60/0x160
[ 846.848486] [c000003ff4ad78a0] [c0000000004c7a60] __writeback_single_inode+0x70/0x540
[ 846.848488] [c000003ff4ad7900] [c0000000004c81e8] writeback_sb_inodes+0x2b8/0x6b0
[ 846.848490] [c000003ff4ad7a30] [c0000000004c8654] __writeback_inodes_wb+0x74/0x130
[ 846.848492] [c000003ff4ad7a90] [c0000000004c8b7c] wb_writeback+0x31c/0x490
[ 846.848494] [c000003ff4ad7b60] [c0000000004ca9a4] wb_workfn+0x4a4/0x630
[ 846.848498] [c000003ff4ad7c80] [c0000000001677dc] process_one_work+0x29c/0x550
[ 846.848500] [c000003ff4ad7d10] [c000000000167b28] worker_thread+0x98/0x5f0
[ 846.848503] [c000003ff4ad7db0] [c000000000171e44] kthread+0x1a4/0x1b0
[ 846.848506] [c000003ff4ad7e20] [c00000000000b648] ret_from_kernel_thread+0x5c/0x74
[ 846.848716] INFO: task jbd2/nvme0n1p2-:14090 blocked for more than 120 seconds.
[ 846.848717] Tainted: P O 5.4.0-21-generic #25-Ubuntu
[ 846.848718] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 846.848719] jbd2/nvme0n1p2- D 0 14090 2 0x00000808
[ 846.848721] Call Trace:
[ 846.848724] [c000003fc974b7a0] [c00000000019c620] load_balance+0x1c0/0xe20 (unreliable)
[ 846.848727] [c000003fc974b980] [c000000000021a2c] __switch_to+0x2dc/0x450
[ 846.848730] [c000003fc974b9f0] [c000000000ede1fc] __schedule+0x2ec/0x930
[ 846.848732] [c000003fc974bad0] [c000000000ede898] schedule+0x58/0x130
[ 846.848735] [c000003fc974bb00] [c000000000603c84] jbd2_journal_commit_transaction+0x274/0x1f60
[ 846.848737] [c000003fc974bd10] [c00000000060c04c] kjournald2+0xfc/0x380
[ 846.848740] [c000003fc974bdb0] [c000000000171e44] kthread+0x1a4/0x1b0
[ 846.848742] [c000003fc974be20] [c00000000000b648] ret_from_kernel_thread+0x5c/0x74
[ 846.848744] INFO: task ext4lazyinit:14092 blocked for more than 120 seconds.
[ 846.848745] Tainted: P O 5.4.0-21-generic #25-Ubuntu
[ 846.848746] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 846.848747] ext4lazyinit D 0 14092 2 0x00000808
[ 846.848749] Call Trace:
[ 846.848750] [c000003fc971f630] [c000003fc971f6e0] 0xc000003fc971f6e0 (unreliable)
[ 846.848753] [c000003fc971f810] [c000000000021a2c] __switch_to+0x2dc/0x450
[ 846.848755] [c000003fc971f880] [c000000000ede1fc] __schedule+0x2ec/0x930
[ 846.848758] [c000003fc971f960] [c000000000ede898] schedule+0x58/0x130
[ 846.848760] [c000003fc971f990] [c000000000ee4678] schedule_timeout+0x2c8/0x3e0
[ 846.848763] [c000003fc971fa70] [c000000000edef88] io_schedule_timeout+0x38/0x70
[ 846.848765] [c000003fc971faa0] [c000000000edf998] wait_for_completion_io+0xf8/0x230
[ 846.848769] [c000003fc971fb30] [c000000000766a6c] submit_bio_wait+0x7c/0xc0
[ 846.848771] [c000003fc971fb90] [c00000000077e9dc] blkdev_issue_zeroout+0xdc/0x290
[ 846.848774] [c000003fc971fc40] [c00000000059550c] ext4_init_inode_table+0x22c/0x4d0
[ 846.848776] [c000003fc971fd10] [c0000000005e36d0] ext4_lazyinit_thread+0x420/0x4b0
[ 846.848779] [c000003fc971fdb0] [c000000000171e44] kthread+0x1a4/0x1b0
[ 846.848782] [c000003fc971fe20] [c00000000000b648] ret_from_kernel_thread+0x5c/0x74
[ 846.848784] INFO: task rsync:14172 blocked for more than 120 seconds.
[ 846.848785] Tainted: P O 5.4.0-21-generic #25-Ubuntu
[ 846.848786] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 846.848787] rsync D 0 14172 14171 0x00040000
[ 846.848789] Call Trace:
[ 846.848790] [c000003eff673430] [c000003eff6734e0] 0xc000003eff6734e0 (unreliable)
[ 846.848793] [c000003eff673610] [c000000000021a2c] __switch_to+0x2dc/0x450
[ 846.848795] [c000003eff673680] [c000000000ede1fc] __schedule+0x2ec/0x930
[ 846.848798] [c000003eff673760] [c000000000ede898] schedule+0x58/0x130
[ 846.848800] [c000003eff673790] [c0000000005fe67c] wait_transaction_locked+0xac/0x110
[ 846.848803] [c000003eff673800] [c0000000005fe8c8] add_transaction_credits+0x1e8/0x480
[ 846.848805] [c000003eff673890] [c0000000005fed5c] start_this_handle+0x15c/0x5a0
[ 846.848808] [c000003eff6739b0] [c0000000005ff75c] jbd2__journal_start+0x14c/0x2f0
[ 846.848810] [c000003eff673a30] [c00000000057d844] __ext4_journal_start_sb+0x84/0x170
[ 846.848813] [c000003eff673a90] [c0000000005a9db0] ext4_da_write_begin+0x270/0x680
[ 846.848816] [c000003eff673b50] [c000000000361340] generic_perform_write+0xf0/0x280
[ 846.848818] [c000003eff673c00] [c000000000368130] __generic_file_write_iter+0x180/0x2f0
[ 846.848820] [c000003eff673c60] [c00000000058ec94] ext4_file_write_iter+0x214/0x430
[ 846.848823] [c000003eff673ce0] [c000000000473ebc] new_sync_write+0x13c/0x1f0
[ 846.848826] [c000003eff673d80] [c00000000047534c] vfs_write+0x11c/0x2a0
[ 846.848828] [c000003eff673dd0] [c000000000477ea4] ksys_write+0x84/0x140
[ 846.848831] [c000003eff673e20] [c00000000000b278] system_call+0x5c/0x68
[ 849.028500] nvme nvme0: Device not ready; aborting reset