multipath nvme, failed to install with multipath disabled install failed crashed with CalledProcessError

[Dimitri John Ledkov]

multipath nvme, failed to install with multipath disabled install failed crashed with CalledProcessError

so trying to install with nvme_core.multipath=0 set on the cmdline, and that fails.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: subiquity (1641)
ProcVersionSignature: Ubuntu 5.4.0-21.25-generic 5.4.27
Uname: Linux 5.4.0-21-generic ppc64le
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu24
Architecture: ppc64el
CasperVersion: 1.443
CrashDB:
 {
        "impl": "launchpad",
        "project": "subiquity",
        "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml"
     }
Date: Wed Apr 8 11:35:20 2020
ExecutablePath: /snap/subiquity/1638/usr/bin/subiquity
InstallerLog: Error: [Errno 2] No such file or directory: 'logfile'
InterpreterPath: /snap/subiquity/1638/usr/bin/python3.6
LiveMediaBuild: Ubuntu-Server 20.04 LTS "Focal Fossa" - Beta ppc64el (20200408)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 002: ID 1d6b:0107 Linux Foundation USB Virtual Hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
ProcAttrCurrent: snap.subiquity.subiquity (complain)
ProcCmdline: /snap/subiquity/1638/usr/bin/python3 /snap/subiquity/1638/usr/bin/subiquity
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: ip=dhcp url=http://cdimage.ubuntu.com/hostname/daily-live/pending/focal-live-server-ppc64el.iso subiquity-channel=latest/edge nvme_core.multipath=0 http_proxy=http://91.189.89.11:3128 --- quiet
ProcLoadAvg: 0.11 1.05 0.96 1/1380 14592
ProcLocks:
 1: FLOCK ADVISORY WRITE 5503 00:18:751 0 EOF
 2: POSIX ADVISORY WRITE 5198 00:18:647 0 EOF
 3: POSIX ADVISORY WRITE 5520 00:18:760 0 EOF
ProcSwaps: Filename Type Size Used Priority
ProcVersion: Linux version 5.4.0-21-generic (buildd@bos02-ppc64el-028) (gcc version 9.3.0 (Ubuntu 9.3.0-8ubuntu1)) #25-Ubuntu SMP Sat Mar 28 13:10:37 UTC 2020
Python3Details: /usr/bin/python3.8, Python 3.8.2, python3-minimal, 3.8.2-0ubuntu2
PythonDetails: N/A
SnapChannel:

SnapRevision: 1638
SnapUpdated: False
SnapVersion: 20.03.3+git132.a0dae13d
SourcePackage: subiquity
Title: install failed crashed with CalledProcessError
UpgradeStatus: No upgrade log present (probably fresh install)
UsingAnswers: False
VarLogDump_list: total 0
cpu_cores: Number of cores present = 32
cpu_coreson: Number of cores online = 32
cpu_smt: SMT=4

[Dimitri John Ledkov (xnox)]

Looks like with nvme_core.multipath=0 nvme install still fails.

[Dimitri John Ledkov (xnox)]

/dev/nvme0* is now gone, yet still mounted as /target. Yeap this is borken.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Ryan Harper (raharper)]

The current error looks like /target got unmounted ... or there was some
corruption that forced the mount into read-only mode...

Running command ['sh', '-c', 'mkdir -p "$2" && cd "$2" && rsync -aXHAS --one-file-system "$1/" .', '--', '/media/filesystem', '/target'] with allowed return codes [0] (capture=False)
rsync: write failed on "/target/usr/lib/powerpc64le-linux-gnu/liblmdb.so.0.0.0": Read-only file system (30)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]
finish: cmd-install/stage-extract/builtin/cmd-extract: FAIL: acquiring and extracting image from cp:///media/filesystem
finish: cmd-install/stage-extract/builtin/cmd-extract: FAIL: curtin command extract
Traceback (most recent call last):
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/commands/main.py", line 202, in main
    ret = args.func(args)
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/commands/extract.py", line 249, in extract
    copy_to_target(source['uri'], target)
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/commands/extract.py", line 212, in copy_to_target
    '--', source, target])
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/util.py", line 275, in subp
    return _subp(*args, **kwargs)
  File "/snap/subiquity/1638/lib/python3.6/site-packages/curtin/util.py", line 141, in _subp
    cmd=args)
curtin.util.ProcessExecutionError: Unexpected error while running command.

/me looks a CurrentDmesg

[ 712.956086] nvme nvme0: I/O 288 QID 110 timeout, aborting
[ 712.956105] nvme nvme0: I/O 289 QID 110 timeout, aborting
[ 712.956111] nvme nvme0: I/O 290 QID 110 timeout, aborting
[ 712.956118] nvme nvme0: I/O 291 QID 110 timeout, aborting
[ 712.956124] nvme nvme0: I/O 292 QID 110 timeout, aborting
[ 712.956142] nvme nvme0: Abort status: 0x0
[ 712.956146] nvme nvme0: Abort status: 0x0
[ 712.956148] nvme nvme0: Abort status: 0x0
[ 712.956150] nvme nvme0: Abort status: 0x0

And *boom*

[ 744.445233] nvme nvme0: I/O 288 QID 110 timeout, reset controller
[ 805.887142] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
[ 846.848434] INFO: task kworker/u257:0:7 blocked for more than 120 seconds.
[ 846.848437] Tainted: P O 5.4.0-21-generic #25-Ubuntu
[ 846.848438] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 846.848440] kworker/u257:0 D 0 7 2 0x00000808
[ 846.848448] Workqueue: writeback wb_workfn (flush-259:0)
[ 846.848450] Call Trace:
[ 846.848453] [c000003ff4ad7020] [c000003ff4ad70d0] 0xc000003ff4ad70d0 (unreliable)
[ 846.848458] [c000003ff4ad7200] [c000000000021a2c] __switch_to+0x2dc/0x450
[ 846.848461] [c000003ff4ad7270] [c000000000ede1fc] __schedule+0x2ec/0x930
[ 846.848464] [c000003ff4ad7350] [c000000000ede898] schedule+0x58/0x130
[ 846.848467] [c000003ff4ad7380] [c0000000005fe67c] wait_transaction_locked+0xac/0x110
[ 846.848470] [c000003ff4ad73f0] [c0000000005fe8c8] add_transaction_credits+0x1e8/0x480
[ 846.848472] [c000003ff4ad7480] [c0000000005fed5c] start_this_handle+0x15c/0x5a0
[ 846.848475] [c000003ff4ad75a0] [c0000000005ff75c] jbd2__journal_start+0x14c/0x2f0
[ 846.84...

[Ryan Harper (raharper)]

I'm marking curtin task invalid; this looks like kernel/platform issue at this point. Please reopen curtin task if curtin needs to fix something.

[Paolo Pisati (p-pisati)]

There were several nvme related commits in the last kernel upload (Ubuntu-5.4.0-24.28), could you try it?

[Michael Hudson-Doyle (mwhudson)]

I'm not sure of the status of this bug now. Was it all fixed by release?

[Chee Yang Chau (chaucheeyang)]

I am having the same issue too. Any update so far?