NVDIMM-N doesn't work properly on Dell EMC PowerEdge R840
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | ndctl |
Fix Released
|
Unknown
|
||
| | linux (Ubuntu) |
Undecided
|
Unassigned | ||
| | ndctl (Ubuntu) |
Undecided
|
Unassigned | ||
| | Bionic |
Medium
|
Unassigned | ||
| | Disco |
Medium
|
Unassigned | ||
| | Eoan |
Undecided
|
Unassigned | ||
Bug Description
On ubuntu 18.04.1 :
1°) With 4.15.0-43 two default namespaces in raw mode are visible while there shouldn't be any but no settings change could be applied.
root@server:~# ndctl list -R
[
{
"dev"
"size"
"available_
"type":"pmem",
"numa_node":1,
"persistenc
},
{
"dev"
"size"
"available_
"type":"pmem",
"numa_node":0,
"persistenc
}
]
root@server:~# ndctl list
[
{
"dev"
"mode":"raw",
"size"
"sector_
"blockdev"
"numa_node":1
},
{
"dev"
"mode":"raw",
"size"
"sector_
"blockdev"
"numa_node":0
}
]
root@server:~# ndctl create-namespace --reconfig=
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
libndctl: sizeof_
failed to reconfigure namespace: No such device
2°) With 4.18.0-13 no active namespaces are seen which seems more coherent for non initialized NVDIMM-N with labels but no namespace can be created.
Both patches (kernel & ndctl) listed in the github issue below needs to be applied in order to get them working properly. Tested against the current hwe 4.18 kernel.
https:/
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jan 15 10:55 seq
crw-rw---- 1 root audio 116, 33 Jan 15 10:55 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Dell Inc. PowerEdge R840
NonfreeKernelMo
Package: ndctl 61.2-0ubuntu1~
PackageArchitec
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware 1.173.2
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic
Uname: Linux 4.15.0-43-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 11/21/2018
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.3.9
dmi.board.name: 08XR9M
dmi.board.vendor: Dell Inc.
dmi.board.version: A01
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.
dmi.product.family: PowerEdge
dmi.product.name: PowerEdge R840
dmi.sys.vendor: Dell Inc.
Related branches
- Rafael David Tinoco: Needs Fixing on 2019-11-25
- Christian Ehrhardt : Needs Fixing on 2019-11-25
-
Diff: 119 lines (+99/-0)3 files modifieddebian/changelog (+6/-0)
debian/patches/ndctl-init-labels-Fix-label-slot-accounting-per-UEFI.patch (+92/-0)
debian/patches/series (+1/-0)
- Rafael David Tinoco: Needs Fixing on 2019-11-25
- Christian Ehrhardt : Needs Fixing on 2019-11-25
-
Diff: 132 lines (+101/-1)4 files modifieddebian/changelog (+6/-0)
debian/control (+2/-1)
debian/patches/ndctl-init-labels-Fix-label-slot-accounting-per-UEFI.patch (+92/-0)
debian/patches/series (+1/-0)
| Changed in linux (Ubuntu): | |
| status: | New → Incomplete |
apport information
| information type: | Public → Public Security |
| information type: | Public Security → Private Security |
| information type: | Private Security → Public Security |
| information type: | Public Security → Public |
| tags: | added: apport-collected bionic |
| description: | updated |
apport information
apport information
apport information
apport information
apport information
apport information
apport information
apport information
apport information
| Changed in linux (Ubuntu): | |
| status: | Incomplete → Confirmed |
| Changed in ndctl: | |
| status: | Unknown → New |
| JulietDeltaGolf (julietdeltagolf) wrote : | #12 |
In case anybody else runs into this issue, here's some things you could do :
On 4.15, you can reconfigure the namespace with ndctl by adding '--no-autolabel' to the command line. You'll still have to find a way to make the namespaces configuration persistent across boots since you're not making uses of the labels.
On 4.18, if you use a fixed kernel + fixed ndctl to setup the namespace(s), the configuration will be properly read at boot by the current linux hwe 4.18 kernel.
| Changed in ndctl: | |
| status: | New → Fix Released |
| Jerry Clement (jerry-clement) wrote : | #13 |
Missing Kernel Fix?
| JulietDeltaGolf (julietdeltagolf) wrote : | #14 |
Everything is explained in the ndctl issue 78.
| Launchpad Janitor (janitor) wrote : | #15 |
Status changed to 'Confirmed' because the bug affects multiple users.
| Changed in ndctl (Ubuntu): | |
| status: | New → Confirmed |
| torel (torehl) wrote : | #16 |
Needed --no-autolabel to get it working. New ndctl didn't help. Which is nice as I didn't have to interrupt metadata operations of the servers with nvdimm's.
# uname -ar
Linux localhost 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# ndctl create-namespace --no-autolabel --mode fsdax --map dev -e namespace0.0 -f -vvv
# ndctl create-namespace --no-autolabel --mode fsdax --map dev -e namespace1.0 -f -vvv
# mkfs.xfs -f -m reflink=0 -L PMEM0 /dev/pmem0
# mkfs.ext4 -L PMEM1 -F /dev/pmem1
# mount -o dax /dev/pmem1 /mnt/pmem1-ext4/
# mount -o dax /dev/pmem0 /mnt/pmem0-xfs
# mount | grep dax
/dev/pmem0 on /mnt/pmem0-xfs type xfs (rw,relatime,
/dev/pmem1 on /mnt/pmem1-ext4 type ext4 (rw,relatime,
Nice if this could be fixed in kernels above 4.15.0-62. Thx!
libndctl: sizeof_
This issue is fixed by:
commit 9c6aae5
Author: Dan Williams <email address hidden>
Date: Tue Jan 15 01:51:45 2019
ndctl/
Quoting from Linux kernel commit 9e694d9c18dd "libnvdimm, label: change
nvdimm_
1024 bytes label storage area. nvdimm_
slots while the area is only big enough for 2 slots.
Change nvdimm_
according to UEFI 2.7 spec.
Without this fix attempts to initialize labels on a small (1K) label
area results in the following:
libndctl: sizeof_
libndctl: sizeof_
Based on an original patch by Toshi Kani
Fixes: bdaec95463ca ("ndctl: introduce ndctl_dimm_
Reported-by: Sujith Pandel <email address hidden>
Link: https:/
Signed-off-by: Dan Williams <email address hidden>
Signed-off-by: Vishal Verma <email address hidden>
Upstream.
| Changed in linux (Ubuntu): | |
| status: | Confirmed → Invalid |
| Changed in ndctl (Ubuntu): | |
| importance: | Undecided → Medium |
| assignee: | nobody → Rafael David Tinoco (rafaeldtinoco) |
It fixes an issue introduced by :
Patch: bdaec95463ca
In version v58
And the fix (9c6aae5) was introduced in v64.
$ rmadison ndctl
ndctl | 61.2-0ubuntu1~
ndctl | 63-1.3 | disco/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
ndctl | 65-1 | eoan/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
ndctl | 67-1 | focal/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x
Bionic and Disco are affected.
| no longer affects: | linux (Ubuntu Bionic) |
| no longer affects: | linux (Ubuntu Disco) |
| no longer affects: | linux (Ubuntu Eoan) |
| Changed in ndctl (Ubuntu): | |
| status: | Confirmed → Fix Released |
| Changed in ndctl (Ubuntu Eoan): | |
| status: | New → Fix Released |
| Changed in ndctl (Ubuntu Disco): | |
| status: | New → Confirmed |
| Changed in ndctl (Ubuntu Bionic): | |
| status: | New → Confirmed |
| Changed in ndctl (Ubuntu): | |
| assignee: | Rafael David Tinoco (rafaeldtinoco) → nobody |
| Changed in ndctl (Ubuntu Bionic): | |
| assignee: | nobody → Rafael David Tinoco (rafaeldtinoco) |
| Changed in ndctl (Ubuntu Disco): | |
| assignee: | nobody → Rafael David Tinoco (rafaeldtinoco) |
| Changed in ndctl (Ubuntu): | |
| importance: | Medium → Undecided |
| Changed in ndctl (Ubuntu Disco): | |
| importance: | Undecided → Medium |
| Changed in ndctl (Ubuntu Bionic): | |
| importance: | Undecided → Medium |
A PPA containing the fixes (Disco and Bionic) will be available here:
https:/
Hello Torel and Juliet,
If possible, could you please test the PPA above with either Bionic and/or Disco and make sure the packages fix the issue ? I'm preparing bdctl to be in "main" pocket for 20.04:
https:/
and having this case fix would help to get traction in the MIR (main inclusion request).
Thank you very much for reporting this and contributing to Ubuntu.
| Changed in ndctl (Ubuntu Bionic): | |
| status: | Confirmed → In Progress |
| Changed in ndctl (Ubuntu Disco): | |
| status: | Confirmed → In Progress |
The kernel issue pointed at the github was solved by:
commit cbaee3aa0e23
Author: Dan Williams <email address hidden>
Date: Thu Feb 7 20:56:50 2019
acpi/nfit: Fix bus command validation
BugLink: https:/
in kernel: Ubuntu-5.0.0-9.10 (Disco)
AND
commit 3b8d3410b3d8
Author: Dan Williams <email address hidden>
Date: Thu Feb 7 20:56:50 2019
acpi/nfit: Fix bus command validation
BugLink: https:/
commit ebe9f6f19d80d89
in kernel: Ubuntu-4.15.0-59.66 (Bionic)
So kernel part is good.... (or kind of.. actually it is missing 2 fixes on top of that):
----
TODO: KERNEL TEAM SHOULD BACKPORT THIS TO BIONIC AND DISCO:
commit ebe9f6f19d80
Author: Dan Williams <email address hidden>
Date: Thu Feb 7 20:56:50 2019
acpi/nfit: Fix bus command validation
Fixes: 11189c1089da ("acpi/nfit: Fix command-supported detection")
commit 0171b6b78131
Author: Dan Williams <email address hidden>
Date: Sun Feb 3 17:17:27 2019
acpi/nfit: Require opt-in for read-only label configurations
Fixes: 11189c1089da ("acpi/nfit: Fix command-supported detection")
TO FIX THE ISSUE'S FIX
-----
TODO: FEEDBACK ABOUT PROPOSED USERLAND FIX
and the userland fix was, indeed, the ndctl patch I did a SRU for.
UNFORTUNATELY I can't reproduce the issue with QEMU. I have tried different label sizes in multiple attempts and wasn't able to cause the labeling error (making sure it was fixed). With that it is hard for me to guarantee the fix is good and I'll depend on a feedback from reporter.
There is a PPA in comment #19 containing the fix. I'll flag this case as incomplete while I have no feedback, as we wouldn't be able to verify the fix in order for it to land -updates archive.
Nevertheless, documentation here might be important to others, in order to understand the errors.
| Changed in linux (Ubuntu): | |
| status: | Invalid → Fix Released |
| Changed in ndctl (Ubuntu Bionic): | |
| assignee: | Rafael David Tinoco (rafaeldtinoco) → nobody |
| Changed in ndctl (Ubuntu Disco): | |
| assignee: | Rafael David Tinoco (rafaeldtinoco) → nobody |
| Changed in ndctl (Ubuntu Bionic): | |
| status: | In Progress → Incomplete |
| Changed in ndctl (Ubuntu Disco): | |
| status: | In Progress → Incomplete |
TODOs in this bug:
- End user must confirm the issue is still reproducible and test PPA package
- Mark the case as Confirmed again (instead of Incomplete)
- From that point, the SRU would continue (merge requests already present in this bug)
- Patches would be sent out to kernel team for inclusion in the next SRU
| torel (torehl) wrote : | #23 |
Tested latest kernel 4.15.0-72-generic on bionic.
root@srl-mds1:~# dpkg -l | grep -i ndctl
ii libndctl6 61.2-0ubuntu1~
ii ndctl 61.2-0ubuntu1~
root@srl-mds1:~# ndctl create-namespace -e "namespace0.0" -m fsdax -f -vvv
enable_labels:945: region0: failed to initialize labels
namespace_
failed to reconfigure namespace: No such device
root@srl-mds1:~#
But if I add --no-autolabel it works
root@srl-mds1:~# ndctl create-namespace --no-autolabel --mode fsdax --map dev -e namespace0.0 -f -vvv
{
"dev"
"mode":"fsdax",
"map":"dev",
"size":"189.00 GiB (202.94 GB)",
"uuid"
"sector_
"blockdev"
"numa_node":0
}
root@srl-mds1:~# ndctl create-namespace --no-autolabel --mode fsdax --map dev -e namespace1.0 -f -vvv
{
"dev"
"mode":"fsdax",
"map":"dev",
"size":"189.00 GiB (202.94 GB)",
"uuid"
"sector_
"blockdev"
"numa_node":1
}
Bionic LTS kernel still needs kernel fix.
| Changed in ndctl (Ubuntu Disco): | |
| status: | Incomplete → Won't Fix |


This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 1811785
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.