[Impact]
Description of problem:
Observed device_dax related probe errors in dmesg when HBM CPU is set to flat mode. Duplicate device_dax ids were created and hence probing is failing.
How reproducible:
Frequently
Version-Release
Release: 22.04.2, 22.10
[Test Case]
Steps to Reproduce:
1. Set HBM cpu to flat mode in memory settings in BIOS.
2. Boot to the OS.
3. Perform OS warm boot cycle test.
4. Observe the dax2.0/dax3.0/dax4.0/dax5.0 probe error.
Actual results:
Observed device_dax related errors in dmesg, device Dax is creating dummy/duplicate devices and probe failing.
Expected results:
Dummy/duplicate devices should not create.
[Fix]
Upstream Fix
https://lore.kernel<email address hidden>/T/
[Where problems could occur]
[Other Info]
https://code.launchpad.net/~mreed8855/ubuntu/+source/linux/+git/jammy/+ref/lp_2028158_device_dax_2
Additional info:
SUT is having 2*32C HBM cpus. Eligible system-ram mode change devices should be only 2[dax0.0, dax1.0], but under "daxctl list -u" is showing 1st time 4 devices [dax0.0, 1.0, 2.0, 3.0], 2 is "state":"disabled" and 2 more devices is "mode":"devdax" which are actuall devadax to system-ram convertible devices. After reconfigure-device dax0.0, dax1.0 when you list the devices couple of more dummy/dumplicate devices are creating with "state":"disabled"[Ex: dax4.0, 5.0 etc..].
root@ubuntu:/home/ubuntu# daxctl list -u
[
{
"chardev":"dax1.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":3,
"align":2097152,
"mode":"devdax"---------------> HBM CPU 1, This we can change the devdax to
system-ram
},
{
"chardev":"dax2.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":2, --------------------> Duplicate device
"align":2097152,
"mode":"devdax",
"state":"disabled"
},
{
"chardev":"dax3.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":3, --------------------> Duplicate device
"align":2097152,
"mode":"devdax",
"state":"disabled"
},
{
"chardev":"dax0.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":2,
"align":2097152,
"mode":"devdax" ---------------> HBM CPU 1, This we can change the devdax to
system-ram
}
]
root@ubuntu:/home/ubuntu# dmesg | grep -i error
[ 12.748884] device_dax: probe of dax2.0 failed with error -16
[ 12.748902] device_dax: probe of dax3.0 failed with error -16
After reconfig-device devdax to system-ram below are the results:
-------------------------------------------------------------------
root@ubuntu:/home/ubuntu# daxctl reconfigure-device -m system-ram dax0.0 -u
{
"chardev":"dax0.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":2,
"align":2097152,
"mode":"system-ram",
"online_memblocks":32,
"total_memblocks":32,
"movable":true
}
reconfigured 1 device
root@ubuntu:/home/ubuntu# daxctl reconfigure-device -m system-ram dax1.0 -u
{
"chardev":"dax1.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":3,
"align":2097152,
"mode":"system-ram",
"online_memblocks":32,
"total_memblocks":32,
"movable":true
}
reconfigured 1 device
root@ubuntu:/home/ubuntu# daxctl list -u
[
{
"chardev":"dax4.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":2, --------------------> Duplicate device
"align":2097152,
"mode":"devdax",
"state":"disabled"
},
{
"chardev":"dax1.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":3,
"align":2097152,
"mode":"system-ram",-----------> Converted from devdax - system-ram
"online_memblocks":32,
"total_memblocks":32,
"movable":true
},
{
"chardev":"dax5.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":3, --------------------> Duplicate device
"align":2097152,
"mode":"devdax",
"state":"disabled"
},
{
"chardev":"dax2.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":2, --------------------> Duplicate device
"align":2097152,
"mode":"devdax",
"state":"disabled"
},
{
"chardev":"dax3.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":3, --------------------> Duplicate device
"align":2097152,
"mode":"devdax",
"state":"disabled"
},
{
"chardev":"dax0.0",
"size":"64.00 GiB (68.72 GB)",
"target_node":2,
"align":2097152,
"mode":"system-ram", -----------> Converted from devdax - system-ram
"online_memblocks":32,
"total_memblocks":32,
"movable":true
}
]
root@ubuntu:/home/ubuntu# dmesg | grep -i dax
[ 12.748880] device_dax dax2.0: mapping0: 0x2080000000-0x307fffffff could not reserve range
[ 12.748884] device_dax: probe of dax2.0 failed with error -16
[ 12.748901] device_dax dax3.0: mapping0: 0x5080000000-0x607fffffff could not reserve range
[ 12.748902] device_dax: probe of dax3.0 failed with error -16
[ 812.677056] device_dax dax4.0: mapping0: 0x2080000000-0x307fffffff could not reserve range
[ 812.677070] device_dax: probe of dax4.0 failed with error -16
[ 821.092762] device_dax dax5.0: mapping0: 0x5080000000-0x607fffffff could not reserve range
[ 821.092774] device_dax: probe of dax5.0 failed with error -16
Found Upstream kernel patch for hmem duplicate dax_device creation, applied on top of ubuntu 22.04.2& 22.10 working as expected, not observed any dax related errors on dmesg and no duplicate dax_device creations.
https:/
Please consider the above patch to pull into ubuntu release.
After applied patch test results:
-------
root@ubuntu:
[
{
"chardev"
"size":"64.00 GiB (68.72 GB)",
"target_
"align"
"mode":"devdax"
},
{
"chardev"
"size":"64.00 GiB (68.72 GB)",
"target_
"align"
"mode":"devdax"
}
]
ubuntu@ubuntu:~$ lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000
0x0000000100000
Memory block size: 2G
Total online memory: 384G
Total offline memory: 0B