Slower NVMe I/O when using threads on X299 chipset

[igor]

A simple C++ app with a read loop from an NVMe SSD while using threads or not runs with the same speed on a Z590 i7-11700K PC, even older chipsets like Z270, 200GB/min. But on a PC with X299 chipset and i9-10900X CPU runs 150GB/min when using threads and 200GB/min when not using threads. Tested on Lubuntu 20.04.3, kernel 5.14.0-1005-oem. The same has been observed on other X299 motherboards.

Below is the test app that can be compiled with or without #define USE_THREAD

//
// A simple application to test reading speed
// from an SSD with threads and without threads
//
// To build:
// g++ -std=c++11 -pthread -o nvmetest nvmetest.cpp
//
// To run:
// sudo ./nvmetest /dev/nvme0n1
//

#include <string>
#include <iostream>
#include <thread>
#include <future>

#include <fcntl.h>
#include <linux/types.h>
#include <unistd.h>
#include <sys/time.h>

using namespace std;

//#define USE_THREAD

void Read(int fd, void *buf, ssize_t chunkSize, promise<ssize_t> && result)
{
    result.set_value(read(fd, buf, chunkSize));
}

int main(int argc, char *argv[])
{
    if (argc < 2)
    {
        cerr << "Usage: sudo ./nvmetest /dev/nvme0n1";
        return 1;
    }

#ifdef USE_THREAD
    cout << "Using thread" << endl;
#else
    cout << "NOT using thread" << endl;
#endif

    // open the device for reading
    int fd = ::open(argv[1], O_RDONLY);
    if (fd <= 0)
    {
        cerr << "Failed to open " << argv[1] << endl;
        exit(1);
    }

    // read 32MB at a time
    const int chunkSize = 32*1024*1024;
    void *buf = malloc(chunkSize);

    int64_t readSpeedMBPerMin = 0, sectorsRead = 0, secPassed = 0, lastSecPassed = 0;

    struct timeval tp;
    gettimeofday(&tp, NULL);
    int64_t startTime = tp.tv_sec + tp.tv_usec / 1000000;

    while (true)
    {
        ssize_t result;
#ifdef USE_THREAD
        promise<ssize_t> res;
        auto f = res.get_future();
        thread rd(Read, fd, buf, chunkSize, move(res));
        rd.join();
        result = f.get();
#else
        result = read(fd, buf, chunkSize);
#endif
        if (result != chunkSize)
            break;

        sectorsRead += chunkSize/512;

        gettimeofday(&tp, NULL);
        int64_t currentTime = tp.tv_sec + tp.tv_usec / 1000000;

        secPassed = currentTime - startTime;
        if (secPassed - lastSecPassed > 0)
        {
            lastSecPassed = secPassed;
            readSpeedMBPerMin = (sectorsRead*60) / (secPassed*2000);
            cout << "Read=" << sectorsRead/2000000 << "GB" <<
                    " seconds=" << secPassed <<
                    " speed=" << readSpeedMBPerMin/1000 << "GB/min" << endl;
        }
    }

    free(buf);
    close(fd);
    return 0;
}

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Libera.chat.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1995016/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]