dkms modules are not built anymore for linux-kvm

[Roxana Nicolescu]

SRU Justification

[Impact]

Some dkms modules are not built anymore when a retry is triggred from adt matrix page.
It was noticed because these modules previously failed during build (because a config options is disabled) but now they pass (artificial pass).

Affected modules:
1. digimend-dkms
2. oss4
3. rtl8812au
4. rtl8812ce

Looking at digimend-dkms for jammy:linux-kvm, first time the test was correct (it failed) was when the new kernel was just pushed to proposed and adt testing was triggered. When a manual retry was done, it passed because build was ignored.
Looking at the artifacts (attached for both cases), ADT_TEST_TRIGGERS differs:
1. when build is triggered, ADT_TEST_TRIGGERS=linux-meta-kvm/5.15.0.1030.26 linux-kvm/5.15.0-1030.35 linux-signed-kvm/5.15.0-1030.35
logs https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/d/digimend-dkms/20230213_171044_032fa@/log.gz

2. when build is not triggered, ADT_TEST_TRIGGERS=linux-meta-kvm/5.15.0.1030.26
logs:
https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/d/digimend-dkms/20230301_110606_17e6d@/log.gz

Last log line present in both cases is:
"I: Testing binary package $pkg".

Looking at dkms: debian/scripts/dkms-autopkgtest

After that line and the line where build is triggered the following code:
    dkms_pkg=$(bash -c ". $dkms_conf > /dev/null; echo \$PACKAGE_NAME" 2>/dev/null)
    dkms_ver=$(bash -c ". $dkms_conf > /dev/null; echo \$PACKAGE_VERSION" 2>/dev/null)

    for k in /lib/modules/*/build
    do
        test -d "$k" || continue
        kver="${k%/build}"
        kver="${kver#/lib/modules/}"

        # If any linux-meta is in triggers, only test abistems that
        # match triggers otherwise continue. This helps integration
        # with adt-matrix which specifically requests test results
        # against each individual linux-meta and tracks unique results
        # per kernel abi.
        abistem=$(echo $kver | sed 's/-[a-z]*$//')
  ## abistem is 5.15.0-130
        case "${ADT_TEST_TRIGGERS-}" in
            *linux-meta*)
                case "$ADT_TEST_TRIGGERS" in
                    *"$abistem"*)
                        ;;
                    *)
                        continue
                        ;;
                esac
        esac

        echo "I: Trying to build $dkms_pkg/$dkms_ver for $kver"

What happens here is that it never gets passed the switch case and build is not triggered when ADT_TEST_TRIGGERS consists of linux-meta only, because abistem=<kernel_version>-<abi_version>, but meta version=<kernel_version>.<abi_version>. Notice the dot instead of the dash.

This justifies why the module is built when ADT_TEST_TRIGGERS=linux-meta-kvm/5.15.0.1030.26 linux-kvm/5.15.0-1030.35 linux-signed-kvm/5.15.0-1030.35, but not when ADT_TEST_TRIGGERS=linux-meta-kvm/5.15.0.1030.26

[Fix]
Add an extra check for meta_version in that switch case statement.

[Testcase]
1. Tested locally with ADT_TEST_TRIGGERS=linux-meta.. before fix
I: linux-headers-virtual
I: Testing binary package digimend-dkms
autopkgtest [11:14:45]: test dkms-autopkgtest: -----------------------]
autopkgtest [11:14:46]: test dkms-autopkgtest: - - - - - - - - - - results - - - - - - - - - -
dkms-autopkgtest PASS
autopkgtest [11:14:47]: @@@@@@@@@@@@@@@@@@@@ summary
dkms-autopkgtest PASS
qemu-system-x86_64: terminating on signal 15 from pid 15111 (/usr/bin/python3)
2. Tested locally with ADT_TEST_TRIGGERS=linux-meta.. with the fix
I: Testing binary package digimend-dkms
I: Trying to build digimend/10 for 5.15.0-1029-kvm
Creating symlink /var/lib/dkms/digimend/10/source -> /usr/src/digimend-10

Kernel preparation unnecessary for this kernel. Skipping...

Building module:
cleaning build area...
make -j1 KERNELRELEASE=5.15.0-1029-kvm KVERSION=5.15.0-1029-kvm...(bad exit status: 2)
ERROR (dkms apport): kernel package linux-headers-5.15.0-1029-kvm is not supported
Error! Bad return status for module build on kernel: 5.15.0-1029-kvm (x86_64)
Consult /var/lib/dkms/digimend/10/build/make.log for more information.
E: digimend/10 failed to build for 5.15.0-1029-kvm
========== /var/lib/dkms/digimend/10/build/make.log ==========
DKMS make.log for digimend-10 for kernel 5.15.0-1029-kvm (x86_64)
Wed Mar 15 11:20:05 CET 2023
make -C /lib/modules/5.15.0-1029-kvm/build M=/var/lib/dkms/digimend/10/build modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.0-1029-kvm'
  CC [M] /var/lib/dkms/digimend/10/build/hid-kye.o
  CC [M] /var/lib/dkms/digimend/10/build/hid-uclogic-core.o
  CC [M] /var/lib/dkms/digimend/10/build/hid-uclogic-rdesc.o
  CC [M] /var/lib/dkms/digimend/10/build/hid-uclogic-params.o
/var/lib/dkms/digimend/10/build/hid-uclogic-params.c: In function ‘uclogic_params_init’:
/var/lib/dkms/digimend/10/build/hid-uclogic-params.c:1147:20: warning: this statement may fall through [-Wimplicit-fallthrough]
 1147 | if (bNumInterfaces != 3) {
      | ^
/var/lib/dkms/digimend/10/build/hid-uclogic-params.c:1165:9: note: here
 1165 | case VID_PID(USB_VENDOR_ID_HUION,
      | ^~~~
  LD [M] /var/lib/dkms/digimend/10/build/hid-uclogic.o
  CC [M] /var/lib/dkms/digimend/10/build/hid-polostar.o
  CC [M] /var/lib/dkms/digimend/10/build/hid-viewsonic.o
  MODPOST /var/lib/dkms/digimend/10/build/Module.symvers
ERROR: modpost: "usb_string" [/var/lib/dkms/digimend/10/build/hid-uclogic.ko] undefined!
ERROR: modpost: "usb_control_msg" [/var/lib/dkms/digimend/10/build/hid-uclogic.ko] undefined!
ERROR: modpost: "usb_hid_driver" [/var/lib/dkms/digimend/10/build/hid-uclogic.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:133: /var/lib/dkms/digimend/10/build/Module.symvers] Error 1
make[2]: *** Deleting file '/var/lib/dkms/digimend/10/build/Module.symvers'
make[1]: *** [Makefile:1821: modules] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.15.0-1029-kvm'
make: *** [Makefile:25: modules] Error 2
====================
./digimend/10/build/make.log
autopkgtest [11:20:08]: test dkms-autopkgtest: -----------------------]
autopkgtest [11:20:09]: test dkms-autopkgtest: - - - - - - - - - - results - - - - - - - - - -
dkms-autopkgtest FAIL non-zero exit status 1
autopkgtest [11:20:10]: @@@@@@@@@@@@@@@@@@@@ summary
dkms-autopkgtest FAIL non-zero exit status 1
qemu-system-x86_64: terminating on signal 15 from pid 16171 (/usr/bin/python3)

[Regression potential]
We will see now failures instead of passes, but for the modules listed above, fixes are in the making.

[Note]
Not sure why ADT_TEST_TRIGGERS is different for manual retries vs when test is automated triggered when the new kernel is in proposed.

[Paride Legovini (paride)]

Hi Roxana, you linked the same log twice, so I can't see the "Logs where test passes". Having a link to the unexpected pass log will help us understand the issue. At the moment it is not clear to me the report is about the dep8 tests, the autpkgtest package or the autopkgtest infrastructure. Thanks!

[Roxana Nicolescu (roxanan)]

Hi Paride. My bad, I updated the description.

> At the moment it is not clear to me the report is about the dep8 tests, the autpkgtest package or the autopkgtest infrastructure.
Yeah, that's something I am also struggling with. I checked the repos, tried to reproduce it locally but no luck so far. Any hint would be super helpful.

[Paride Legovini (paride)]

I didn't dig into the specifics, but the error the we see in the passed test log comes from this this line from /usr/lib/dkms/dkms-autopkgtest:

# Try and remove dkms to spot packages which miss a dkms dependency
echo "I: Checking for missing dkms dependency by trying to deinstall dkms"
dpkg --remove dkms || true

So failures to execute that command are explicitly ignored (|| true). I don't see an issue with autopkgtest (package or infra) here.

[Roxana Nicolescu (roxanan)]

Hi Paride. The error you mention is in both logs and I don't think it's the issue here.

By accident, I discovered that when the new kernel is submitted to proposed and adt test is triggered, the result is as expected.
But then, is a user retries manually the test (like I did) then the status is PASS (supperficial).

After looking at the artifacts between the two triggers (I attached them but you can also check them here https://autopkgtest.ubuntu.com/packages/d/digimend-dkms/jammy/amd64 -- version 5.15.0.1030), I noticed that ADT_TEST_TRIGGERS is different.
1. When the build is done, ADT_TEST_TRIGGERS is `linux-meta-kvm/5.15.0.1030.26 linux-kvm/5.15.0-1030.35 linux-signed-kvm/5.15.0-1030.35`
2. When the build is ignored, ADT_TEST_TRIGGERS is `linux-meta-kvm/5.15.0.1030.26`.

If I check the code in /usr/lib/dkms/dkms-autopkgtest, right after `Testing binary package digimend-dkms`, the last line we see in both logs:

    echo "I: Testing binary package $pkg"

    dkms_pkg=$(bash -c ". $dkms_conf > /dev/null; echo \$PACKAGE_NAME" 2>/dev/null)
    dkms_ver=$(bash -c ". $dkms_conf > /dev/null; echo \$PACKAGE_VERSION" 2>/dev/null)

    for k in /lib/modules/*/build
    do
        test -d "$k" || continue
        kver="${k%/build}"
        kver="${kver#/lib/modules/}"

        # If any linux-meta is in triggers, only test abistems that
        # match triggers otherwise continue. This helps integration
        # with adt-matrix which specifically requests test results
        # against each individual linux-meta and tracks unique results
        # per kernel abi.
        abistem=$(echo $kver | sed 's/-[a-z]*$//')
  ## abistem is 5.15.0-130
        case "${ADT_TEST_TRIGGERS-}" in
            *linux-meta*)
                case "$ADT_TEST_TRIGGERS" in
                    *"$abistem"*)
                        ;;
                    *)
                        continue
                        ;;
                esac
        esac

I think the issue is in the switch case in the for loop. When build is ignored, it never gets passed that because
abistem=$(echo $kver | sed 's/-[a-z]*$//') is 5.15.0-1030. Notice the difference, it is not 5.15.0.1030 therefore when ADT_TEST_TRIGGERS is only linux-meta-kvm/5.15.0.1030.26 it won't match.

Do you know why ADT_TEST_TRIGGERS is different when a manual trigger is done?

[Roxana Nicolescu (roxanan)]

Attaching the artifacts for failure, for some reason I could not attach two archives

[Roxana Nicolescu (roxanan)]

I created a merge request to push the fix to dkms when ADT_TRIGGERS=linux-meta only.
I will update the description with details from my previous comment.