Comment 14 for bug 1892163

Thanks, You-Sheng and Hui, I would suggest (and maybe that is what you meant) to use dkms (remove or uninstall for the 5.4 kernel version) to drop the built dkms module from the 5.4 kernel as a first test. That way it is available for any older kernel as long as they are kept.
If this works and it is unlikely to have to go back to 5.3, then it is better to completely remove the backport driver. Otherwise it gets rebuild/installed for every kernel update.

I would also take back (at least for now) the wont-fix status for the dkms package. With the 5.4 kernel being the latest HWE kernel for Bionic, we should see whether we can make it not compiling for 5.4.