echo blacklist jh7110_tdm > /etc/modprobe.d/blacklist-jh7110_tdm.conf
did not solve the issue. I still get oopses like:

[  201.478403] Oops [#1]
[  201.480681] Modules linked in: tls cfg80211 binfmt_misc nls_iso8859_1 cdns3 cdns_usb_common udc_core cdns3_starfive ofpart cmdlinepart spi_nor mtd axp20x_pek pwm_starfive dw_axi_dmac_platform sfctemp uio_pdrv_genirq uio drm dm_multipath efi_pstore backlight ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear motorcomm dwmac_starfive stmmac_platform stmmac nvme xhci_pci axp20x_regulator xhci_pci_renesas nvme_core dw_mmc_starfive pcs_xpcs phylink nvme_common dw_mmc_pltfm clk_starfive_jh7110_aon clk_starfive_jh7110_isp clk_starfive_jh7110_vout pinctrl_starfive_jh7110_aon dw_mmc axp20x_i2c axp20x spi_cadence_quadspi jh7110_trng phy_jh7110_usb
[  201.548071] CPU: 0 PID: 612 Comm: systemd-network Not tainted 6.5.0-9-generic #9.1-Ubuntu
[  201.556251] Hardware name: StarFive VisionFive 2 v1.3B (DT)
[  201.561824] epc : __d_lookup_rcu+0x4a/0x100
[  201.566024]  ra : lookup_fast+0x36/0xfc
[  201.569870] epc : ffffffff80315146 ra : ffffffff803042bc sp : ffffffc800ecb8b0
[  201.577091]  gp : ffffffff822602d8 tp : ffffffd8c6f6d100 t0 : ffffffc800ecba58
[  201.584320]  t1 : 0000000000000000 t2 : 0000000000000008 s0 : ffffffc800ecb8e0
[  201.591547]  s1 : ffffffc800ecb970 a0 : ffffffd8c49b8000 a1 : ffffffc800ecb980
[  201.598772]  a2 : ffffffc800ecb9b4 a3 : 0000000000000001 a4 : 3b91a71338836970
[  201.605996]  a5 : 3b91a71338836970 a6 : 0000361d00000000 a7 : 0000000000000000
[  201.613222]  s2 : 0000000000000001 s3 : ffffffd8c49b8000 s4 : fffffffffffff000
[  201.620444]  s5 : 0000000000000001 s6 : ffffffc800ecbca8 s7 : 0000000000000000
[  201.627666]  s8 : ffffffd8c705ca00 s9 : 000000000000000a s10: 7fffffffffffffff
[  201.634887]  s11: 0000003fc689f768 t3 : ffffffd8c49b8000 t4 : 000000060361d211
[  201.642109]  t5 : ffffffc800ecb9b4 t6 : ffffffd8c24b102d
[  201.647420] status: 0000000200000120 badaddr: ffffff933883696c cause: 000000000000000d
[  201.655337] [<ffffffff80315146>] __d_lookup_rcu+0x4a/0x100
[  201.660834] [<ffffffff803084cc>] walk_component+0x26/0x194
[  201.666325] [<ffffffff8030911e>] path_lookupat+0x76/0x196
[  201.671730] [<ffffffff80309ba4>] filename_lookup+0x96/0x166
[  201.677306] [<ffffffff80309cac>] kern_path+0x38/0x60
[  201.682274] [<ffffffff80be4e4c>] unix_find_other+0x50/0x36a
[  201.687850] [<ffffffff80be7306>] unix_dgram_sendmsg+0x43c/0x81a
[  201.693771] [<ffffffff80a8cc6e>] sock_sendmsg+0x4e/0x98
[  201.699004] [<ffffffff80a8cfce>] ____sys_sendmsg+0x1d8/0x20a
[  201.704664] [<ffffffff80a8f3c0>] ___sys_sendmsg+0x90/0xd6
[  201.710066] [<ffffffff80a8f9cc>] __sys_sendmsg+0x84/0xd2
[  201.715380] [<ffffffff80a8fa38>] sys_sendmsg+0x1e/0x2c
[  201.720521] [<ffffffff80ce48f0>] do_trap_ecall_u+0xd6/0x154
[  201.726099] [<ffffffff80003e30>] ret_from_exception+0x0/0x64
[  201.731773] Code: 4685 8e2a 8f32 7793 ffe7 e563 00e6 a885 639c c7b5 (a703) ffc7 
[  201.739422] Unable to handle kernel paging request at virtual address ffffffbf0099039c
[  201.739559] Unable to handle kernel paging request at virtual address 000000678895ce42
[  201.740238] ---[ end trace 0000000000000000 ]---