ubuntu_unionmount_ovlfs failed on Google n2d-standard-2 with kernel tainted

[Po-Hsu Lin]

Issue found on:
  * F-gke 5.4.0-1079.85 (sur-20220711)
  * F-gcp 5.4.0-1084.92 (sru-20220620)
  * F-gcp-fips 5.4.0-1081.88+fips1 (sru-20220620)

It seems a bit flaky, not happen with every kernel on this instance.

Test failed with:
  ***
  *** ./run --ov --xino open-plain
  ***
  TEST open-plain.py:10: Open O_RDONLY
   ./run --open-file /mnt/a/foo100 -r -R :xxx:yyy:zzz
   ./run --open-file /mnt/a/foo100 -r -R :xxx:yyy:zzz
  TEST open-plain.py:18: Open O_WRONLY
   ./run --open-file /mnt/a/foo101 -w -W q
   ./run --open-file /mnt/a/foo101 -r -R qxxx:yyy:zzz
   ./run --open-file /mnt/a/foo101 -w -W p
   ./run --open-file /mnt/a/foo101 -r -R pxxx:yyy:zzz
  TEST open-plain.py:28: Open O_APPEND|O_WRONLY
   ./run --open-file /mnt/a/foo102 -a -W q
   ./run --open-file /mnt/a/foo102 -r -R :xxx:yyy:zzzq
   ./run --open-file /mnt/a/foo102 -a -W p
   ./run --open-file /mnt/a/foo102 -r -R :xxx:yyy:zzzqp
  TEST open-plain.py:38: Open O_RDWR
   ./run --open-file /mnt/a/foo103 -r -w -W q
   ./run --open-file /mnt/a/foo103 -r -R qxxx:yyy:zzz
   ./run --open-file /mnt/a/foo103 -r -w -W p
   ./run --open-file /mnt/a/foo103 -r -R pxxx:yyy:zzz
  TEST open-plain.py:48: Open O_APPEND|O_RDWR
   ./run --open-file /mnt/a/foo104 -r -a -W q
   ./run --open-file /mnt/a/foo104 -r -R :xxx:yyy:zzzq
   ./run --open-file /mnt/a/foo104 -r -a -W p
   ./run --open-file /mnt/a/foo104 -r -R :xxx:yyy:zzzqp
  stderr:
  Traceback (most recent call last):
    File "./run", line 369, in
      check_not_tainted()
    File "/home/ubuntu/autotest/client/tmp/ubuntu_unionmount_ovlfs/src/unionmount-testsuite/tool_box.py", line 49, in check_not_tainted
      raise RuntimeError("TAINTED " + current_taint + " -> ", taint)
  RuntimeError: ('TAINTED 0\n -> ', '512\n')
     END ERROR ubuntu_unionmount_ovlfs.overlayfs ubuntu_unionmount_ovlfs.overlayfs timestamp=1658709711 localtime=Jul 25 00:41:51

Flag 512 here means kernel issued warning:
 [ 273.676972] nvme 0000:00:04.0: overflow 0x0008000203502000+262144 of DMA mask ffffffffffffffff bus mask 0
 [ 273.687031] ------------[ cut here ]------------
 [ 273.687036] WARNING: CPU: 1 PID: 8 at kernel/dma/direct.c:35 report_addr+0x37/0x60
 [ 273.687037] Modules linked in: nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua input_leds serio_raw sch_fq_codel drm ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper nvme psmouse nvme_core gve
 [ 273.687052] CPU: 1 PID: 8 Comm: kworker/u4:0 Not tainted 5.4.0-1079-gke #85-Ubuntu
 [ 273.687052] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/29/2022
 [ 273.687056] Workqueue: writeback wb_workfn (flush-259:0)
 [ 273.687058] RIP: 0010:report_addr+0x37/0x60
 [ 273.687060] Code: 48 8b 87 30 02 00 00 48 89 75 f8 48 85 c0 74 2a 4c 8b 00 b8 fe ff ff ff 49 39 c0 76 11 80 3d f7 aa b2 01 00 0f 84 5a 78 96 00 <0f> 0b c9 c3 48 83 bf 40 02 00 00 00 74 f2 eb e3 80 3d db aa b2 01
 [ 273.687060] RSP: 0018:ffffa6e640057548 EFLAGS: 00010246
 [ 273.687061] RAX: 0000000000000000 RBX: ffff9809fb21a000 RCX: 0000000000000000
 [ 273.687062] RDX: ffff980a37b2c780 RSI: ffff980a37b1c8c8 RDI: ffff980a37b1c8c8
 [ 273.687062] RBP: ffffa6e640057550 R08: ffff980a37b1c8c8 R09: 0000000000000001
 [ 273.687062] R10: ffffffffb919e228 R11: ffffa6e640057290 R12: 0000000000000000
 [ 273.687063] R13: 0000000000040000 R14: 0000000000000001 R15: 0000000000000100
 [ 273.687066] FS: 0000000000000000(0000) GS:ffff980a37b00000(0000) knlGS:0000000000000000
 [ 273.687067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 [ 273.687067] CR2: 00007fca1531f4b0 CR3: 0008000216b74000 CR4: 0000000000340ee0
 [ 273.687070] Call Trace:
 [ 273.687073] dma_direct_map_page+0xdc/0xf0
 [ 273.687075] dma_direct_map_sg+0x68/0xb0
 [ 273.687077] nvme_map_data+0x13b/0x9a0 [nvme]
 [ 273.687080] ? recalibrate_cpu_khz+0x10/0x10
 [ 273.687082] ? blk_add_timer+0x5b/0xa0
 [ 273.687084] nvme_queue_rq+0xa2/0x1d0 [nvme]
 [ 273.687085] ? crypto_shash_update+0x1e/0x30
 [ 273.687087] __blk_mq_try_issue_directly+0x11a/0x1c0
 [ 273.687088] blk_mq_request_issue_directly+0x4b/0xe0
 [ 273.687090] ? common_interrupt+0xa/0xf
 [ 273.687091] ? recalibrate_cpu_khz+0x10/0x10
 [ 273.687092] blk_mq_try_issue_list_directly+0x46/0xb0
 [ 273.687093] blk_mq_sched_insert_requests+0xae/0x100
 [ 273.687094] blk_mq_flush_plug_list+0x1e8/0x290
 [ 273.687094] ? __blk_mq_get_tag+0x28/0x80
 [ 273.687096] blk_flush_plug_list+0xe3/0x110
 [ 273.687096] blk_mq_make_request+0x24f/0x5b0
 [ 273.687098] generic_make_request+0xcf/0x320
 [ 273.687099] submit_bio+0x48/0x1d0
 [ 273.687100] ext4_io_submit+0x4d/0x60
 [ 273.687102] ext4_writepages+0x673/0x950
 [ 273.687103] ? nvme_submit_cmd+0xff/0x110 [nvme]
 [ 273.687106] do_writepages+0x43/0xd0
 [ 273.687106] ? mpage_map_and_submit_extent+0x4b0/0x4b0
 [ 273.687107] ? do_writepages+0x43/0xd0
 [ 273.687108] __writeback_single_inode+0x40/0x300
 [ 273.687110] writeback_sb_inodes+0x22d/0x4b0
 [ 273.687111] wb_writeback+0xd1/0x2f0
 [ 273.687112] wb_workfn+0xd3/0x4f0
 [ 273.687113] ? __switch_to_asm+0x40/0x70
 [ 273.687114] ? __switch_to_asm+0x34/0x70
 [ 273.687114] ? __switch_to_asm+0x40/0x70
 [ 273.687116] ? finish_task_switch+0x72/0x230
 [ 273.687118] ? __schedule+0x2eb/0x740
 [ 273.687120] process_one_work+0x1eb/0x3b0
 [ 273.687121] worker_thread+0x4d/0x400
 [ 273.687122] kthread+0x104/0x140
 [ 273.687123] ? process_one_work+0x3b0/0x3b0
 [ 273.687124] ? kthread_park+0x90/0x90
 [ 273.687125] ret_from_fork+0x35/0x40
 [ 273.687126] ---[ end trace 612d400c38106d5d ]---
 [ 274.845534] overlayfs: "xino" feature enabled using 1 upper inode bits.

[Po-Hsu Lin (cypressyew)]

One interesting fact is that if you try to run this test locally on the target instance, it won't trigger this warning.

However if you run it remotely from our jenkins, it will trigger this warning message.