ubuntu_nbd_smoke_test failed on Google n2d instances (Error: Failed to setup device, check dmesg)

[Po-Hsu Lin]

Issue found on Google n2d instances with Focal 5.4.0-144.161
  * n2d-standard-2
  * n2d-standard-64

Passed on the other instances.

Test output:
 Running '/home/google/autotest/client/tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.sh'
 creating backing nbd image /tmp/nbd_image.img

 --------------------------------------------------------------------------------
 Image path: /tmp/nbd_image.img
 Mount point: /mnt/nbd-test-9797
 Date: Fri Feb 17 08:06:56 UTC 2023
 Host: f-l-gen-5-4-n2dstd2-u-nbd-smk-test
 Kernel: 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023
 Machine: f-l-gen-5-4-n2dstd2-u-nbd-smk-test x86_64 x86_64
 CPUs online: 2
 CPUs total: 2
 Page size: 4096
 Pages avail: 1725564
 Pages total: 2036380
 Image size: 64 MB
 File size: 42 MB
 Free space:
 Filesystem Size Used Avail Use% Mounted on
 udev 3.9G 0 3.9G 0% /dev
 tmpfs 796M 976K 795M 1% /run
 /dev/nvme0n1p1 9.6G 3.8G 5.8G 40% /
 tmpfs 3.9G 0 3.9G 0% /dev/shm
 tmpfs 5.0M 0 5.0M 0% /run/lock
 tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
 /dev/loop0 56M 56M 0 100% /snap/core18/2679
 /dev/loop1 64M 64M 0 100% /snap/core20/1822
 /dev/loop2 338M 338M 0 100% /snap/google-cloud-cli/105
 /dev/loop3 92M 92M 0 100% /snap/lxd/24061
 /dev/loop4 50M 50M 0 100% /snap/snapd/17950
 /dev/nvme0n1p15 105M 6.1M 99M 6% /boot/efi
 tmpfs 796M 0 796M 0% /run/user/1005
 --------------------------------------------------------------------------------

 NBD device /dev/nbd0 created
 found nbd export
 NBD exports found:
 test
 starting client with NBD device /dev/nbd0
 Negotiation: ..size = 64MB
 Error: Failed to setup device, check dmesg

 Exiting.
 nbd-client failed to start
 unmounting /mnt/nbd-test-9797
 umount: /mnt/nbd-test-9797: no mount point specified.
 stopping client
 killing server

dmesg:
[ 3705.957723] creating backing nbd image /tmp/nbd_image.img
[ 3706.807381] NBD device /dev/nbd0 created
[ 3707.124702] found nbd export
[ 3718.135532] starting client with NBD device /dev/nbd0
[ 3718.140612] nbd: nbd0 already in use
[ 3718.148557] nbd-client failed to start
[ 3718.148615] unmounting /mnt/nbd-test-9797
[ 3719.153072] stopping client
[ 3719.155427] block nbd0: NBD_DISCONNECT
[ 3720.156592] killing server

If you run the test twice, it will pass.

dmesg of second run:
[ 440.909524] creating backing nbd image /tmp/nbd_image.img
[ 441.360914] NBD device /dev/nbd0 created
[ 441.477164] found nbd export
[ 452.482712] starting client with NBD device /dev/nbd0
[ 452.508205] creating ext4 on /dev/nbd0
[ 452.535404] mkfs on /dev/nbd0 succeeded after 0 attempt(s)
[ 452.551887] checking ext4 on /dev/nbd0
[ 452.564639] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null)
[ 452.564646] ext4 filesystem being mounted at /mnt/nbd-test-2215 supports timestamps until 2038 (0x7fffffff)
[ 452.578204] mounted on /dev/nbd0
[ 452.579956] creating large file /mnt/nbd-test-2215/largefile
[ 452.858830] removing file /mnt/nbd-test-2215/largefile
[ 452.863105] unmounting /mnt/nbd-test-2215
[ 454.026781] stopping client
[ 454.028432] block nbd0: NBD_DISCONNECT
[ 454.028474] block nbd0: Disconnected due to user request.
[ 454.028475] block nbd0: shutting down sockets
[ 455.030761] killing server

I assume the "nbd: nbd0 already in use" might be the clue.

This is not a regression. We can see this issue in the past cycles.