cp file from external HDD to NFS on Pi 3B causes ubuntu 19.10 and 20.04 kernel to hang

[David Woodgate]

Copying a large file using cp on my Raspberry Pi 3B (not 3B+) from an externally-powered USB hard disk (Seagate Expansion 2TB) to an NFS-mounted filesystem across Ethernet causes the kernel of releases 19.10 and 20.04 to hang/freeze.

This happens every time the copy is attempted, although the exact duration from the start of the copy to the hang varies from a few seconds to a couple of minutes. The kernel does not recover even after long waits (30mins+), and a power reset (pulling the plug) is required. (Although I admit I haven't tried yet simply disconnecting the USB cable of the HDD during a hang. Update: I've now tried that - the system doesn't recover after pulling the USB cable out.)

Likewise, messages are erratic and I suspect are only symptoms, notably watchdog BUG soft lockup messages and RCU stall messages on different CPU cores for different processes/threads. There are no hardware messages that I've seen recorded in log files (or appearing on the console) matching the incidents.

When a hang happen, network connectivity is lost - ssh sessions fail, and the file copy stops (even when the cp is run on the console). Pings to the system from other hosts on the same Ethernet LAN (and switch) fail with destination host unreachable.

Although I have had a swapfile configured on the SD card, the hangs continue to happen doing the cp after "swapoff -a". I've also tried playing with taskset affinities and snappiness to no obvious effect - the hangs still keep happening.

Copying from the external HDD to the SD card is successful, as is copying from the SD card to the NFS.

Raspbian 4.19.0-8 (armmp-lpae) does not have this problem; it successfully copies the file from HDD to NFS across Ethernet without stalling.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.4.0-1011-raspi 5.4.0-1011.11
ProcVersionSignature: Ubuntu 5.4.0-1011.11-raspi 5.4.34
Uname: Linux 5.4.0-1011-raspi aarch64
ApportVersion: 2.20.11-0ubuntu27
Architecture: arm64
CasperMD5CheckResult: skip
Date: Fri Jun 5 07:37:44 2020
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: linux-raspi
UpgradeStatus: Upgraded to focal on 2020-05-21 (14 days ago)

[Francis Ginther (fginther)]

I hit this during scp of a large file (> 2GB) from a usb thumbdrive to a local host on my network:

scp /media/usbstick/large-file.tar.gz my@myserver:/tmp/

There were no messages to the vga console. I tried it 3 times and it hung twice. Had to pull power to get the system back.

[Juerg Haefliger (juergh)]

I cannot reproduce this. Copied a 3GB file from a USB attached HD to an NFS mount. 10 times in a loop, works just fine.

$ uname -a
Linux rpi-3b-rev1d2-62e2 5.4.0-1011-raspi #11-Ubuntu SMP Fri May 8 07:43:33 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

$ cat /proc/device-tree/model
Raspberry Pi 3 Model B Rev 1.2

[Brian Murray (brian-murray)]

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[krot (krot-mstu)]

I have the same issue. I just copy files via network(via cable) with rsync from local PC to USB HDD (USB HDD is attached to raspberry). In 30-90 minutes Raspberry hangs every time. I made this test with 2 different hardware sets (Raspberry + SD card + external USB HDD). I had the same issue with different hardware.
I've added photo to attach.

[Juerg Haefliger (juergh)]

@ub4pi @fginther @krot-mstu Is this still an issue with the latest kernel? Can you please retest?

[Francis Ginther (fginther)]

@juergh, I retested this with focal 5.4.0-1028-raspi and was unable to reproduce.

[Juerg Haefliger (juergh)]

Closing this as it's not reproduceable. Please open a new ticket if you run into the problem again.