Serious design flaw using large external drives.

[Robert Pearson]

I am trying to recover data from an HFS+ files system on a 4TB external USB 3 drive. Copying 3TB from the 4TB HFS+ drive to a 4TB NTFS drive using MATE Terminal cp -anR from to took 2 days.

When I try to copy the same data to a 6TB NTFS USB 3 drive the results are glacial. 750GB has taken over 6 days. Obviously there is a problem of scale drive in Ubuntu MATE. I invite the developers to plug in a 12TB drive and see just how glacial (no - not glacial, more on the order of plate techtonics) slow their design is.

Robert Pearson

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-image-4.15.0-38-generic 4.15.0-38.41
ProcVersionSignature: Ubuntu 4.15.0-38.41-generic 4.15.18
Uname: Linux 4.15.0-38-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: robert 2102 F.... pulseaudio
 /dev/snd/controlC0: robert 2102 F.... pulseaudio
CurrentDesktop: MATE
Date: Sun Nov 11 23:50:13 2018
HibernationDevice: RESUME=UUID=aa95f07d-f769-44ca-8b33-0c5b3c78bdeb
InstallationDate: Installed on 2018-10-25 (18 days ago)
InstallationMedia: Ubuntu-MATE 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
MachineType: System manufacturer System Product Name
ProcFB: 0 radeondrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-38-generic root=UUID=26dc3ca5-f3ae-47eb-92b6-4f02ed5ba4f0 ro quiet splash vt.handoff=1
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-38-generic N/A
 linux-backports-modules-4.15.0-38-generic N/A
 linux-firmware 1.173.1
RfKill:
 0: phy0: Wireless LAN
  Soft blocked: no
  Hard blocked: no
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1301
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: A88XM-A
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev X.0x
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1301:bd04/01/2014:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnA88XM-A:rvrRevX.0x:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.family: To be filled by O.E.M.
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.20 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.20-rc2

[Robert Pearson (rpearsonii)]

No this problem occurred when I purchased and mounted the 6TB drive.

No I was not able to test the drive when I was using prior kernel version since I had not yet cought the drive.

I am a Linux newbie. Please tell me how to "test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.20 kernel[0].".

[Robert Pearson (rpearsonii)]

No! This problem did not start until after I had purchased the drive so any questions about behavior before I mounted the drive are not applicable.

You ask me to "test the latest upstream kernel". I am a Linux newbie and do not know how to do this. I will follow your link to see how. If it trashes my system I am about to give up on 18.04 anyway. I did try the latest LTS Cosmic, but after two days got rid of it. If the "upstream kernel" has the same bug, I need to return the drive within the 30-day period for a refund.

[Robert Pearson (rpearsonii)]

Am trying upstream kernel v4.20-rc4 (2e6e902d185027f8e3cb8b7305238f7e35d6a436).

Not easy for a newbie because your link does not include the order in which the modules have to be installed. Downloaded the low-latency (whatever that means) packages and managed to boot the new kernel. It seems less slow (glacial speeds rather than continental drift speeds). I realize copying 3TB of data will take some time (two days 4TB to 4TB as opposed to two months 4TB to 8TB). Am copying just one directory (about 100.9GB from 4TB to 8TB) to see how many days it takes. I am using the Terminal copy command since Caja is 3x slower.

Cannot yet say if fixed or not - time will tell.

Getting properties on folder in a 4TB drive is much faster than getting properties of same folder in a 8TB drive - so something is still wonky.

You say "If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'." How? I see buttons to add attachment but nothing on how to add tag.

[Robert Pearson (rpearsonii)]

The bug is in the upstream kernel and in Ubuntu MATE 18.10. Seagate refused to contact you to find out what the problem is. With no response from Ubuntu, I am returning the drive for a refund. Maybe some decade Ubuntu will support drives larger than 5TB (currently the largest from Seagate is 16TB).

[Robert Pearson (rpearsonii)]

After a month of waiting for confirmation of the bug, I have reformatted the partition as ext4. If this works (can copy 1.5TB in less than 2 days) the bug is in the NTFS support in Ubuntu. If it doesn't, I will return the drive (Seagate says it is in warranty but refuses to provide any support to Linux users) and chalk up another Ubuntu bug that is being ignored by the Ubuntu community.

[Kai-Heng Feng (kaihengfeng)]

Please try kernel parameter "usb-storage.quirks=vid:pid:u", replace vid and pid to the USB device ids.

This disables UAS mode, which makes the IO slower but possibly more reliable.

[Robert Pearson (rpearsonii)]

I am a Linux newbie. How do I do what you asked?

The question is moot for me. While waiting for more than a month for a response to my bug report, I have already found a workaround. I reformatted the drive from NTFS to ext4. This solved the problem for me but I lost the capability of a Windows machine reading my data. Given that Windows 10 installer will not re-install windows on a large drive this is not an issue with me. I have written off Windows as a bad deal with no support.

However others may want to keep compatibility with Windows. In that case the bug in the NTFS driver needs to be fixed.

Robert Pearson

[Po-Hsu Lin (cypressyew)]

For an adding kernel parameters how-to, please refer to this article:
https://wiki.ubuntu.com/Kernel/KernelBootParameters

[Robert Pearson (rpearsonii)]

I read that Paragon Software NTFS drivers will be part of the 3.5 kernel. I would like to use NTFS so that other computers can actually read my data.

The current drives I'm using are 8 TB Seagate external drives with a 2 year warranty. Their price went up to $150 when on sale. I will probably get a Seagate Exos x18 18 TB drive for 2 times the cost and 2.25 times the capacity and 2.5 times the warranty period. (Also greater MTBF and annual data usage.)

It would be nice if Ubuntu could access an 18 TB drive with an NTFS at greater than continental drift speeds.