| 2024-01-17 14:53:28 |
R. Diez |
bug |
|
|
added bug |
| 2024-01-18 09:42:54 |
Matthew Ruffell |
bug |
|
|
added subscriber Matthew Ruffell |
| 2024-01-18 09:43:40 |
Matthew Ruffell |
bug task added |
|
linux (Ubuntu) |
|
| 2024-01-18 09:43:47 |
Matthew Ruffell |
bug task deleted |
linux-hwe-6.5 (Ubuntu) |
|
|
| 2024-01-18 09:43:54 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Mantic |
|
| 2024-01-18 09:43:54 |
Matthew Ruffell |
bug task added |
|
linux (Ubuntu Mantic) |
|
| 2024-02-02 21:14:56 |
R. Diez |
attachment added |
|
testdata.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2049634/+attachment/5744351/+files/testdata.txt |
|
| 2024-02-02 21:15:47 |
R. Diez |
attachment added |
|
testdata-back-from-server.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2049634/+attachment/5744352/+files/testdata-back-from-server.txt |
|
| 2024-02-05 04:28:11 |
Matthew Ruffell |
summary |
SMB 1 broken in kernel 6.5.0.14.14~22.04.7 |
smb1: wsize blocks of bytes followed with binary zeros on copy, destroying data |
|
| 2024-02-05 04:28:15 |
Matthew Ruffell |
linux (Ubuntu Mantic): status |
New |
In Progress |
|
| 2024-02-05 04:28:29 |
Matthew Ruffell |
nominated for series |
|
Ubuntu Noble |
|
| 2024-02-05 04:28:29 |
Matthew Ruffell |
bug task added |
|
linux (Ubuntu Noble) |
|
| 2024-02-05 04:28:34 |
Matthew Ruffell |
linux (Ubuntu Noble): status |
New |
In Progress |
|
| 2024-02-05 04:28:41 |
Matthew Ruffell |
linux (Ubuntu Mantic): importance |
Undecided |
High |
|
| 2024-02-05 04:28:42 |
Matthew Ruffell |
linux (Ubuntu Noble): importance |
Undecided |
High |
|
| 2024-02-05 04:28:44 |
Matthew Ruffell |
linux (Ubuntu Mantic): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2024-02-05 04:28:46 |
Matthew Ruffell |
linux (Ubuntu Noble): assignee |
|
Matthew Ruffell (mruffell) |
|
| 2024-02-05 04:29:14 |
Matthew Ruffell |
description |
Hi all:
I upgraded my Ubuntu yesterday and automatically got the newer Linux Kernel version 6.5.0-14-generic #14~22.04.1-Ubuntu. Previously, I was running kernel version 6.2 .
I still have a legacy system on the network using SMB protocol version 1.0.
With the new kernel version, copying files does not work reliably anymore. Some random byte blocks in the destination files are overwritten with binary zeros. It happens quite often.
This is not the first time the Linux guys temporarily break SMB protocol version 1.0, see for example this bug report of mine:
https://bugs.launchpad.net/ubuntu/+bug/2033732
I checked package linux-image-generic-hwe-22.04 with Synaptic, and now it lists just 2 versions:
6.5.0.14.14~22.04.7 (jammy-updates)
5.15.0.25.27 (jammy)
Is there a way to go back to the latest 6.2 kernel version?
How do I prevent Ubuntu from upgrading to 6.5 next time around? I have searched the Internet, but I haven't
found yet a usable answer. I don't want to go back all the way to Kernel 5.15 if I can avoid it.
Thanks in advance,
rdiez |
[Impact]
Upon installing the 6.5 HWE kernel on Jammy, users with a custom wsize set will see data destruction when copying files from their systems onto a cifs smb 1.0 mount.
wsize defaults to 65535 bytes, but when set to smaller values, like 16850, users will see blocks of 16850 bytes copied over, followed by 3900 binary zeros, followed by the next block of data followed by more binary zeros.
A workaround is to increase wsize, but this only works for small files, as any files larger than wsize will see the bug.
Most users will want to use the 6.2 HWE kernel until this is fixed.
[Testcase]
Start two VMs, one for the server, and the other, the client.
Server
------
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install samba
$ sudo vim /etc/samba/smb.conf
server min protocol = NT1
[sambashare]
comment = Samba on Ubuntu
path = /home/ubuntu/sambashare
read only = no
browsable = yes
$ mkdir ~/sambashare
$ sudo smbpasswd -a ubuntu
Client
------
$ sudo apt update
$ sudo apt install cifs-utils
$ mkdir ~/share
$ sudo mount -t cifs -o username=ubuntu,vers=1.0,wsize=16850 //192.168.122.172/sambashare ~/share
$ ( set -o pipefail && head --bytes=$(( 55 * 1000 )) /dev/zero | openssl enc -aes-128-ctr -nosalt -pass "pass:my-seed" -iter 1 | hexdump --no-squeezing --format '40/1 "%02x"' --format '"\n"' >"testdata.txt" )
$ sha256sum testdata.txt
9ec09af020dce3270ea76531141940106f173c7243b7193a553480fb8500b3f2 testdata.txt
Now copy the file to the share.
Client
------
$ cp testdata.txt share/
Server
------
$ sha256sum sambashare/testdata.txt
9e573a0aa795f9cd4de4ac684a1c056dbc7d2ba5494d02e71b6225ff5f0fd866 sambashare/testdata.txt
The SHA256 hash is different. If you view the file with less, you will find a block of wsize=16850 bytes, then 3900 bytes of binary zeros, followed by another wsize=16850 bytes, then 3900 bytes of binary zeros, etc.
An example of a broken file is:
https://launchpadlibrarian.net/712573213/testdata-back-from-server.txt
[Where problems could occur]
[Other info]
Currently bisecting. Introduced in 6.3-rc1. Currently broken on mainline 6.8-rc3. |
|
| 2024-02-05 04:31:59 |
Matthew Ruffell |
tags |
|
seg |
|
| 2024-02-07 22:26:11 |
Matthew Ruffell |
summary |
smb1: wsize blocks of bytes followed with binary zeros on copy, destroying data |
smb: wsize blocks of bytes followed with binary zeros on copy, destroying data |
|
| 2024-02-22 00:23:39 |
Matthew Ruffell |
description |
[Impact]
Upon installing the 6.5 HWE kernel on Jammy, users with a custom wsize set will see data destruction when copying files from their systems onto a cifs smb 1.0 mount.
wsize defaults to 65535 bytes, but when set to smaller values, like 16850, users will see blocks of 16850 bytes copied over, followed by 3900 binary zeros, followed by the next block of data followed by more binary zeros.
A workaround is to increase wsize, but this only works for small files, as any files larger than wsize will see the bug.
Most users will want to use the 6.2 HWE kernel until this is fixed.
[Testcase]
Start two VMs, one for the server, and the other, the client.
Server
------
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install samba
$ sudo vim /etc/samba/smb.conf
server min protocol = NT1
[sambashare]
comment = Samba on Ubuntu
path = /home/ubuntu/sambashare
read only = no
browsable = yes
$ mkdir ~/sambashare
$ sudo smbpasswd -a ubuntu
Client
------
$ sudo apt update
$ sudo apt install cifs-utils
$ mkdir ~/share
$ sudo mount -t cifs -o username=ubuntu,vers=1.0,wsize=16850 //192.168.122.172/sambashare ~/share
$ ( set -o pipefail && head --bytes=$(( 55 * 1000 )) /dev/zero | openssl enc -aes-128-ctr -nosalt -pass "pass:my-seed" -iter 1 | hexdump --no-squeezing --format '40/1 "%02x"' --format '"\n"' >"testdata.txt" )
$ sha256sum testdata.txt
9ec09af020dce3270ea76531141940106f173c7243b7193a553480fb8500b3f2 testdata.txt
Now copy the file to the share.
Client
------
$ cp testdata.txt share/
Server
------
$ sha256sum sambashare/testdata.txt
9e573a0aa795f9cd4de4ac684a1c056dbc7d2ba5494d02e71b6225ff5f0fd866 sambashare/testdata.txt
The SHA256 hash is different. If you view the file with less, you will find a block of wsize=16850 bytes, then 3900 bytes of binary zeros, followed by another wsize=16850 bytes, then 3900 bytes of binary zeros, etc.
An example of a broken file is:
https://launchpadlibrarian.net/712573213/testdata-back-from-server.txt
[Where problems could occur]
[Other info]
Currently bisecting. Introduced in 6.3-rc1. Currently broken on mainline 6.8-rc3. |
BugLink: https://bugs.launchpad.net/bugs/2049634
[Impact]
Upon installing the 6.5 HWE kernel on Jammy, users with a custom wsize set will see data destruction when copying files from their systems onto a cifs smb mount.
wsize defaults to 65535 bytes, but when set to smaller values, like 16850, users will see blocks of 16850 bytes copied over, followed by 3900 binary zeros, followed by the next block of data followed by more binary zeros. This destroys what you are copying, and the data corruption is completely silent.
A workaround is to set wsize to a multiple of PAGE_SIZE.
[Fix]
This was fixed upstream in 6.8-rc5 by the following:
commit 4860abb91f3d7fbaf8147d54782149bb1fc45892
Author: Steve French <stfrench@microsoft.com>
Date: Tue Feb 6 16:34:22 2024 -0600
Subject: smb: Fix regression in writes when non-standard maximum write size negotiated
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4860abb91f3d7fbaf8147d54782149bb1fc45892
This patch has been selected for upstream stable.
The patch looks at wsize negotiation from the server, and also what is set on the mount command line, and if not a multiple of PAGE_SIZE, rounds it down, taking care to not be 0.
The real corruption bug still exists in netfs/folios subsystems and we are looking for it still, but this solves the immediate data corruption issues on smb.
[Testcase]
Start two VMs, one for the server, and the other, the client.
Server
------
$ sudo apt update
$ sudo apt upgrade
$ sudo apt install samba
$ sudo vim /etc/samba/smb.conf
server min protocol = NT1
[sambashare]
comment = Samba on Ubuntu
path = /home/ubuntu/sambashare
read only = no
browsable = yes
$ mkdir ~/sambashare
$ sudo smbpasswd -a ubuntu
Client
------
$ sudo apt update
$ sudo apt install cifs-utils
$ mkdir ~/share
$ sudo mount -t cifs -o username=ubuntu,vers=1.0,wsize=16850 //192.168.122.172/sambashare ~/share
$ ( set -o pipefail && head --bytes=$(( 55 * 1000 )) /dev/zero | openssl enc -aes-128-ctr -nosalt -pass "pass:my-seed" -iter 1 | hexdump --no-squeezing --format '40/1 "%02x"' --format '"\n"' >"testdata.txt" )
$ sha256sum testdata.txt
9ec09af020dce3270ea76531141940106f173c7243b7193a553480fb8500b3f2 testdata.txt
Now copy the file to the share.
Client
------
$ cp testdata.txt share/
Server
------
$ sha256sum sambashare/testdata.txt
9e573a0aa795f9cd4de4ac684a1c056dbc7d2ba5494d02e71b6225ff5f0fd866 sambashare/testdata.txt
The SHA256 hash is different. If you view the file with less, you will find a block of wsize=16850 bytes, then 3900 bytes of binary zeros, followed by another wsize=16850 bytes, then 3900 bytes of binary zeros, etc.
An example of a broken file is:
https://launchpadlibrarian.net/712573213/testdata-back-from-server.txt
[Where problems could occur]
We are changing the wsize that the SMB server negotiates or the user explicitly sets on the mount command line to a safe value, if the asked for value was not a multiple of PAGE_SIZE.
It is unlikely that any users will notice any difference. If the wsize happens to be rounded down to a significantly smaller number, there may be a small performance impact, e.g. you set wsize=8094 for some reason, it would round down to wsize=4096, and lead to double the writes. At least you have data integrity though.
Most users default to wsize=65535, and will have no impact at all. Only those with very old smb 1.0 servers that negotiate down to 16850 will see any difference.
If a regression were to occur, it could result in user's wsize being incorrectly set, leading to the underlying netfs/folios data corruption being triggered and causing data corruption.
[Other info]
The issue was introduced in 6.3-rc1 by:
commit d08089f649a0cfb2099c8551ac47eef0cc23fdf2
Author: David Howells <dhowells@redhat.com>
Date: Mon Jan 24 21:13:24 2022 +0000
Subject: cifs: Change the I/O paths to use an iterator rather than a page list
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d08089f649a0cfb2099c8551ac47eef0cc23fdf2
Upstream mailing list discussion:
https://lore.kernel.org/linux-cifs/6a65b2d1-7596-438a-8ade-2f7526b15596@rd10.de/T/#m22cd9b7289f87cd945978bd7995bcaf1beebfe67 |
|
| 2024-02-22 00:24:04 |
Matthew Ruffell |
tags |
seg |
mantic noble seg |
|
| 2024-02-23 12:56:04 |
Roxana Nicolescu |
linux (Ubuntu Mantic): status |
In Progress |
Fix Committed |
|
| 2024-03-11 17:50:22 |
Ubuntu Kernel Bot |
tags |
mantic noble seg |
kernel-spammed-mantic-linux-v2 mantic noble seg verification-needed-mantic-linux |
|
| 2024-03-14 04:00:19 |
Matthew Ruffell |
linux (Ubuntu Noble): status |
In Progress |
Fix Committed |
|
| 2024-03-14 04:34:06 |
Matthew Ruffell |
tags |
kernel-spammed-mantic-linux-v2 mantic noble seg verification-needed-mantic-linux |
kernel-spammed-mantic-linux-v2 mantic noble seg verification-done-mantic-linux |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
linux (Ubuntu Mantic): status |
Fix Committed |
Fix Released |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2023-46838 |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2023-50431 |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2024-1085 |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2024-1086 |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2024-22705 |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2024-23850 |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2024-23851 |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2024-26597 |
|
| 2024-04-08 03:50:10 |
Launchpad Janitor |
cve linked |
|
2024-26599 |
|
| 2024-04-08 04:03:06 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-mantic-linux-v2 mantic noble seg verification-done-mantic-linux |
kernel-spammed-mantic-linux-raspi-v2 kernel-spammed-mantic-linux-v2 mantic noble seg verification-done-mantic-linux verification-needed-mantic-linux-raspi |
|
| 2024-04-19 05:41:21 |
Matthew Ruffell |
linux (Ubuntu Noble): status |
Fix Committed |
Fix Released |
|
