CIFS stopped working/is unstable with kernel update to 5.15.0-102.112

[Andreas Besold]

Hi,

updated some Ubuntu 22.04 systems to lastest available state this morning, which caused CIFS mounts (from various fileservers) to stop working. Kernel was updated to version 5.15.0-102-generic.

I can mount the shares without problems (mount -t cifs), but then, df for example tells me: df: /mnt: Resource temporarily unavailable.
I'm able to list and browse all the files, but accessing them (even readonly) is very unstable. Sometimes it works and sometimes it just gives me i/o errors.

Switching back to 5.15.0-101-generic or 5.15.0-100-generic solves the problem and everything works again as expected.

Seems like some bug has been implemented in 5.15.0-102-generic...

To reproduce the problem, I started a while loop on one server to write to some file on a specific mounted CIFS share and read it from another one

root@<hostname1>:~# while true; do echo "$(date) hallo" >> /mnt/hallo.txt; sleep 1 ; done
-bash: /mnt/hallo.txt: Input/output error
-bash: /mnt/hallo.txt: Input/output error
^C

root@<hostname2>:~$ tail -f /mnt/hallo.txt
Tue Apr 9 04:10:52 PM CEST 2024 hallo
Tue Apr 9 04:10:53 PM CEST 2024 hallo
Tue Apr 9 04:10:54 PM CEST 2024 hallo
Tue Apr 9 04:10:55 PM CEST 2024 hallo
Tue Apr 9 04:10:56 PM CEST 2024 hallo
Tue Apr 9 04:10:57 PM CEST 2024 hallo
Tue Apr 9 04:10:58 PM CEST 2024 hallo
Tue Apr 9 04:10:59 PM CEST 2024 hallo
Tue Apr 9 04:11:00 PM CEST 2024 hallo
Tue Apr 9 04:11:01 PM CEST 2024 hallo
tail: cannot determine location of '/mnt/hallo.txt'. reverting to polling: Resource temporarily unavailable
Tue Apr 9 04:11:04 PM CEST 2024 hallo
Tue Apr 9 04:11:05 PM CEST 2024 hallo
Tue Apr 9 04:11:06 PM CEST 2024 hallo
Tue Apr 9 04:11:07 PM CEST 2024 hallo
Tue Apr 9 04:11:08 PM CEST 2024 hallo
Tue Apr 9 04:11:09 PM CEST 2024 hallo
Tue Apr 9 04:11:10 PM CEST 2024 hallo

While doing this, both servers tell me, the resource is unavailable

root@<hostname1>:~# df -h /mnt
df: /mnt: Resource temporarily unavailable

root@<hostname2>:~$ df -h /mnt
df: /mnt: Resource temporarily unavailable

[Andreas Besold (absld)]

See related question

[Axel Schneck (aschneckubuntu)]

Same problems here. Get error messages like "Too many symlinks" and exactly the same as in this description.
Found also, that CIFS mounts to Netapp mostly works, but df -h doesnt show them and mounts to other CIFS shares doesnt work.
Verified it on 5 Linux Boxes, all started failing after patch and working again when go back to snapshot from yesterday (before kernel patch): These patches are he "bad ones":

Start-Date: 2024-04-09 06:21:54
Commandline: /usr/bin/unattended-upgrade
Install: linux-image-5.15.0-102-generic:amd64 (5.15.0-102.112, automatic), linux-modules-5.15.0-102-generic:amd64 (5.15.0-102.112, automatic), linux-headers-5.15.0-102:amd64 (5.15.0-102.112, automatic), linux-modules-extra-5.15.0-102-generic:amd64 (5.15.0-102.112, automatic), linux-headers-5.15.0-102-generic:amd64 (5.15.0-102.112, automatic)
Upgrade: linux-headers-generic:amd64 (5.15.0.101.98, 5.15.0.102.99), linux-generic:amd64 (5.15.0.101.98, 5.15.0.102.99), linux-image-generic:amd64 (5.15.0.101.98, 5.15.0.102.99)
End-Date: 2024-04-09 06:22:23

Start-Date: 2024-04-09 06:22:26
Commandline: /usr/bin/unattended-upgrade
Remove: linux-modules-5.15.0-101-generic:amd64 (5.15.0-101.111), linux-modules-extra-5.15.0-101-generic:amd64 (5.15.0-101.111), linux-image-5.15.0-101-generic:amd64 (5.15.0-101.111)
End-Date: 2024-04-09 06:22:30

Start-Date: 2024-04-09 06:22:33
Commandline: /usr/bin/unattended-upgrade
Remove: linux-headers-5.15.0-101-generic:amd64 (5.15.0-101.111)
End-Date: 2024-04-09 06:22:34

Start-Date: 2024-04-09 06:22:36
Commandline: /usr/bin/unattended-upgrade
Remove: linux-headers-5.15.0-101:amd64 (5.15.0-101.111)
End-Date: 2024-04-09 06:22:37

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Benjamin Richner (benri)]

I collected a bunch of info on it. Seems to be reported all over the place:

- https://www.reddit.com/r/linux/comments/1bzvy5l/ubuntu_22044_latest_updates_492024_break_cifssmb/
- https://www.reddit.com/r/Ubuntu/comments/1bzshdt/ubuntu_2204_smb_shares_stopped_working_lastnight/
- https://askubuntu.com/questions/1509987/cifs-on-22-04-lts-has-something-changed-between-kernel-5-15-0-101-generic-and
- https://forum.level1techs.com/t/cifs-broken-by-ubuntu-server-22-04-4-latest-updates/209639
- https://forums.linuxmint.com/viewtopic.php?t=417371

Oh and there's also a duplicate: https://bugs.launchpad.net/ubuntu/+source/cifs-utils/+bug/2060797

The most promising lead is this:

> Looks like there's a fix that's yet to make it to Linux Mint and Ubuntu kernels:
>
> [PATCH 5.15.y 0/1] smb: client: fix "df: Resource temporarily unavailable" on 5.15 stable kernel
> https://<email address hidden>/T/

Many people are affected.

[Matthew Ruffell (mruffell)]

Hi everyone,

Reading:

https://<email address hidden>/T/

It seems the issue was introduced in

commit 33eae65c6f49770fec7a662935d4eb4a6406d24b
Author: Paulo Alcantara <email address hidden>
Date: Wed Dec 13 12:25:57 2023 -0300
Subject: smb: client: fix OOB in SMB2_query_info_init()
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=33eae65c6f49770fec7a662935d4eb4a6406d24b

this was indeed landed in 5.15.0-102-generic:

$ git log --grep "smb: client: fix OOB in SMB2_query_info_init()" origin/master-next
commit ed30eac9715d0bd5512ee42ca8e8f340d2d9d2d5
...

$ git describe --contains ed30eac9715d0bd5512ee42ca8e8f340d2d9d2d5
Ubuntu-5.15.0-102.112~472

The link mentions it is supposedly fixed in:

commit b5d623611c9cda84ebb5e5bd044587955eaf782f
Author: Kees Cook <email address hidden>
Date: Fri Feb 17 16:24:40 2023 -0800
Subject: smb3: Replace smb2pdu 1-element arrays with flex-arrays
Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b5d623611c9cda84ebb5e5bd044587955eaf782f

Looking at the Jammy kernel tree:

$ git log --grep "smb3: Replace smb2pdu 1-element arrays with flex-arrays" origin/master-next
commit b5d623611c9cda84ebb5e5bd044587955eaf782f

$ git describe --contains b5d623611c9cda84ebb5e5bd044587955eaf782f
Ubuntu-5.15.0-104.114~23

It seems it is already applied and tagged in 5.15.0-104-generic. This seems to be built, but not quite in -proposed yet:

https://kernel.ubuntu.com/reports/kernel-stable-board/

I will write back as soon as 5.15.0-104-generic is in -proposed, with instructions on how to test it, and see if it fixes the issue.

Thanks,
Matthew

[Matthew Ruffell (mruffell)]

Hi everyone,

5.15.0-104-generic just hit jammy-proposed a couple minutes ago. It isn't available for focal users yet.

Can someone please test 5.15.0-104-generic on jammy and let me know if it fixes the issue? Thanks.

Instructions to Install (On a mantic system):
1) cat << EOF | sudo tee /etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed main universe
EOF
2) sudo apt update
3) sudo apt install linux-image-5.15.0-104-generic linux-modules-5.15.0-104-generic linux-modules-extra-5.15.0-104-generic linux-headers-5.15.0-104-generic
4) sudo rm /etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
5) sudo apt update
6) sudo reboot
5.15.0-104-generic #114-Ubuntu SMP Thu Mar 28 15:39:51 UTC 2024

Just a note that 5.15.0-104-generic is a part of the 2024.04.01 SRU cycle, and
will likely be released at the end of month, as per https://kernel.ubuntu.com/,
and won't be coming out any sooner.

Thanks,
Matthew

[engaging374 (engaging374)]

Hello Matthew,

I tested it with 22.04.4 LTS.

Linux srv1 5.15.0-104-generic #114-Ubuntu SMP Thu Mar 28 15:39:51 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

'df -hT /opt/test'

Filesystem Type Size Used Avail Use% Mounted on
//cifssrv/clone1 cifs 9.5G 4.4G 5.2G 46% /opt/test

-> No 'Resource temporarily unavailable' error

I still have an error in 'dmesg | grep -i cifs'

[ 17.961962] CIFS: Attempting to mount \\cifssrv\clone1
[ 18.048245] CIFS: VFS: parse_server_interfaces: malformed interface info

Not sure if this is related to the bug.

[Benjamin Richner (benri)]

> Just a note that 5.15.0-104-generic is a part of the 2024.04.01 SRU cycle, and will likely be released at the end of month, as per https://kernel.ubuntu.com/, and won't be coming out any sooner.

Do I understand this correctly, do you keep kernel update 5.15.0-102.112 in place and let people update into a kernel that breaks SMB/CIFS for almost an entire month? Isn't that dangerous? I manage a bunch of Ubuntu servers and I had to instruct people to hold off on all Ubuntu updates immediately as not to break all the file shares, which would be very disruptive to our operations.

[Axel Schneck (aschneckubuntu)]

Hello,

I just tested your suggestion wit 104
It's working here with this kernel
Dont understand why you will still provide a faulty kernel until end of month?
We have a lot of servers here using CIFS mounts and we have to prevent all of them now from getting a faulty kernel.
bye
Axel

[Andreas Besold (absld)]

Hi,

I also did some testing with 5.15.0-104-generic from proposed repository and can confirm this version works again.

I urgently request you to reconsider the decision to postpone the release until the end of the month. Automatic patching/updating currently breaks several servers every day but I also don't want to leave them unpatched!

Thanks, Andreas

[Roxana Nicolescu (roxanan)]

Hi! Thanks for reporting this.
We will do our best to release new kernels in 1-2 days. In the meantime, the meta package will be deleted so that other people do not upgrade to 5.15.0-102.

[Tim Dee (entropy1024)]

Good to know a new kernel is coming out soon to fix the issue. Thank you.

Where is the best place to look to see when the patch is released?

Many thanks.

[Matthew Ruffell (mruffell)]

The new kernel is tagged as 5.15.0-105-generic, and it is a respin of 5.15.0-103-generic with the fix.

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/log/?h=Ubuntu-5.15.0-105.115

It should be available in -proposed soon, the Kernel Team are still rebuilding all the 5.15 derivatives.

I'll write back once it is in -proposed for testing.

Thanks,
Matthew

[Andy Townsend (someoneelseosm)]

> In the meantime, the meta package will be deleted so that other people do not upgrade to 5.15.0-102.

Just checking, has that happened? I'm still seeing that offered in response to "apt upgrade" on an older system here:

    The following NEW packages will be installed:
      linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic
      linux-image-5.15.0-102-generic linux-modules-5.15.0-102-generic
      linux-modules-extra-5.15.0-102-generic

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.15.0-105.115 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux' to 'verification-done-jammy-linux'. If the problem still exists, change the tag 'verification-needed-jammy-linux' to 'verification-failed-jammy-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra/5.15.0-1024.24 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-done-jammy-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra' to 'verification-failed-jammy-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Jacob Rogers (hato1)]

I had a hard time diagnosing this issue due to the difficulty of finding this bug report. As I use Focal, I didn't think to check the Jammy board. The general Ubuntu board has a report for this issue, but since it's marked as a duplicate of this one it is hidden by default.

[engaging374 (engaging374)]

Yep, Focal is also affected. Will this fix also be available for Focal?

Edit:
I have tried to add it here, but looked wrong. Not sure if it's the right way to add Focal to this bug.

[Matthew Ruffell (mruffell)]

Hi everyone, the fix will also be available for Focal, as part of it getting the Jammy HWE kernel. It will be released at the same time as Jammy's kernel is released.

We got the notification above that the kernel should be in -proposed, but I have been refreshing -proposed all day and the kernel is still not copied in just yet.

Once it reaches -proposed, I will test it, and let you know when you can do some testing too. But for the moment, we need to be patient while it gets copied into -proposed.

Thanks,
Matthew

[S (jaxjexjox)]

I feel compelled to agree with Axel and Andreas:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2060780/comments/9

This bug is particularly surprising for a stable release, breaking CIFS is not a pleasant thing to deal with. I tend to stick to the stable builds to avoid things just like this.

Thank you all for the effort, obviously. It truly is appreciated but the sooner this is fixed up the better, it should really be at critical levels of severity.

[Andreas Besold (absld)]

Hi,

I can confirm that linux-generic 5.15.0.105 is not available yet in proposed repository. Latest version is 104.

# apt policy linux-generic
linux-generic:
  Installed: 5.15.0.102.99
  Candidate: 5.15.0.104.101
  Version table:
     5.15.0.104.101 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
 *** 5.15.0.102.99 500
        500 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     5.15.0.25.27 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

# apt info linux-image-generic
Package: linux-image-generic
Version: 5.15.0.104.101
Priority: optional
Section: kernel
Source: linux-meta
Origin: Ubuntu
Maintainer: Ubuntu Kernel Team <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 21.5 kB
Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-104), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3)
Depends: linux-image-5.15.0-104-generic, linux-modules-extra-5.15.0-104-generic, linux-firmware, intel-microcode, amd64-microcode
Recommends: thermald
Download-Size: 2,514 B
APT-Sources: http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
Description: Generic Linux kernel image
 This package will always depend on the latest generic kernel image
 available.

[Kirkland Schlicht (quirkykirkhax)]

Spent so long troubleshooting a container and Veeam Agent issue just to find out it was a kernel issue....And here I was worried that power outage borked something...

[Matthew Ruffell (mruffell)]

Performing verification for Jammy.

I started two Jammy VMs. One is a server and the other a client.

Server
------

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install samba
$ sudo vim /etc/samba/smb.conf
[sambashare]
    comment = Samba on Ubuntu
    path = /home/ubuntu/sambashare
    read only = no
    browsable = yes
$ mkdir ~/sambashare
$ sudo smbpasswd -a ubuntu

Client
------

$ sudo apt update
$ sudo apt install cifs-utils
$ mkdir ~/share
$ sudo mount -t cifs -o username=ubuntu //192.168.122.172/sambashare ~/share

The client is using 5.15.0-102-generic from updates:

$ uname -rv
5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024

$ sudo -s
# while true; do echo "$(date) hallo" >> ./share/hallo.txt; sleep 1 ; done

Launch a separate terminal and run:

$ df -h ~/share
df: /home/ubuntu/share: Resource temporarily unavailable

The issue is seen.

I then enabled -proposed and installed 5.15.0-105-generic.

I had to use the -proposed2 repository due to the packages not being present
in the primary archive -proposed repository.

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/proposed2

$ uname -rv
5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024

$ sudo -s
# while true; do echo "$(date) hallo" >> ./share/hallo.txt; sleep 1 ; done

$ df -h ~/share
Filesystem Size Used Avail Use% Mounted on
//192.168.122.247/sambashare 9.6G 2.1G 7.5G 23% /home/ubuntu/share

CIFS is back to working again, and we can df the share and see the disk usage.

The respun kernel in -proposed fixes the issue, happy to mark verified for Jammy.

[Thomas Angert (tangert)]

I had no problem installing and testing 5.15.0-104, but when I try to install 105 I get:

Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ...
/etc/kernel/header_postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.15.0-105-generic
Error! The /var/lib/dkms/wireguard/1.0.20210606/5.15.0-105-generic/x86_64/dkms.conf for module wireguard includes a BUILD_EXCLUSIVE directive which does not match this kernel/arch.
The /var/lib/dkms/wireguard/1.0.20210606/5.15.0-105-generic/x86_64/dkms.conf for module wireguard includes a BUILD_EXCLUSIVE directive which does not match this kernel/arch/config.
This indicates that it should not be built.
   ...done.

[Thomas Angert (tangert)]

BUILD_EXCLUSIVE errors were there in 104 also. Wireguard still works.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.15.0-105.115

---------------
linux (5.15.0-105.115) jammy; urgency=medium

  * jammy/linux: 5.15.0-105.115 -proposed tracker (LP: #2061372)

  * Jammy update: v5.15.149 upstream stable release (LP: #2059014) // CIFS
    stopped working/is unstable with kernel update to 5.15.0-102.112
    (LP: #2060780)
    - smb3: Replace smb2pdu 1-element arrays with flex-arrays

linux (5.15.0-103.113) jammy; urgency=medium

  * jammy/linux: 5.15.0-103.113 -proposed tracker (LP: #2059683)

  * Packaging resync (LP: #1786013)
    - [Packaging] drop getabis data

  * Remove getabis scripts (LP: #2059143)
    - [Packaging] Remove getabis

  * CVE-2023-24023
    - Bluetooth: Add more enc key size check

  * CVE-2023-52600
    - jfs: fix uaf in jfs_evict_inode

  * Jammy update: v5.15.149 upstream stable release (LP: #2059014) //
    CVE-2023-52603
    - UBSAN: array-index-out-of-bounds in dtSplitRoot

  * CVE-2024-26581
    - netfilter: nft_set_rbtree: skip end interval element from gc

 -- Roxana Nicolescu <email address hidden> Mon, 15 Apr 2024 10:38:15 +0200

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-bluefield/5.15.0-1042.44 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-bluefield' to 'verification-done-jammy-linux-bluefield'. If the problem still exists, change the tag 'verification-needed-jammy-linux-bluefield' to 'verification-failed-jammy-linux-bluefield'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-lowlatency/5.15.0-105.115 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-lowlatency' to 'verification-done-jammy-linux-lowlatency'. If the problem still exists, change the tag 'verification-needed-jammy-linux-lowlatency' to 'verification-failed-jammy-linux-lowlatency'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia/5.15.0-1051.52 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia' to 'verification-done-jammy-linux-nvidia'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia' to 'verification-failed-jammy-linux-nvidia'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Matthew Ruffell (mruffell)]

Hi everyone,

5.15.0-105-generic has been released to -updates now. It fixes the cifs issue.

https://launchpad.net/ubuntu/+source/linux/5.15.0-105.115

Please let us know if you find any more issues.

Thanks,
Matthew

[Axel Schneck (aschneckubuntu)]

This bug was the first time I came into this forum, so sorry if this a dumb question:
today I got 3 emails that that a fix might be available and I have to test it. If I do nbot the fix will be removed from kernel?
Really? Is it really needed that I do some tests after so many already did? Currently I'm out of office so it's complicated to test anything.
As far as I understand it is already fixed and confirmed?

[engaging374 (engaging374)]

Hi Matthew,

tested Jammy & Focal (hwe). Looks good for me. :-)

Thank you & the Kernel Team!

[engaging374 (engaging374)]

@Alex

Well it's officially released now. It's up to you to install the patch. :-)

I think someone needs to verify the patch as soon as it's in -proposed state. Without verification it will be dropped from the source code. ( -> no official fix will be provided in -updates.)

Normally it's not the case if multiple users are affected. Someone just needs to do the verification work.

[Axel Schneck (aschneckubuntu)]

Thanks a lot !
Axel
>>> engaging374 <email address hidden> 19.04.2024 10:16 >>>
@Alex

Well it's officially released now. It's up to you to install the patch.
:-)

I think someone needs to verify the patch as soon as it's in -proposed
state. Without verification it will be dropped from the source code. (
-> no official fix will be provided in -updates.)

Normally it's not the case if multiple users are affected. Someone just
needs to do the verification work.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/2060780

Title:
  CIFS stopped working/is unstable with kernel update to 5.15.0-102.112

Status in linux package in Ubuntu:
  Invalid
Status in linux source package in Jammy:
  Fix Released

Bug description:
  Hi,

  updated some Ubuntu 22.04 systems to lastest available state this
  morning, which caused CIFS mounts (from various fileservers) to stop
  working. Kernel was updated to version 5.15.0-102-generic.

  I can mount the shares without problems (mount -t cifs), but then, df for example tells me: df: /mnt: Resource temporarily unavailable.
  I'm able to list and browse all the files, but accessing them (even readonly) is very unstable. Sometimes it works and sometimes it just gives me i/o errors.

  Switching back to 5.15.0-101-generic or 5.15.0-100-generic solves the
  problem and everything works again as expected.

  Seems like some bug has been implemented in 5.15.0-102-generic...

  To reproduce the problem, I started a while loop on one server to
  write to some file on a specific mounted CIFS share and read it from
  another one

  root@<hostname1>:~# while true; do echo "$(date) hallo" >> /mnt/hallo.txt; sleep 1 ; done
  -bash: /mnt/hallo.txt: Input/output error
  -bash: /mnt/hallo.txt: Input/output error
  ^C

  root@<hostname2>:~$ tail -f /mnt/hallo.txt
  Tue Apr 9 04:10:52 PM CEST 2024 hallo
  Tue Apr 9 04:10:53 PM CEST 2024 hallo
  Tue Apr 9 04:10:54 PM CEST 2024 hallo
  Tue Apr 9 04:10:55 PM CEST 2024 hallo
  Tue Apr 9 04:10:56 PM CEST 2024 hallo
  Tue Apr 9 04:10:57 PM CEST 2024 hallo
  Tue Apr 9 04:10:58 PM CEST 2024 hallo
  Tue Apr 9 04:10:59 PM CEST 2024 hallo
  Tue Apr 9 04:11:00 PM CEST 2024 hallo
  Tue Apr 9 04:11:01 PM CEST 2024 hallo
  tail: cannot determine location of '/mnt/hallo.txt'. reverting to polling: Resource temporarily unavailable
  Tue Apr 9 04:11:04 PM CEST 2024 hallo
  Tue Apr 9 04:11:05 PM CEST 2024 hallo
  Tue Apr 9 04:11:06 PM CEST 2024 hallo
  Tue Apr 9 04:11:07 PM CEST 2024 hallo
  Tue Apr 9 04:11:08 PM CEST 2024 hallo
  Tue Apr 9 04:11:09 PM CEST 2024 hallo
  Tue Apr 9 04:11:10 PM CEST 2024 hallo

  While doing this, both servers tell me, the resource is unavailable

  root@<hostname1>:~# df -h /mnt
  df: /mnt: Resource temporarily unavailable

  root@<hostname2>:~$ df -h /mnt
  df: /mnt: Resource temporarily unavailable

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2060780/+subscriptions

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-fips/5.15.0-105.115+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-fips' to 'verification-done-jammy-linux-fips'. If the problem still exists, change the tag 'verification-needed-jammy-linux-fips' to 'verification-failed-jammy-linux-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Axel Schneck (aschneckubuntu)]

I'm totally confused now. Last note her says you're wating for confirmation, but I got 2 emails telling me, that the fixed kernel has been released now. But when I do a apt-get update and apt-get upgrade still 104 will be installed? Did I misunderstood something?
Axel

[Axel Schneck (aschneckubuntu)]

Seems apt-upgrade will not download new kernel:
Die folgenden Pakete sind zurückgehalten worden:
  linux-generic linux-headers-generic linux-image-generic ubuntu-advantage-tools ubuntu-pro-client-l10n

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp-tcpx/5.15.0-1011.11 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-gcp-tcpx' to 'verification-done-focal-linux-gcp-tcpx'. If the problem still exists, change the tag 'verification-needed-focal-linux-gcp-tcpx' to 'verification-failed-focal-linux-gcp-tcpx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Axel Schneck (aschneckubuntu)]

Running an apt upgrade rather than apt-get upgrade installed the new kernel now.
Just installed on testserver and tried CIFS mounts with a short test.
It seems working now, thanks !
I'll install as soon as possible this kernel on 2 more productive servers and try what's happen
Bye
Axel

[Daniel Dawson (dotdawson)]

Hi, I believe that kernel 5.15.0-105-generic may not have have solved the issue entirely.

I upgraded to 104 from -proposed, and then when 105 was available, I updated to 105, and then removed 104 and the -proposed repo entirely. While, the issue does not present itself as frequently, I still encounter similar if not the same CIFS errors that were not present in older kernels.

After updating to 105, I no longer encountered the issue when I first connect to the SMB share. However, I encounter the issue during long file copies to an SMB share. After trying to copy 20GB of data to the smb server, the the copy is interrupted after some time, and then the mount point is broken and unable to be accessed.

I am using dockers cifs volumes to mount smb shares into docker containers. The host uses cifs and mounts these into the /var/lib/docker/volumes/... directories which are remapped to containers in some way.

I have encountered the issue on 105 after the following steps.
On the host, I directly added a mount in `/etc/fstab`, updated `systemctl daemon-reload`, and mounted the share `mount -a`.
I then copy 20GB of data to the share `cp -r /path/to/my/data /mnt/rf/data`. After about a minute, the copy terminates.
In `dmesg`, I see the cifs errors. When I look at the host CIFS mount point, I can see that the folder is inaccessible.

uname -a
```
Linux docker-gpu-01 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
```

apt list --installed | grep "cifs-utils"
```
cifs-utils/jammy-updates,jammy-security,now 2:6.14-1ubuntu0.1 amd64 [installed]
```

cat /etc/fstab
```
...
//10.0.6.2/rf /mnt/rf cifs uid=1000,gid=1000,credentials=/home/user/.smb 0 0
```

sudo dmesg
```
[138420.302415] CIFS: Attempting to mount \\10.0.6.2\rf
[138420.315798] CIFS: VFS: parse_server_interfaces: malformed interface info
[138538.756550] CIFS: VFS: \\10.0.6.2 sends on sock 000000009d8f9284 stuck for 15 seconds
[138538.757040] CIFS: VFS: \\10.0.6.2 Error -11 sending data on socket to server
[138543.032429] CIFS: reconnect tcon failed rc = -13
[138543.051221] CIFS: VFS: No writable handle in writepages rc=-13
[138543.053159] CIFS: VFS: No writable handle in writepages rc=-13
[138543.063685] CIFS: VFS: No writable handle in writepages rc=-13
[138543.065707] CIFS: VFS: No writable handle in writepages rc=-13
[138543.077380] CIFS: VFS: No writable handle in writepages rc=-13
[138543.080055] CIFS: VFS: No writable handle in writepages rc=-13
[138543.090757] CIFS: VFS: No writable handle in writepages rc=-13
[138543.092996] CIFS: VFS: No writable handle in writepages rc=-13
[138543.151054] CIFS: VFS: \\10.0.6.2\media Close unmatched open for MID:116
```

Notes:
You can see that at `138420` I ran `mount -a` and the mount was added.
There was an error `parse_server_interfaces: malformed interface info`, but the file system was mounted and available.
Shortly after, I started the file transfer (I started it probably after 20-30 seconds.).
This file transfer failed by `138538` and produced the `VFS: \\10.0.6.2 sends on sock 000000009d8f9284 stuck for 15 seconds` error.
I can see on the sever that about `6GB/20GB` of data has transferred at thi...

[Matthew Ruffell (mruffell)]

Hi Axel,

Sorry about you getting spammed about all these verifications. The robot that posts these messages looks for the Launchpad bug number in the package changelogs, and just makes a post for every package it sees it in.

You might not know, but Ubuntu has slightly over 100 kernels these days, across all series releases. These are split in the primary kernels, like -generic, and then all the derivatives like -aws, -azure, -gcp, -fips, -virtual, and so on, and then including the HWE kernels and their derivatives too.

This got fixed in all the 5.15 kernels, so the robot goes crazy and spams those "please verify" comments for all the derivatives.

We usually just verify the -generic kernel only, unless it has greater impact to one of those derivatives.

There's no further action needed on your part. The fix won't be removed for not verifying a derivative you have never heard of.

Hope that clears it up a bit.

Thanks,
Matthew

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-riscv-5.15/5.15.0-1056.60~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-riscv-5.15' to 'verification-done-focal-linux-riscv-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-riscv-5.15' to 'verification-failed-focal-linux-riscv-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Nelle (nelle-u)]

Today, on my setup (TrueNAS-SCALE-22.12.4.2) I didn't notice any problems with new kernel 5.15.0-105-generic #115-Ubuntu when I copied 5 files with a total size of over 430GB from local disk to the mounted network CIFS drive.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-5.15/5.15.0-1061.67~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-aws-5.15' to 'verification-done-focal-linux-aws-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-aws-5.15' to 'verification-failed-focal-linux-aws-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Daniel Dawson (dotdawson)]

Nelle,

Seems no one else is experiencing the issue I described above, so it's probably a me problem. I will do more troubleshooting on my end. Thanks for sharing your results.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure/5.15.0-1063.72 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-azure' to 'verification-done-jammy-linux-azure'. If the problem still exists, change the tag 'verification-needed-jammy-linux-azure' to 'verification-failed-jammy-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra-5.15/5.15.0-1024.24~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-nvidia-tegra-5.15' to 'verification-done-focal-linux-nvidia-tegra-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-nvidia-tegra-5.15' to 'verification-failed-focal-linux-nvidia-tegra-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra-igx/5.15.0-1011.11 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-nvidia-tegra-igx' to 'verification-done-jammy-linux-nvidia-tegra-igx'. If the problem still exists, change the tag 'verification-needed-jammy-linux-nvidia-tegra-igx' to 'verification-failed-jammy-linux-nvidia-tegra-igx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-xilinx-zynqmp/5.15.0-1029.33 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-xilinx-zynqmp' to 'verification-done-jammy-linux-xilinx-zynqmp'. If the problem still exists, change the tag 'verification-needed-jammy-linux-xilinx-zynqmp' to 'verification-failed-jammy-linux-xilinx-zynqmp'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!