Mounting SOFS SMB shares fails

Bug #1792580 reported by Lucian Petrut on 2018-09-14
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Medium
Joseph Salisbury
Bionic
Medium
Joseph Salisbury
Cosmic
Medium
Unassigned

Bug Description

== SRU Justification ==
Commit 395a2076b406 is needed to fix a bug in Bionic which causes mounting
SOFS SMB shares to fail. It was found that this commit fixes the bug
after a reverse bisect.

This fix has already landed in Cosmic.

== Fix ==
395a2076b406 ("cifs: connect to servername instead of IP for IPC$ share)"

== Regression Potential ==
Low. This is a one liner limited to cifs. It was also cc'd to upstream
stable, so it has had additional upstrem review.

== Test Case ==
A test kernel was built with this patch and tested by the original bug reporter.
The bug reporter states the test kernel resolved the bug.

Ubuntu 18.04.1 LTS 4.15.0-33.36-generic
cifs-utils 2:6.8-1

Mounting scale-out SMB shares is failing.

mount -t cifs -o credentials=/tmp/creds,domain=cbs.com,uid=114,gid=119,vers=3.0,_netdev,nodfs //s2dscaleout.cbs.com/glance-images /tmp/test_mount
mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

What works:
* mounting the share from a 16.04 host
* mounting standalone smb3 shares

Relevant error from the log (full logs attached separately):
Sep 14 11:12:05 r07-u13 kernel: Status code returned 0xc00000c9 STATUS_NETWORK_NAME_DELETED
Sep 14 11:12:05 r07-u13 kernel: /build/linux-81MBYC/linux-4.15.0/fs/cifs/smb2maperror.c: Mapping SMB2 status code 0xc00000c9 to POSIX err -5

Share server os: Windows Server 2016 10.0.14393
PS C:\Users\jujuadmin> get-smbshare glance-images | fl *

ShareState : Online
AvailabilityType : ScaleOut
ShareType : FileSystemDirectory
FolderEnumerationMode : Unrestricted
CachingMode : Manual
SmbInstance : Default
CATimeout : 0
ConcurrentUserLimit : 0
ContinuouslyAvailable : True
CurrentUsers : 2
Description :
EncryptData : False
Name : glance-images
Path : C:\ClusterStorage\Volume1\GLANCE-IMAGES
Scoped : True
ScopeName : S2DSCALEOUT
ShadowCopy : False
Special : False
Temporary : False
Volume : \\?\Volume{5faf880d-d308-4058-9ec2-d19d48089495}\
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 7 16:51 seq
 crw-rw---- 1 root audio 116, 33 Sep 7 16:51 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Dell Inc. PowerEdge R710
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-33-generic root=UUID=ba74e4cf-9634-4049-abd8-0ed721836a41 ro
ProcVersionSignature: Ubuntu 4.15.0-33.36-generic 4.15.18
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-33-generic N/A
 linux-backports-modules-4.15.0-33-generic N/A
 linux-firmware 1.173.1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic uec-images
Uname: Linux 4.15.0-33-generic x86_64
UnreportableReason: This report is about a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: False
dmi.bios.date: 07/23/2013
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 6.4.0
dmi.board.name: 0YDJK3
dmi.board.vendor: Dell Inc.
dmi.board.version: A14
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr6.4.0:bd07/23/2013:svnDellInc.:pnPowerEdgeR710:pvr:rvnDellInc.:rn0YDJK3:rvrA14:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R710
dmi.sys.vendor: Dell Inc.

CVE References

Lucian Petrut (petrutlucian94) wrote :
Lucian Petrut (petrutlucian94) wrote :
tags: added: kernel-bug

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1792580

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
description: updated

apport information

tags: added: apport-collected bionic uec-images
description: updated

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.19 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19-rc4

Lucian Petrut (petrutlucian94) wrote :

Hi Joseph, thanks for replying.

I've just tested the latest upstream kernel (4.19) and the issue no longer occurs. I'm attaching a log as it may help understanding the issue and pinpointing whatever patch may need to be backported.

For what is worth, I no longer get the following error after installing the latest kernel:
Sep 14 11:12:05 r07-u13 kernel: Status code returned 0xc00000c9 STATUS_NETWORK_NAME_DELETED
Sep 14 11:12:05 r07-u13 kernel: /build/linux-81MBYC/linux-4.15.0/fs/cifs/smb2maperror.c: Mapping SMB2 status code 0xc00000c9 to POSIX err -5

Note that this did not happen after an update, it was just a vm spawned on the fly to debug this issue. Also, Ubuntu 16.04 (using 4.4.0 kernel) did not have this issue.

Lucian Petrut (petrutlucian94) wrote :
tags: added: kernel-fixed-upstream
tags: added: xenial
description: updated
description: updated
tags: removed: xenial
Joseph Salisbury (jsalisbury) wrote :

Can you see if the fix in mainline was also cc'd to 4.15 stable? That latest kernel is available from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15.18/

If 4.15.18 still exhibits the bug, we can perform a "Reverse" bisect to identify the commit that fixes the bug in 4.19

Changed in linux (Ubuntu):
importance: Undecided → Medium
Changed in linux (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → Medium
Changed in linux (Ubuntu):
status: Confirmed → Triaged
tags: added: kernel-da-key
Lucian Petrut (petrutlucian94) wrote :

I just gave 4.15.18 a try and it works properly, thanks.

Joseph Salisbury (jsalisbury) wrote :

Would it be possible for you to test the proposed kernel and post back if it resolves this bug?
See https://wiki.ubuntu.compo/Testing/EnableProposed for documentation how to enable and use -proposed.

Thank you in advance!

Lucian Petrut (petrutlucian94) wrote :

Sure, I just gave the proposed kernel (4.15.0-35-generic) a try and the issue persists.

Joseph Salisbury (jsalisbury) wrote :

Thanks for testing -proposed! The 4.15.0-35 kernel contains the 4.15.18 upstream stable updates, which resolve this bug per comment #27. This might indicate this bug was introduced by an Ubuntu SAUCE patch, since it does not exist upstream.

We can perform a kernel bisect to identify the SAUCE patch that introduced this regression. To do that, we first need to identify the last Ubuntu kernel that did not have this bug, and the first version that does.

Can you test the following kernels:
4.13.0-32.35: https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa/+build/14271945

4.14.0-16.19: https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa2/+build/14223014

If the 4.13.0-32 kernel also has the bug, we will have to work further back and test older kernel versions.

Thanks in advance!

Lucian Petrut (petrutlucian94) wrote :

Both 4.13.0-32 and 4.14.0-16 worked as expected.

Joseph Salisbury (jsalisbury) wrote :
Changed in linux (Ubuntu):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Bionic):
assignee: nobody → Joseph Salisbury (jsalisbury)
status: Triaged → In Progress
Changed in linux (Ubuntu):
status: Triaged → In Progress
Lucian Petrut (petrutlucian94) wrote :

This one works as well.

Lucian Petrut (petrutlucian94) wrote :

4.15.0-20 seems to have the issue as well. I'd try in between builds but it looks like those have been removed.

Joseph Salisbury (jsalisbury) wrote :

Can you next try 4.15.0-15? It can be found at:
https://launchpad.net/ubuntu/+source/linux/4.15.0-15.16/+build/14530348

All of the Bionic kernel versions can be found at:
https://launchpad.net/ubuntu/bionic/+source/linux

From that page, you click on the version number. Then one the following page, select your arch from under the "Builds" section.

Lucian Petrut (petrutlucian94) wrote :

Thanks for the link, for some reason I was looking in the wrong place.

The issue seems to have been introduced in 4.15.0-13 (4.15.0-12 works). There is in fact a CIFS related patch[1][2][3] that was backported and is most probably the one introducing this regression.

[1] https://www.spinics.net/lists/linux-cifs/msg13861.html
[2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1747572
[3] https://bugzilla.samba.org/show_bug.cgi?id=12917

Joseph Salisbury (jsalisbury) wrote :

Thanks for the feedback. I submitted those three commits for bug 1747572. To see if they introduced the regression, I built a test kernel with those three commits reverted.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1792580

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

Lucian Petrut (petrutlucian94) wrote :

Awesome, thanks for the build. Worked as expected.

Joseph Salisbury (jsalisbury) wrote :

I built a second test kernel.

The test kernel can also be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1792580

Can you test this kernel and see if it resolves this bug?

Lucian Petrut (petrutlucian94) wrote :

This one fails:

uname -a
Linux juju-1ffc5a-3-kvm-19 4.15.0-34-generic #38~lp1792580 SMP Mon Oct 1 18:48:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

mount.cifs kernel mount options: ip=10.107.1.110,unc=\\s2dscaleout.cbs.com\glance-images,vers=3.0,uid=114,gid=119,user=glance,domain=cbs.com,pass=********
mount error(5): Input/output error

Joseph Salisbury (jsalisbury) wrote :

I built a third 'v3' test kernel.

The test kernel can also be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1792580

Can you test this kernel and see if it resolves this bug?

Lucian Petrut (petrutlucian94) wrote :

Same error with 4.15.0-34-generic #38~lp1792580SecondPatchRevertedv3

mount.cifs kernel mount options: ip=10.107.1.110,unc=\\s2dscaleout.cbs.com\glance-images,vers=3.0,uid=114,gid=119,user=glance,domain=cbs.com,pass=********
mount error(5): Input/output error

Joseph Salisbury (jsalisbury) wrote :

I built a 'v4' test kernel. The kernel has the last two of the three commits.

The test kernel can also be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1792580

Can you test this kernel and see if it resolves this bug?

Lucian Petrut (petrutlucian94) wrote :

This one fails as well: 4.15.0-34-generic #38~lp1792580SecondAndThirdPatchRevertedv4

mount.cifs kernel mount options: ip=10.107.1.110,unc=\\s2dscaleout.cbs.com\glance-images,vers=3.0,uid=114,gid=119,user=glance,domain=cbs.com,pass=********
mount error(5): Input/output error

Joseph Salisbury (jsalisbury) wrote :

I built one more test kernel. This test kernel has a pick of commits 83b7739180de1 and 661bb943a98de.

The test kernel can also be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1792580

Can you test this kernel and see if it resolves this bug?

Lucian Petrut (petrutlucian94) wrote :

This one fails with the same error.

Joseph Salisbury (jsalisbury) wrote :

To see if we are missing commits in Bionic, I built a mainline kernel that was reset to v4.15 and had a pick of commit b327a717e50698039.

The test kernel can also be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1792580

Can you test this kernel and see if it resolves this bug?

Lucian Petrut (petrutlucian94) wrote :

Tried 4.15.0-041500-generic #201810151610, still the same issue.

Joseph Salisbury (jsalisbury) wrote :

That is actually good news. That means the backport of commit b327a717e50698039 is good. It also means we are missing a commit in v4.16-rc1 from Bionic.

Can you confirm v4.16-rc1 does not exhibit the bug:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc1/

If it is good, I can perform a "Reverse" bisect to identify the missing commit.

Lucian Petrut (petrutlucian94) wrote :

Still failing. Thanks for bearing with me on this. Just to confirm the version:

uname -a
Linux juju-1ffc5a-3-kvm-19 4.16.0-041600rc1-generic #201802120030 SMP Mon Feb 12 00:31:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Joseph Salisbury (jsalisbury) wrote :

Thanks for testing. That would mean the fix is not in v4.16-rc1, but a newer kernel.

Can you test v4.17 final:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.17/

Lucian Petrut (petrutlucian94) wrote :

Same issue with 4.17.

Lucian Petrut (petrutlucian94) wrote :

Failing as well.

Joseph Salisbury (jsalisbury) wrote :

We know that v4.19-rc4 does not have the bug per comment 15. We can narrow down which release candidate has the fix. Can you test v4.19-rc1:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19-rc1/

Lucian Petrut (petrutlucian94) wrote :

Looks like there's a fix in rc3 which works, while rc2 and rc1 don't.

Joseph Salisbury (jsalisbury) wrote :

There are two cifs specific commits between -rc2 and -rc3:

395a2076b406 cifs: connect to servername instead of IP for IPC$ share
c15e3f19a6d5 fs/cifs: don't translate SFM_SLASH (U+F026) to backslash

I built a test kernel with these commits to rule them out as being the fix.

The test kernel can also be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1792580

Can you test this kernel and see if it resolves this bug?

If neither are the fix, we can perform a reverse bisect.

Lucian Petrut (petrutlucian94) wrote :

The last build works properly.

Joseph Salisbury (jsalisbury) wrote :

I built one more test kernel with only commit:
395a2076b406 cifs: connect to servername instead of IP for IPC$ share

The test kernel can also be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1792580

Can you test this kernel and see if it resolves this bug?

Lucian Petrut (petrutlucian94) wrote :

This one works, I think we pinned the fix.

Joseph Salisbury (jsalisbury) wrote :

That's great news. I'll SRU that commit, so it gets added to Bionic. Thanks for all the help testing!

Stefan Bader (smb) wrote :

Fix is from v4.19 and got picked up via stable into Cosmic/4.18. So Disco should be and stay being fixed.

Changed in linux (Ubuntu Cosmic):
importance: Undecided → Medium
status: New → Fix Released
Changed in linux (Ubuntu):
status: In Progress → Fix Released
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Committed
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Lucian Petrut (petrutlucian94) wrote :

Tried the proposed kernel (4.15.0-40-generic), worked as expected. Thanks for backporting the fix.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Launchpad Janitor (janitor) wrote :
Download full text (3.1 KiB)

This bug was fixed in the package linux - 4.15.0-42.45

---------------
linux (4.15.0-42.45) bionic; urgency=medium

  * linux: 4.15.0-42.45 -proposed tracker (LP: #1803592)

  * [FEAT] Guest-dedicated Crypto Adapters (LP: #1787405)
    - KVM: s390: reset crypto attributes for all vcpus
    - KVM: s390: vsie: simulate VCPU SIE entry/exit
    - KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
    - KVM: s390: refactor crypto initialization
    - s390: vfio-ap: base implementation of VFIO AP device driver
    - s390: vfio-ap: register matrix device with VFIO mdev framework
    - s390: vfio-ap: sysfs interfaces to configure adapters
    - s390: vfio-ap: sysfs interfaces to configure domains
    - s390: vfio-ap: sysfs interfaces to configure control domains
    - s390: vfio-ap: sysfs interface to view matrix mdev matrix
    - KVM: s390: interface to clear CRYCB masks
    - s390: vfio-ap: implement mediated device open callback
    - s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
    - s390: vfio-ap: zeroize the AP queues
    - s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
    - KVM: s390: Clear Crypto Control Block when using vSIE
    - KVM: s390: vsie: Do the CRYCB validation first
    - KVM: s390: vsie: Make use of CRYCB FORMAT2 clear
    - KVM: s390: vsie: Allow CRYCB FORMAT-2
    - KVM: s390: vsie: allow CRYCB FORMAT-1
    - KVM: s390: vsie: allow CRYCB FORMAT-0
    - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1
    - KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2
    - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2
    - KVM: s390: device attrs to enable/disable AP interpretation
    - KVM: s390: CPU model support for AP virtualization
    - s390: doc: detailed specifications for AP virtualization
    - KVM: s390: fix locking for crypto setting error path
    - KVM: s390: Tracing APCB changes
    - s390: vfio-ap: setup APCB mask using KVM dedicated function
    - s390/zcrypt: Add ZAPQ inline function.
    - s390/zcrypt: Review inline assembler constraints.
    - s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h.
    - s390/zcrypt: fix ap_instructions_available() returncodes
    - s390/zcrypt: remove VLA usage from the AP bus
    - s390/zcrypt: Remove deprecated ioctls.
    - s390/zcrypt: Remove deprecated zcrypt proc interface.
    - s390/zcrypt: Support up to 256 crypto adapters.
    - [Config:] Enable CONFIG_S390_AP_IOMMU and set CONFIG_VFIO_AP to module.

  * Bypass of mount visibility through userns + mount propagation (LP: #1789161)
    - mount: Retest MNT_LOCKED in do_umount
    - mount: Don't allow copying MNT_UNBINDABLE|MNT_LOCKED mounts

  * CVE-2018-18955: nested user namespaces with more than five extents
    incorrectly grant privileges over inode (LP: #1801924) // CVE-2018-18955
    - userns: also map extents in the reverse map to kernel IDs

  * kdump fail due to an IRQ storm (LP: #1797990)
    - SAUCE: x86/PCI: Export find_cap() to be used in early PCI code
    - SAUCE: x86/quirks: Add parameter to clear MSIs early on boot
    - SAUCE: x86/quirks: Scan all busses for early PCI quirks

 -- Thadeu Lima de Souza Cascardo <email address hidden> Thu, 15 Nov 2018 17:01:46 ...

Read more...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.