browsing samba share got NT_STATUS_IO_TIMEOUT only on specific subfolder inside share

[Andrea]

browsing samba share fail NT_STATUS_IO_TIMEOUT on client side only on specific subdirectories inside a share

looking at wireshark on client side, the server miss SMB2 'Find Response' after a 'Find Request File: Index2/Enterprise/P2 SMB_FIND_ID_BOTH_DIRECTORY_INFO Pattern: *' request

in this case listing the folder path 'Index2/Enterprise/P2' under 'mynetworkshare' share from a client side, got the timeout

looking on serverside for smbd processes

 ps -aefw | grep smbd
usera 11356 24126 0 12:27 ? 00:00:00 /usr/sbin/smbd --foreground --no-process-group
root 18586 24126 0 16:19 ? 00:00:00 /usr/sbin/smbd --foreground --no-process-group
root 24126 1 0 Oct03 ? 00:00:00 /usr/sbin/smbd --foreground --no-process-group
root 24128 24126 0 Oct03 ? 00:00:09 /usr/sbin/smbd --foreground --no-process-group
root 24129 24126 0 Oct03 ? 00:00:00 /usr/sbin/smbd --foreground --no-process-group
root 24139 24126 0 Oct03 ? 00:00:00 /usr/sbin/smbd --foreground --no-process-group
root 24499 24126 0 Oct03 ? 00:03:02 /usr/sbin/smbd --foreground --no-process-group
root 24615 24126 0 Oct03 ? 00:02:49 /usr/sbin/smbd --foreground --no-process-group
usera 25581 1 0 Sep21 ? 00:00:13 /usr/sbin/smbd --foreground --no-process-group
root 28380 8773 0 16:39 pts/0 00:00:00 grep --color=auto smbd
usera 28410 24126 0 14:19 ? 00:00:00 /usr/sbin/smbd --foreground --no-process-group
usera 30280 24126 0 14:24 ? 00:00:27 /usr/sbin/smbd --foreground --no-process-group

and check 'usera' pids by lsof and filtering for samba document root, seems to find files involved on the directory listing involved with the timout

lsof -p 11356,25581,28410,30280 | grep opt\/samba

smbd 11356 usera cwd DIR 0,51 4096 13038926575565132978 /opt/samba/share/mynetworkshare
smbd 11356 usera 10r DIR 0,51 4096 10381959745566703383 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2
smbd 25581 usera cwd DIR 0,51 4096 13038926575565132978 /opt/samba/share/mynetworkshare
smbd 25581 usera 27r REG 0,51 852 12979494917769495112 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/livelink.22
smbd 25581 usera 39u REG 0,51 11 13757545946466167183 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/livelink.ctl
smbd 25581 usera 40r REG 0,51 1583664 9704199600817284546 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/accumLog.1
smbd 25581 usera 44r REG 0,51 4063910 12036954814899122047 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODaccumLog.1
smbd 25581 usera 46u REG 0,51 105309 9637570933862580747 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/coreobj.dat
smbd 25581 usera 47u REG 0,51 305137 11017905758939296307 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/coreoff.dat
smbd 25581 usera 49u REG 0,51 8887 10135057150036388979 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/coreskip.idx
smbd 25581 usera 50r REG 0,51 4873203 11065466833306867067 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODcheckLog.0
smbd 25581 usera 51u REG 0,51 1849 13073850013481250157 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/coreidx1.idx
smbd 25581 usera 52u REG 0,51 43616 11417437259087223145 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/otherobj.dat
smbd 25581 usera 53u REG 0,51 116513 12964205646912969054 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/otheroff.dat
smbd 25581 usera 54u REG 0,51 3811 13644200039721141848 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/otherskip.idx
smbd 25581 usera 55u REG 0,51 910 12320753658158512067 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/otheridx1.idx
smbd 25581 usera 56u REG 0,51 0 9947148168314732658 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/regionobj.dat
smbd 25581 usera 57u REG 0,51 0 12486901350105603972 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/regionoff.dat
smbd 25581 usera 58u REG 0,51 0 12078852358539564894 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/regionskip.idx
smbd 25581 usera 59u REG 0,51 0 11326616914780603369 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/1/regionidx1.idx
smbd 25581 usera 61u REG 0,51 268082 9273695434388025801 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/coreobj.dat
smbd 25581 usera 62u REG 0,51 456013 11946849653214943722 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/coreoff.dat
smbd 25581 usera 63u REG 0,51 23600 10631198232544745061 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/coreskip.idx
smbd 25581 usera 64u REG 0,51 135435 11253899111274063607 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/coreidx1.idx
smbd 25581 usera 65u REG 0,51 537212 13570599415659801343 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/otherobj.dat
smbd 25581 usera 66u REG 0,51 893889 13354021437680018367 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/otheroff.dat
smbd 25581 usera 67u REG 0,51 40645 12910687493492508655 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/otherskip.idx
smbd 25581 usera 68u REG 0,51 836295 9874230618985030580 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/otheridx1.idx
smbd 25581 usera 69u REG 0,51 5301 9350909775579182602 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/otheridx2.idx
smbd 25581 usera 70u REG 0,51 0 10480458378791946435 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/regionobj.dat
smbd 25581 usera 71u REG 0,51 0 10536849453028932340 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/regionoff.dat
smbd 25581 usera 72u REG 0,51 0 11466714005613920107 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/regionskip.idx
smbd 25581 usera 73u REG 0,51 0 10594787178114352353 /opt/samba/share/mynetworkshare/Index2/Enterprise/P2/MODindex/1/regionidx1.idx
smbd 28410 usera cwd DIR 0,51 4096 13038926575565132978 /opt/samba/share/mynetworkshare
smbd 30280 usera cwd DIR 0,51 4096 13038926575565132978 /opt/samba/share/mynetworkshare
smbd 30280 usera 37u REG 0,51 11 11371402119174226161 /opt/samba/share/mynetworkshare/Index1/Enterprise/P1/livelink.ctl
smbd 30280 usera 38u REG 0,51 10 11244458627038375594 /opt/samba/share/mynetworkshare/Index3/Elib/P1/livelink.ctl

doing a strace on pid 25581, cause also strace to stuck into, and the only way to detached il to do a kill -9 of strace pid

restarting samba solve the issue.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: samba 2:4.7.6+dfsg~ubuntu-0ubuntu2.2
ProcVersionSignature: Ubuntu 4.15.0-1020.20-aws 4.15.18
Uname: Linux 4.15.0-1020-aws x86_64
ApportVersion: 2.20.9-0ubuntu7.3
Architecture: amd64
Date: Thu Oct 4 16:26:59 2018
Ec2AMI: ami-ea9ab101
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-central-1a
Ec2InstanceType: t2.2xlarge
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ExecutablePath: /usr/sbin/smbd
ProcEnviron:
 LANG=C.UTF-8
 PATH=(custom, no user)
SambaServerRegression: Yes
SmbConfIncluded: No
SourcePackage: samba
TestparmExitCode: 0
UbuntuFailedConnect: Yes
UpgradeStatus: No upgrade log present (probably fresh install)

[Andrea (andrelisci)]

samba is run under ctdb 2 node cluster and the samba document root is over a gluster volume dual replica

[Andreas Hasenack (ahasenack)]

Thanks for filing this bug in Ubuntu.

You seem to be hitting a performance issue, on a non-trivial setup. This is too broad for a bug report. I suggest you seek help in the samba mailing list at https://lists.samba.org/archive/samba/.

In the meantime, I'll mark this bug as incomplete.

[Andreas Hasenack (ahasenack)]

Sorry, a better url for the samba users mailing list is https://lists.samba.org/mailman/listinfo/samba

[Launchpad Janitor (janitor)]

[Expired for samba (Ubuntu) because there has been no activity for 60 days.]