connection flood to port 445 on mounting cifs volume under kernel

[eladner]

This is identical to the bug reported on the Debian kernel list, only I'm running 4.4.0, not 4.[89].0

After about 15 mintues, the cifsd daemon starts flooding echo requests on port 445 to the windows server causing a constant 1MB/s load (around 10,000 packets per second) that eventually leads to system instability that forces a reboot to correct. This is repeatable and as far as I know started about a week ago (possibly with the installation of 4.4.0-75).

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=856843

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1686099

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[eladner (eric-ladner)]

The system is behind a firewall so I can't run apport-collect directly. Is there a way to run it to a file and upload separately?

[eladner (eric-ladner)]

and the "apport-bug linux" attachment isn't sufficient?

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.11 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.11-rc8

[eladner (eric-ladner)]

Maybe. This is a production system so I'll have to schedule something.

[Eric Twose (esowteric)]

I've got the same flood here, on port 445, across my LAN. If I block that port, I get a flood instead via port 139.

There was 1MB/s upload/download between my Windows 10 machine and a Raspberry Pi 3 (4.9.24-v7+).

I blocked that and had the same flood between two Raspberry Pi 3s.

The same thing happened if I fired up my Lubuntu boxes (I can't remember if it was the 4.4.0-75-generic or the 4.8.0-29-generic, and I can't re-enable Samba right now on the network, as it's "live" and I don't want to disrupt the web service).

Have had to temporarily block all 139/445 Samba traffic on all networked machines, just allowing 139 and 445 from my Windows 10 machine, which is tolerable, as I need to be able to access network shared folders.

I've attached a screenshot from Wireshark, and I also have the .pcapng capture file if needed.

[Eric Twose (esowteric)]

The screenshot above is from traffic between the two RPis, which run Raspbian (RPi + Debian).

[Dave (furiousd)]

I've got the same problem from a number of Ubuntu 16.04 machines. They're running on Hyper-V and so they've got the following installed as per Microsoft recommendations:

linux-virtual-lts-xenial linux-tools-virtual-lts-xenial linux-cloud-tools-virtual-lts-xenial

It happens a few minutes after mounting an SMB share. The share eventually disconnects. I get the errors below on the console and it starts flooding our SMB share servers. In one instance, we connect to DFS shares so it is flooding the domain controllers.

Errors seen on console:
CIFS VFS: Autodisabling the use of server inode numbers on \\server\share. This server doesn't seem to support them properly. Hard dlinks will not be recognized on this mount.
CIFS VFS: Error -104 sending data on the socket to server.

Since I have the linux-virtual-lts-xenial kernel installed is there a virtual kernel I can upgrade to or just mainline?

[Dave (furiousd)]

As a workaround, I've added a cron job that does a directory listing on the share every minute. It has not stopped the flood but it seems to be allowing the cifs client to stay connected for a longer period of time (a major annoyance when the application relies on that share being available.)

[Dave (furiousd)]

I failed to mention my kernel version: 4.4.0-75-generic #96-Ubuntu SMP

[Dave (furiousd)]

My workaround may have added some time, but eventually the share dismounted :(

[Dave (furiousd)]

I did try to install the upstream kernel linux-image-4.11.0-041100rc8-generic_4.11.0-041100rc8.201704232131_amd64.deb and linux-headers-4.11.0-041100rc8-generic_4.11.0-041100rc8.201704232131_amd64.deb but unfortunately the system barely boots and the networking is messed up after the change. I can't tell if the cifs problem is any better.

[eladner (eric-ladner)]

If you reboot (or unmount/remount) with the "ls every minute" cron job active, it should prevent the timeout on the share and not reach the keep-alive echo that starts the flood. I think it is about the 15-minute mark (of inactivity) where the client starts the echo flood.

[Dave (furiousd)]

I downgraded to linux-image-4.4.0-66-generic and no longer have the flooding issue.

[MichaelB (mrbou)]

I have 5 desktop on Ubuntu 16.04 with various official kernel 4.4, 4.8, 4.10. All this desktop are connected via CIFS to a Windows Server 2008 R2 for a simple shared folder.

After some minutes of activity i got a high CIFS flood traffic between desktop and server. Many giga per desktop are transferred!

CMD (on desktop): sudo tcpdump -p -s 0 port 445

EXAMPLE (This 5 lines below are in loop and repeated all the time):

12:10:14.778859 IP oracle.microsoft-ds > res3-ubuntu.35514: Flags [R.], seq 1, ack 43, win 0, length 0
12:10:14.778891 IP res3-ubuntu.35516 > oracle.microsoft-ds: Flags [S], seq 2388943218, win 29200, options [mss 1460,sackOK,TS val 11568532 ecr 0,nop,wscale 7], length 0
12:10:14.779104 IP oracle.microsoft-ds > res3-ubuntu.35516: Flags [S.], seq 1906593829, ack 2388943219, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 172827841 ecr 11568532], length 0
12:10:14.779118 IP res3-ubuntu.35516 > oracle.microsoft-ds: Flags [.], ack 1, win 229, options [nop,nop,TS val 11568532 ecr 172827841], length 0
12:10:14.779210 IP res3-ubuntu.35516 > oracle.microsoft-ds: Flags [P.], seq 1:43, ack 1, win 229, options [nop,nop,TS val 11568532 ecr 172827841], length 42 SMB PACKET: SMBecho (REQUEST)

[Eric Twose (esowteric)]

At https://bugzilla.kernel.org/show_bug.cgi?id=194531 Steve French writes:

"The fix was merged into mainline kernel last week, and should be backported to at least a few older kernels due to cc:stable."

Does that mean it will be fixed for folk reporting issues with Ubuntu 4.4.0-75?

[Eric Twose (esowteric)]

At GitHub raspberrypi/linux, 6by9 writes: "[The fix] was applied to the 4.4 tree 5 days ago, so should be in 4.4.64 and later. When Ubuntu bump the kernel in their repos is totally up to them."

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/log/?id=refs/tags/v4.4.65

[Dave (furiousd)]

Fix released in what version?

[Marat Khalili (mkh-t)]

4.4.0-77-generic #98-Ubuntu, the bug is still here.

[MichaelB (mrbou)]

4.8.0-52-generic #55~16.04.1-Ubuntu released today (2017-05-16) not ok - The bug is still here too! Please fix this quickly because my company network are overloaded because of this bug. Thanks!

[Joseph Salisbury (jsalisbury)]

This has been fixed upstream by the following commit:

commit 62a6cfddcc0a5313e7da3e8311ba16226fe0ac10
Author: Sachin Prabhu <email address hidden>
Date: Sun Apr 16 20:37:24 2017 +0100

    cifs: Do not send echoes before Negotiate is complete

It has been cc'd to upstream stable and has landed in Xenial and Zesty master-next via stable updates.

The Yakkety kernel is 4.8 based, which is not maintained upstream anymore, so I'll submit an SRU request for it to be included in the Ubuntu kernels.

[MichaelB (mrbou)]

@Joseph Salisbury (jsalisbury)

Can you please tell me from which version of ubuntu kernel this is applied. I tried also the last 4.4 on Ubuntu 16.04 (4.4.0-78-generic #99-Ubuntu SMP) and i have the same problem.

Thanks

[Joseph Salisbury (jsalisbury)]

@MichaelB, The commit is in Xenial master-next. It has not been tagged for a specific version. It looks like it will be in 4.4.0-80.

[Joseph Salisbury (jsalisbury)]

I built a Yakkety test kernel with a cherry pick of commit 62a6cfddcc. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1686099/

Before I SRU this commit to Yakkety, can folks affected by this bug test this kernel?

[MichaelB (mrbou)]

@Joseph Salisbury (jsalisbury)

I installed your patched kernel on my Desktop today and it run now from 3 hours without problem, no more cifs flood.. :)

So, i will continue my test today, but apparently this bug is fixed in your patched kernel.

If is confirmed, can you tell me approximatively when this kernel will appear in the main ubuntu repository? (4.4, 4.8, 4.10)

[Janosch Machowinski (jmachowinski)]

Hi,
I also ran the kernel for the past three days, and no
connection spam appeared.

[Kleber Sacilotto de Souza (kleber-souza)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-yakkety' to 'verification-done-yakkety'. If the problem still exists, change the tag 'verification-needed-yakkety' to 'verification-failed-yakkety'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Janosch Machowinski (jmachowinski)]

Had 4.8.0-55-generic #58~16.04.1-Ubuntu the whole day, now connection spam appeared.

[Janosch Machowinski (jmachowinski)]

Uhm, sorry too late in the evening...
What I wanted to say is: I tried the proposed kernel (4.8.0-55-generic #58~16.04.1-Ubuntu)
the whole day long, and NO connection spam appeared. This means the fix seems to work.
As I don't know how to set the tag, or have insufficient rights, please modify the tag
to verification-done-yakkety.

[MichaelB (mrbou)]

Is ok for me, 4.8.0-55-generic #58~16.04.1-Ubuntu SMP
No more flood with this kernel version.

[woldemar (vladimir.v)]

when the problem will be solved in a stable kernel ?

4.8.0-56-generic #61~16.04.1-Ubuntu SMP
The bug is still there

[Andrij Abyzov (drolevar)]

I also confirm that the issue is here with
4.8.0-56-generic #61~16.04.1-Ubuntu

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.8.0-58.63

---------------
linux (4.8.0-58.63) yakkety; urgency=low

  * linux: 4.8.0-58.63 -proposed tracker (LP: #1700533)

  * CVE-2017-1000364
    - Revert "UBUNTU: SAUCE: mm: Only expand stack if guard area is hit"
    - Revert "mm: do not collapse stack gap into THP"
    - Revert "mm: enlarge stack guard gap"
    - mm: vma_adjust: remove superfluous confusing update in remove_next == 1 case
    - mm: larger stack guard gap, between vmas
    - mm: fix new crash in unmapped_area_topdown()
    - Allow stack to grow up to address space limit

linux (4.8.0-57.62) yakkety; urgency=low

  * linux: 4.8.0-57.62 -proposed tracker (LP: #1699035)

  * CVE-2017-1000364
    - SAUCE: mm: Only expand stack if guard area is hit

  * CVE-2017-7374
    - fscrypt: remove broken support for detecting keyring key revocation

  * CVE-2017-100363
    - char: lp: fix possible integer overflow in lp_setup()

  * CVE-2017-9242
    - ipv6: fix out of bound writes in __ip6_append_data()

  * CVE-2017-9075
    - sctp: do not inherit ipv6_{mc|ac|fl}_list from parent

  * CVE-2017-9074
    - ipv6: Prevent overrun when parsing v6 header options

  * CVE-2017-9076
    - ipv6/dccp: do not inherit ipv6_mc_list from parent

  * CVE-2017-9077
    - ipv6/dccp: do not inherit ipv6_mc_list from parent

  * CVE-2017-8890
    - dccp/tcp: do not inherit mc_list from parent

  * extend-diff-ignore should use exact matches (LP: #1693504)
    - [Packaging] exact extend-diff-ignore matches

  * APST quirk needed for Intel NVMe (LP: #1686592)
    - nvme: Quirk APST on Intel 600P/P3100 devices

  * regression: the 4.8 hwe kernel does not create the
    /sys/block/*/device/enclosure_device:* symlinks (LP: #1691899)
    - scsi: ses: Fix SAS device detection in enclosure

  * datapath: Add missing case OVS_TUNNEL_KEY_ATTR_PAD (LP: #1676679)
    - openvswitch: Add missing case OVS_TUNNEL_KEY_ATTR_PAD

  * connection flood to port 445 on mounting cifs volume under kernel
    (LP: #1686099)
    - cifs: Do not send echoes before Negotiate is complete

  * Support IPMI system interface on Cavium ThunderX (LP: #1688132)
    - i2c: octeon: Rename driver to prepare for split
    - i2c: octeon: Split the driver into two parts
    - [Config] CONFIG_I2C_THUNDERX=m
    - i2c: thunderx: Add i2c driver for ThunderX SOC
    - i2c: thunderx: Add SMBUS alert support
    - i2c: octeon,thunderx: Move register offsets to struct
    - i2c: octeon: Sort include files alphabetically
    - i2c: octeon: Use booleon values for booleon variables
    - i2c: octeon: thunderx: Add MAINTAINERS entry
    - i2c: octeon: Fix set SCL recovery function
    - i2c: octeon: Avoid sending STOP during recovery
    - i2c: octeon: Fix high-level controller status check
    - i2c: octeon: thunderx: TWSI software reset in recovery
    - i2c: octeon: thunderx: Remove double-check after interrupt
    - i2c: octeon: thunderx: Limit register access retries
    - i2c: thunderx: Enable HWMON class probing

  * CVE-2017-5577
    - drm/vc4: Return -EINVAL on the overflow checks failing.

  * Merlin SGMII fail on Ubuntu Xenial HWE kernel (LP: #1686305)
    - net: phy: marvell: fix Marvell 88E1512 u...

[Eric Twose (esowteric)]

Hi, it may have been fixed in 16.10 yakkety 4.8.0-58.63, but 16.04.1 xenial (LTS) still has 4.8.0-56 as latest on one of my boxes, as woldemar and Andrij Abyzov report above.

[Eric Twose (esowteric)]

(A couple of other boxes with 16.04 (not 16.04.1) xenial have 4.4.0-81 as latest).

[Marat Khalili (mkh-t)]

Hello, is it going to appear in 4.4? I'm getting tired of running 4.4.0-66 (jk works ok but still would prefer to update).

[Casper (casper-cg)]

Has this bug been fixed in Ubuntu 18.04?
I seem to have the same issue (kernel 4.15.0-34).

I do not have any SMB drives mounted, yet old connections from several months ago are open and still flooding 445:

> sudo netstat -tnp | grep 445
tcp 0 0 x.x.x.x:54508 a.a.a.a:445 ESTABLISHED -
tcp 0 0 x.x.x.x:60350 b.b.b.b:445 ESTABLISHED -
tcp 0 0 x.x.x.x:56534 c.c.c.c:445 ESTABLISHED -