Excessive Disconnect unmatched entries from sshd

[Ben Johnson]

[Impact]

User ssh disconnect messages in syslog aren't handled by logwatch, and thus end up in the "Unmatched Entries" section, one per event. This clutters up the logwatch reports unnecessarily.

[Test Case]

# lxc launch ubuntu-daily:cosmic tester
# lxc exec tester bash

# apt update
# apt dist-upgrade -y
# apt install -y logwatch openssh-server mailutils
  * mail configuration : Local only
  * System mail name: (use default)

# sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config
# systemctl restart sshd
# passwd ubuntu
  * choose a password
# ssh ubuntu@localhost
  * login, then exit

# logwatch --detail Med --mailto root --service all --range today
# sleep 1
# mail
  * select message 1
  * Search for SSHD:
    /SSHD

You will see unmatched entries:
   **Unmatched Entries**
   Disconnected from user ubuntu 127.0.0.1 port 53084 : 1 time(s)

[Original Description]

# lsb_release -rd
Description: Ubuntu 16.04.1 LTS
Release: 16.04

# apt-cache policy logwatch
logwatch:
  Installed: 7.4.2-1ubuntu1
  Candidate: 7.4.2-1ubuntu1
  Version table:
 *** 7.4.2-1ubuntu1 500
        500 http://mirrors.digitalocean.com/ubuntu xenial/main amd64 Packages
        500 http://mirrors.digitalocean.com/ubuntu xenial/main i386 Packages
        100 /var/lib/dpkg/status

The issue seems to be exactly as described here:

https://bugzilla.redhat.com/show_bug.cgi?id=1317620

In synopsis, Logwatch's "SSHD" output contains excessive "Unmatched Entries" regarding SSH disconnections. They look like this:

Received disconnect from 123.123.123.123 port 6887:11: disconnected by user : 1 time(s)
 Received disconnect from 123.123.123.123 port 8310:11: disconnected by user : 1 time(s)
 Disconnected from 123.123.123.123 port 1306 : 1 time(s)
 Received disconnect from 123.123.123.123 port 3720:11: disconnected by user : 1 time(s)
 Received disconnect from 123.123.123.123 port 3001:11: disconnected by user : 1 time(s)
 Disconnected from 123.123.123.123 port 1054 : 1 time(s)
 Received disconnect from 123.123.123.123 port 9741:11: disconnected by user : 1 time(s)
 Received disconnect from 123.123.123.123 port 3261:11: disconnected by user : 1 time(s)
 Received disconnect from 123.123.123.123 port 4650:11: disconnected by user : 1 time(s)
 Received disconnect from 123.123.123.123 port 13235:11: disconnected by user : 1 time(s)
 Received disconnect from 123.123.123.123 port 1065:11: disconnected by user : 1 time(s)
 Received disconnect from 123.123.123.123 port 13868:11: disconnected by user : 1 time(s)
 Disconnected from 123.123.123.123 port 8542 : 1 time(s)

I should mention that these connections are from me, and are legitimate; they are not from "bots" or other types of probes/scans that are, for example, check for the availability of vulnerable ciphers.

The key finding from the above report seems to be:

"I don't know why there are two different format disconnect messages, but the bit that seems to confuse logwatch was adding the port number to the message."

There seem to be several (3-5) such messages that result from a normal connect/disconnect cycle.

[Nish Aravamudan (nacc)]

I think this is resolved upstream, and needs some verification of which patches would need backporting. sshd changed the log formatting to include the port, which broke logwatch's regex.

[mithras (jsluiman)]

I have a variety of lines that do not fit into Logwatch' regex for sshd. They are mainly caused by the lack of a port in the formatting. There are three typical lines that are affected:

Received disconnect from [***] port [***]:11: [preauth] : 1 time(s)
Disconnected from [***] port [***] [preauth] : 1 time(s)
Connection reset by [***] port [***] [preauth] : 3 time(s)

The first line is matched by using a provided patch: https://bugzilla.redhat.com/attachment.cgi?id=1136417&action=diff
The other two still remain in my case, I manually patched /usr/share/logwatch/scripts/services/sshd for the "Received disconnect from"

What is the update for this issue? I couldn't find any details about patches from upstream, otherwise I could test those as well.

[Craig Francis (craig.francis)]

Is this likely to be fixed with a patch?

At the moment my LogWatch reports contain about 700-1000 lines per day in the SSHD "Unmatched Entries" section, which makes it difficult to read/check.

I'd rater not edit the LogWatch config files myself if it's going to be patched.

[Nish Aravamudan (nacc)]

Sorry for the delay on my end.

Here is one upstream commit that is relevant: https://sourceforge.net/p/logwatch/git/ci/0490a1c4ccf1b811bce0cd68dfce8a9e346d7648/

This will correct for: "Received disconnect from..." with a port on the end.

Another upstream commit: https://sourceforge.net/p/logwatch/git/ci/50cd9b06

This will correct for:

"Disconnected from..." with a port on the end

and

"Connection reset by..."

I can build (hopefully later today) a test package of logwatch with only the sshd changes mentioned above. I will need to evaluate the latter commit, as it includes a number of further changes than what is explicitly requested in this bug.

[Andreas Hasenack (ahasenack)]

Confirmed still happening in bionic.

[Andreas Hasenack (ahasenack)]

This was fixed in cosmic in 7.4.3+git20161207-2ubuntu2. The bug didn't auto-close because the bug number wasn't correctly mentioned in the changelog of the package (it missed a "#").

I added tasks for xenial and bionic.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brian Murray (brian-murray)]

Hello Ben, or anyone else affected,

Accepted logwatch into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/logwatch/7.4.3+git20161207-2ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Ben, or anyone else affected,

Accepted logwatch into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/logwatch/7.4.2-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Bryce Harrington (bryce)]

When sshd disconnects it leaves a pair of messages:

   Received disconnect from NN.NN.NN.NN port 42004:11: disconnected by user : 1 time(s)
   Disconnected from NN.NN.NN.NN port 42072 : 1 time(s)

The second message, "Disconnected from" is what this bug report deals with. The first message, "Received disconnect", was fixed in logwatch 7.4.3, so bionic and newer have this fix but xenial does not.

I've verified that the update in bionic-proposed displays a clean report:

 --------------------- SSHD Begin ------------------------

 SSHD Killed: 1 Time(s)

 SSHD Started: 2 Time(s)

 Failed logins from:
    127.0.0.1 (localhost): 1 time
       ubuntu/password: 1 time

 Users logging in through sshd:
    ubuntu:
       127.0.0.1 (localhost): 1 time

 Received disconnect:
    disconnected by user
       127.0.0.1 : 1 Time(s)

 ---------------------- SSHD End -------------------------

And the update in xenial-proposed is correct in that it suppresses the second message ("Disconnect..."), but the "Received disconnect" message is still present:

 --------------------- SSHD Begin ------------------------

 SSHD Killed: 1 Time(s)

 SSHD Started: 4 Time(s)

 Users logging in through sshd:
    ubuntu:
       127.0.0.1 (localhost): 1 time

 **Unmatched Entries**
 Received disconnect from 127.0.0.1 port 50276:11: disconnected by user : 1 time(s)

 ---------------------- SSHD End -------------------------

So, technically I think the fix can be validated in that it suppresses the log message as intended. However, a perfect fix for xenial would also address the additional log message. I think it could either be left to a follow up or I can respin the xenial portion of this SRU with the additional fix.

[Bryce Harrington (bryce)]

Per discussion on IRC, going to go with the SRU as-is since it's an incremental improvement that doesn't make things worse. A subsequent SRU could improve further.

[Launchpad Janitor (janitor)]

This bug was fixed in the package logwatch - 7.4.3+git20161207-2ubuntu1.2

---------------
logwatch (7.4.3+git20161207-2ubuntu1.2) bionic; urgency=medium

  [ Bryce Harrington ]
  * d/p/0020-dhcpd-Ignore-lease-age-under-threshold-messages.patch:
    dhcpd: Ignore lease age under threshold messages
    (LP: #1578001)
  * d/p/0018-audit-Treat-Denial-Errors-same-as-Denied.patch:
    audit: Treat Denial-Errors same as Denied.
    (LP: #1577948)
  * d/p/0017-audit-Apparmor-DENIED-entries-don-t-always-include-p.patch:
    audit: Apparmor DENIED entries don't always include parent=N.
    (LP: #1577948)
  * d/p/0014-zz-sys-Suppress-warnings-if-Sys-CPU-or-Sys-MemInfo-a.patch:
    zz-sys: Suppress warnings if Sys::CPU or Sys::MemInfo are missing.
    These are not installed by default in Ubuntu's logwatch packaging.
    (LP: #1890749)
  * d/p/0012-postfix-Handle-backwards-compatible-mode.patch:
    postfix: Handle backwards-compatible mode.
    (LP: #1583705)
  * d/p/0011-postfix-Ignore-Resolved-loghost-to-127.0.0.1.patch:
    postfix: Ignore Resolved loghost to 127.0.0.1.
    (LP: #1583705)
  * d/p/0010-00-debspecific-disable-su-reporting-in-secure.diff.patch:
    Use $PATH to determine location of zpool and zfs.
    (LP: #1880211)

  [ Karl Stenerud ]
  * d/p/ssh-ignore-disconnected.patch:
    sshd: ignore disconnected from user USER
    (LP: #1644057)

 -- Bryce Harrington <email address hidden> Thu, 03 Sep 2020 04:21:53 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for logwatch has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package logwatch - 7.4.2-1ubuntu1.1

---------------
logwatch (7.4.2-1ubuntu1.1) xenial; urgency=medium

  [ Bryce Harrington ]
  * d/p/0020-dhcpd-Ignore-lease-age-under-threshold-messages.patch:
      dhcpd: Ignore lease age under threshold messages
      (LP: #1578001)
  * d/p/0018-audit-Treat-Denial-Errors-same-as-Denied.patch:
    audit: Treat Denial-Errors same as Denied.
    (LP: #1577948)
  * d/p/0017-audit-Apparmor-DENIED-entries-don-t-always-include-p.patch:
    audit: Apparmor DENIED entries don't always include parent=N.
    (LP: #1577948)
  * d/p/0014-zz-sys-Suppress-warnings-if-Sys-CPU-or-Sys-MemInfo-a.patch:
    zz-sys: Suppress warnings if Sys::CPU or Sys::MemInfo are missing.
    These are not installed by default in Ubuntu's logwatch packaging.
    (LP: #1890749)
  * d/p/0012-postfix-Handle-backwards-compatible-mode.patch:
    postfix: Handle backwards-compatible mode.
    (LP: #1583705)
  * d/p/0011-postfix-Ignore-Resolved-loghost-to-127.0.0.1.patch:
    postfix: Ignore Resolved loghost to 127.0.0.1.
    (LP: #1583705)
  * d/p/0010-00-debspecific-disable-su-reporting-in-secure.diff.patch:
    Use $PATH to determine location of zpool and zfs.
    (LP: #1880211)

  [ Karl Stenerud ]
  * d/p/ssh-ignore-disconnected.patch:
    sshd: ignore disconnected from user USER
    (LP: #1644057)

 -- Bryce Harrington <email address hidden> Thu, 03 Sep 2020 04:21:50 +0000