Large messages from client causing message-servers to crash

[Haw Loeung]

Hi,

We've found frequently that large messages from clients are causing the message app servers to crash or lock up causing a DoS. This requires restarting. We've updated the logging used on the Landscape servers as follows:

| LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %D %I" combined2

This logs the duration the request took (%D) and the no. of bytes or size (%I).

An example log showing crash:

| 93.xxx.xxx.xxx - - [05/Nov/2022:13:38:25 +0000] "POST /message-system HTTP/1.1" 502 35040 "-" "landscape-client/18.01-0ubuntu13" 563132272 699197886

Looking at the user agent, it looks legitimate.

Perhaps the landscape client itself could break up these messages into smaller ones?

[Haw Loeung (hloeung)]

The related server-side bug filed for this is LP:1994534. Not sure how best we want to handle this.

[Kevin Nasto (silverdrake11)]

https://github.com/CanonicalLtd/landscape-client/pull/126 The pull request for the client is right there.

There are two changes that need to be made, server side not to accept big messages and client side, not to send them

[Haw Loeung (hloeung)]

"not to accept", but if these are legitimate messages, wouldn't something be lost?

[Kevin Nasto (silverdrake11)]

Yes it would be lost, but out of the million of messages we receive these would be around 10 or 20 per month

[Kevin Nasto (silverdrake11)]

And they are logged so we can keep an eye on it

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 23.02-0ubuntu1

---------------
landscape-client (23.02-0ubuntu1) lunar; urgency=medium

  * New upstream release 23.02:
    - Preventing the generation of large messages and logs that can overwhelm
      Landscape Server (LP: #1995775)
    - Improved MOTD slowdown on machines with many tap network interfaces
      (LP: #2006396)
    - No longer using deprecated apt-key when storing trusted GPG keys
      (LP: #1973202)
    - Fixed issue recognising Parallels VMs as Virtual Machine clients
      (LP: #1827909)
    - Fixes for incorrect logfile rotation config (LP: #1968189)
    - Client-side backoff handling to moderate traffic to Landscape Server
      during high load (LP: #1947399)
    - Avoid sending empty messages when catching up to expected next message
      (LP: #1917540)
    - --is-registered CLI option to quickly check if client is registered
      (LP: #1912516)
    - Can now report Ubuntu Pro attachment information if the version of
      Landscape Server it is registered to supports this (LP: #2006401)
    - Packages installed as dependencies as part of package profiles are now
      appropriately autoremovable (LP: #1878957)
    - Registration timeouts give an error instead of timing out (LP: #1889464)
    - RHEV hypervisor VMs are now recognized as virtual machines (LP: #1884116)
    - Doing a Landscape-driven release upgrade from a release running python 2
      to one running python 3 no longer hangs forever (LP: #1943291)

 -- Mitch Burton <email address hidden> Wed, 08 Feb 2023 10:23:31 -0800

[Haw Loeung (hloeung)]

Can we get the fix to landscape-client backported to other releases too?

[Mitch Burton (mitchburton)]

Yes, we will be working on SRU for a number of landscape-client bugfixes.

[Launchpad Janitor (janitor)]

This bug was fixed in the package landscape-client - 23.02-0ubuntu1~20.04.2

---------------
landscape-client (23.02-0ubuntu1~20.04.2) focal; urgency=medium

  * d/p/0001-start-service-during-config.patch: fix landscape-config does not
    start landscape-client service (LP: #2040189)
  * d/landscape-sysinfo.wrapper: fix handle using cache when permissions allow
    (LP: #2040924)

landscape-client (23.02-0ubuntu1~20.04.1) focal; urgency=medium

  * Backporting release 23.02 for SRU (LP: #2006402):
    - Service is no longer stopped on upgrade (LP: #2027613)

 -- Mitch Burton <email address hidden> Wed, 14 Feb 2024 15:34:10 -0800