check_http inefficient and sometimes times out

[Haw Loeung]

[Impact]

 * Based on the test case numbers, and the numbers provided by the end user, it seems that memory reclaiming for the http buffer, in check_http module, is the real issue here. It turns out that, a heavily overloaded servers might be under big memory pressure and having difficulties trying to reclaim heap memory pages (flushing dirty cache/buffers, swapped_cached pages, or worst) for the check_http network buffer.

[Test Case]

 * Execute check_http with a 150MB file:

- Unpatched (~ 108 sec)

| $ /usr/lib/nagios/plugins/check_http -H archive.ubuntu.com -j GET -u /ubuntu/pool/main/l/linux-hwe-5.0/linux-hwe-5.0_5.0.0.orig.tar.gz -t 600
| HTTP OK: HTTP/1.1 200 OK - 162594115 bytes in 108.216 second response time |time=108.215539s;;;0.000000 size=162594115B;;;0

- Patched (~ 0.5 sec)

| $ /usr/lib/nagios/plugins/check_http -H archive.ubuntu.com -j GET -u /ubuntu/pool/main/l/linux-hwe-5.0/linux-hwe-5.0_5.0.0.orig.tar.gz -t 600
| HTTP OK: HTTP/1.1 200 OK - 162594115 bytes in 0.499 second response time
|time=0.498710s;;;0.000000;600.000000 size=162594115B;;;0

[Regression Potential]

 * Very little but existent for the network receiving buffer logic under check_http module.
 * Change is based in an bigger upstream commit and only a minor subset of the upstream patch is being backported.

[Other Info]

 * Original Description:

Hi,

We're frequently seeing socket timeouts with some checks using the check_http plugin. It seems this is a known issue with some fixes upstream:

| https://github.com/nagios-plugins/nagios-plugins/commit/2b38350d546ef9632ccd90e300eeaf6eda0ca32c
| https://github.com/nagios-plugins/nagios-plugins/commit/fc1bf94655aaed32eeb6fb5c33b6d093f5564bad

Think we can backport these fixes?

[Haw Loeung (hloeung)]

Tested in my PPA:

| https://launchpad.net/~hloeung/+archive/ubuntu/nagios-nrpe/+packages

With output:

| $ time ./check_http -I 127.0.0.1 -p 9103 -u /metrics -r haproxy_rate -t 50
| HTTP OK: HTTP/1.0 200 OK - 13239810 bytes in 1.157 second response time |time=1.156696s;;;0.000000;50.000000 size=13239810B;;;0
|
| real 0m1.194s
| user 0m0.045s
| sys 0m0.064s

vs.

| $ time /usr/lib/nagios/plugins/check_http -I 127.0.0.1 -p 9103 -u /metrics -r haproxy_rate -t 50
| HTTP OK: HTTP/1.0 200 OK - 13239139 bytes in 14.090 second response time |time=14.090071s;;;0.000000;50.000000 size=13239139B;;;0
|
| real 0m14.128s
| user 0m4.650s
| sys 0m8.808s

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "check_http_socket_timeouts.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Rafael David Tinoco (rafaeldtinoco)]

$ git log -1 -p 2b38350d546ef9632ccd90e300eeaf6eda0ca32c | diffstat
 check_http.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------
 1 file changed, 77 insertions(+), 50 deletions(-)

$ git describe --tags 2b38350d546ef9632ccd90e300eeaf6eda0ca32c
release-2.2.1-82-g2b38350d

----

$ git log -1 -p fc1bf94655aaed32eeb6fb5c33b6d093f5564bad | diffstat
 check_http.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

$ git describe --tags fc1bf94655aaed32eeb6fb5c33b6d093f5564bad
release-2.2.1-135-gfc1bf946

----

$ rmadison monitoring-plugins
 monitoring-plugins | 2.1.2-2ubuntu2 | xenial | source, all
 monitoring-plugins | 2.2-3ubuntu2 | bionic/universe | source, all
 monitoring-plugins | 2.2-5 | disco/universe | source, all
 monitoring-plugins | 2.2-6build2 | eoan/universe | source, all
 monitoring-plugins | 2.2-6ubuntu1 | focal/universe | source, all
 monitoring-plugins | 2.2-6ubuntu1 | groovy/universe | source, all

[Rafael David Tinoco (rafaeldtinoco)]

@Haw,

I need some clarification here. You have applied a temporary mitigation that improves check_http by not constantly freeing and allocating heap memory. You have indicated upstream patches (2 of them) dealing with the constant timeout for the checker.

monitoring-plugins 2.2 version check_http SEEMs to be close to nagios-plugins 1.5 version.

The nagios-plugins you pointed out are backportable to release-2.2.0 of NAGIOS-PLUGINS package and not monitoring-plugins. Have you tested those ? Do you need nagios-plugins backport (of those 2 patches) to test and see if it mitigates your issue ?

Per SRU guidelines I can't simply backport those 2 without a testcase and the guarantee they're addressing your issue with a successful testcase being fixed.

Waiting a feedback here.

Thanks for reporting this.

[Haw Loeung (hloeung)]

On Thu, Apr 30, 2020 at 03:01:28PM -0000, Rafael David Tinoco wrote:
> I need some clarification here. You have applied a temporary mitigation
> that improves check_http by not constantly freeing and allocating heap
> memory. You have indicated upstream patches (2 of them) dealing with the
> constant timeout for the checker.
>
> monitoring-plugins 2.2 version check_http SEEMs to be close to nagios-
> plugins 1.5 version.

Aren't the nagios-plugin packages deprecated in favor of
monitoring-plugins?

| $ apt-cache search nagios-plugins-basic
| nagios-plugins-basic - transitional dummy package

> The nagios-plugins you pointed out are backportable to release-2.2.0 of
> NAGIOS-PLUGINS package and not monitoring-plugins. Have you tested those
> ? Do you need nagios-plugins backport (of those 2 patches) to test and
> see if it mitigates your issue ?

I initially tried to apply the patches as is from upstream but it
didn't apply cleanly requiring various other things (ssl/utils, utils,
etc.). Ended up being way too large:

| https://launchpadlibrarian.net/477251813/monitoring-plugins_2.2-3ubuntu2haw4_2.2-3ubuntu2haw5.diff.gz

So took a step back and revisited the changes from upstream. The most
important bit being this:

| https://github.com/nagios-plugins/nagios-plugins/commit/2b38350d546ef9632ccd90e300eeaf6eda0ca32c#diff-458353164a55a2e140dfc531b6ac2c38L1131-R1150

So took that, applied it, and built a package with the diff being much
much smaller:

| https://launchpadlibrarian.net/477257299/monitoring-plugins_2.2-3ubuntu2_2.2-3ubuntu2haw5.diff.gz

The attached patch (9 lines) seems to be all that's required, has been
tested and is what we're currently running with to resolve the socket
timeout issues we have seen.

[Haw Loeung (hloeung)]

I set something up for you to test:

| /usr/lib/nagios/plugins/check_http -p 443 --ssl -H people.canonical.com -j GET -u '/~hloeung/tmp/lp1875517-test.txt' -r haproxy -t 60

Without the patch, it's ~25secs. With reduces that to under 1sec.

This works for testing too (~2secs vs. ~0.1secs):

| /usr/lib/nagios/plugins/check_http -H archive.ubuntu.com -j GET -u /ubuntu/ls-lR.gz -t 60

That's for a 20MBytes file. Could also pick a larger 150Mbytes file:

| $ /usr/lib/nagios/plugins/check_http -H archive.ubuntu.com -j GET -u /ubuntu/pool/main/l/linux-hwe-5.0/linux-hwe-5.0_5.0.0.orig.tar.gz -t 600
| HTTP OK: HTTP/1.1 200 OK - 162594115 bytes in 0.499 second response time |time=0.498710s;;;0.000000;600.000000 size=162594115B;;;0

With the patch vs. the following without:

| $ /usr/lib/nagios/plugins/check_http -H archive.ubuntu.com -j GET -u /ubuntu/pool/main/l/linux-hwe-5.0/linux-hwe-5.0_5.0.0.orig.tar.gz -t 600
| HTTP OK: HTTP/1.1 200 OK - 162594115 bytes in 108.216 second response time |time=108.215539s;;;0.000000 size=162594115B;;;0

[Rafael David Tinoco (rafaeldtinoco)]

Nice! Knowing that the patch is a a small subset of an upstream patch makes our life easier! Since you provided the test, we can go ahead and do the SRU with the testcase you provided. I'll sponsor the backport for you.

[Rafael David Tinoco (rafaeldtinoco)]

Haw,

I have just uploaded all SRUs to the archive. Now the SRU team will be in charge of approving/reviewing them. After, and if, they are accepted, you will be asked to verify the fixes from [proposed] repository and change tags here at this bug, from verification-needed to verification-done.

Would you be able to verify the fixes for me ? Please let me know if there is anything else you need on this.

Best Regards

Rafael D. Tinoco

[Launchpad Janitor (janitor)]

This bug was fixed in the package monitoring-plugins - 2.2-6ubuntu2

---------------
monitoring-plugins (2.2-6ubuntu2) groovy; urgency=medium

  [Haw Loeung]
  * fix check_http - was having constant socket timeouts (LP: #1875517)
  - d/p/14_check_http_socket_timeouts

 -- Rafael David Tinoco <email address hidden> Sat, 02 May 2020 06:43:05 +0000

[Steve Langasek (vorlon)]

Hello Haw, or anyone else affected,

Accepted monitoring-plugins into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/monitoring-plugins/2.2-6ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Haw, or anyone else affected,

Accepted monitoring-plugins into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/monitoring-plugins/2.2-6ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Haw, or anyone else affected,

Accepted monitoring-plugins into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/monitoring-plugins/2.2-3ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Haw Loeung (hloeung)]

Tested with the packages from *-proposed. Looks good.

Bionic:

| https://pastebin.ubuntu.com/p/gkw5JWqtN7/

Focal:

| https://pastebin.ubuntu.com/p/rJgmHhfJ6v/

[Haw Loeung (hloeung)]

Eoan:

| https://pastebin.ubuntu.com/p/gWFFFJdP2d/

[Brian Murray (brian-murray)]

The Eoan verification is actually better than the other two as it shows the problem occurring.

[Launchpad Janitor (janitor)]

This bug was fixed in the package monitoring-plugins - 2.2-6ubuntu1.1

---------------
monitoring-plugins (2.2-6ubuntu1.1) focal; urgency=medium

  [Haw Loeung]
  * fix check_http - was having constant socket timeouts (LP: #1875517)
  - d/p/14_check_http_socket_timeouts

 -- Rafael David Tinoco <email address hidden> Sat, 02 May 2020 06:43:05 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for monitoring-plugins has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package monitoring-plugins - 2.2-6ubuntu0.1

---------------
monitoring-plugins (2.2-6ubuntu0.1) eoan; urgency=medium

  [Haw Loeung]
  * fix check_http - was having constant socket timeouts (LP: #1875517)
  - d/p/14_check_http_socket_timeouts

 -- Rafael David Tinoco <email address hidden> Sat, 02 May 2020 06:39:28 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package monitoring-plugins - 2.2-3ubuntu3

---------------
monitoring-plugins (2.2-3ubuntu3) bionic; urgency=medium

  [Haw Loeung]
  * fix check_http - was having constant socket timeouts (LP: #1875517)
  - d/p/14_check_http_socket_timeouts

 -- Rafael David Tinoco <email address hidden> Sat, 02 May 2020 06:35:43 +0000