Client queues up lshw calls if talking to old server

Bug #1053057 reported by Andreas Hasenack on 2012-09-19
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Landscape Client
Critical
Thomas Herve
landscape-client (Ubuntu)
Undecided
Unassigned
Lucid
Undecided
Unassigned
Natty
Undecided
Unassigned
Oneiric
Undecided
Unassigned
Precise
Undecided
Unassigned

Bug Description

This is an SRU request for landscape-client in lucid, natty, oneiric and precise.

Landscape is *NOT* invoking the SRU exception this time
(https://wiki.ubuntu.com/StableReleaseUpdates#Landscape).

Explanation
===========
If a 12.04.X or 12.05 client is talking to a server which does not support the
hardware-info messages (like LDS 11.07.X), it will queue calls to /usr/bin/lshw in
memory, one per day. That in itself is not a big problem per se.

But if the server the client is talking to is then upgraded to a version that
does support hardware-info (like LDS 12.09), all those lshw calls will happen
at once, creating a storm of one lshw process per day it was talking to the old
server. This could be just a few processes, or hundreds, depending on how long
landscape-client was running and talking to the old server.

A restart will wipe that queue and is the recommended workaround: restart the
clients just before upgrading LDS.

Test case
=========
You will need one lucid machine and access to both LDS 11.07.X and 12.09. The test itself is simple, but the preparation requires some work.

- deploy LDS 11.07.X in quickstart mode
- install landscape-client 12.05 (can be on the same machine or another one with
  network access to the LDS one)
- tweak the client code to invoke the hardware-info plugin every minute instead of
  once per day. In the file /usr/share/pyshared/landscape/manager/hardwareinfo.py,
  change run_interval to 60
- copy /usr/bin/lshw to /usr/bin/lshw.orig and change /usr/bin/lshw to this script:
"""
#!/bin/bash

date >> /tmp/lshw.log
/usr/bin/lshw.orig $@
"""
- run "lshw -xml" as root once, confirm that you get XML output and that /tmp/lshw.log is updated
- register the client with LDS. AFTER THIS, DO NOT RESTART THE CLIENT AGAIN FOR ANY REASON.
- in this scenario, lshw is not being called because the server does not support that information, but the client is accumulating the calls in memory.
- leave the client talking to the server for about 10min. Create a simple script activity to make sure it's working (like, send "ifconfig" to the client)
- after 10min, adjust the sources.list line to grab LDS 12.09 and issue apt-get update followed by apt-get dist-upgrade. The upgrade will take a few minutes. Answer any dpkg conf file question with "N", keeping the original file installed.
- keep tailing -f /tmp/lshw.log and /var/log/landscape/broker.log on the client
- the broker log will get errors while LDS is down upgrading, that's normal
- once the client is able to talk to the server again, it will notice the
  server version switch and log something like this:
"""
2012-09-19 19:41:43,139 INFO [MainThread] Accepted types changed: +hardware-info computer-info operation-result memory-info mount-info text-message network-activity mount-activity custom-graph active-process-info reboot-required change-packages-result temperature apt-preferences test users reboot-required-info processor-info load-average packages unknown-package-hashes add-packages free-space action-info resynchronize package-reporter-result eucalyptus-info network-device distribution-info -hardware-inventory -client-uptime -package-locks -computer-uptime
"""
- this is the moment where the *buggy* client will spawn a lot of lshw calls, like this:
"""
root@ubuntu:/var/log/landscape# tail -F /tmp/lshw-run.log
tail: cannot open `/tmp/lshw-run.log' for reading: No such file or directory
tail: `/tmp/lshw-run.log' has appeared; following end of new file
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:43 UTC 2012
Wed Sep 19 19:41:44 UTC 2012
Wed Sep 19 19:41:44 UTC 2012
Wed Sep 19 19:41:44 UTC 2012
Wed Sep 19 19:41:44 UTC 2012
Wed Sep 19 19:41:44 UTC 2012
Wed Sep 19 19:41:44 UTC 2012
Wed Sep 19 19:41:45 UTC 2012
Wed Sep 19 19:41:45 UTC 2012
"""
- the *fixed* client will be more well behaved, with no such storm of lshw calls (note: log taken from a different test run, so timestamps won't match):
"""
root@ubuntu:~# tail -F /tmp/lshw-run.log
Wed Sep 19 20:19:23 UTC 2012
Wed Sep 19 20:20:21 UTC 2012
(and one per minute from here on)
"""

Regression potential
====================
The fix itself is unit tested, and I tested it in all ubuntu releases with the
attached debdiff and the test case above, confirming that the issue is gone.
The patch is very specific and affects hardware reporting only.

Andreas Hasenack (ahasenack) wrote :

Patch from therve confirmed working:

=== modified file 'landscape/manager/hardwareinfo.py'
--- landscape/manager/hardwareinfo.py 2012-03-15 11:13:13 +0000
+++ landscape/manager/hardwareinfo.py 2012-09-19 19:13:33 +0000
@@ -13,10 +13,13 @@
     run_immediately = True
     command = "/usr/bin/lshw"

+ def register(self, registry):
+ super(HardwareInfo, self).register(registry)
+ self.call_on_accepted(self.message_type, self.send_message)
+
     def run(self):
- self.call_on_accepted(self.message_type, self.send_message)
         return self.registry.broker.call_if_accepted(
- self.message_type, self.send_message)
+ self.message_type, self.send_message)

     def send_message(self):
         result = getProcessOutput(

Andreas Hasenack (ahasenack) wrote :

Restart workaround also confirmed working, it wipes that queue of calls to lshw.

description: updated
Thomas Herve (therve) on 2012-09-20
Changed in landscape-client:
assignee: nobody → Thomas Herve (therve)
Thomas Herve (therve) on 2012-09-20
Changed in landscape-client:
status: New → In Progress
Thomas Herve (therve) on 2012-09-20
Changed in landscape-client:
status: In Progress → Fix Committed
Andreas Hasenack (ahasenack) wrote :

debdiff for quantal

Andreas Hasenack (ahasenack) wrote :

The debdiff contains the patch applied upstream in commit 579.

tags: added: rls-q-incoming
Clint Byrum (clint-fewbar) wrote :

This is waiting in the quantal-proposed queue.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 12.05-0ubuntu2

---------------
landscape-client (12.05-0ubuntu2) quantal-proposed; urgency=low

  * Added fix for lshw storm when the client was talking to an old
    Landscape server which was then upgraded (LP: #1053057).
 -- Andreas Hasenack <email address hidden> Thu, 20 Sep 2012 15:28:37 -0700

Changed in landscape-client (Ubuntu):
status: New → Fix Released
Andreas Hasenack (ahasenack) wrote :

debdiff for lucid

Andreas Hasenack (ahasenack) wrote :

debdiff for natty

Andreas Hasenack (ahasenack) wrote :

debdiff for oneiric

Andreas Hasenack (ahasenack) wrote :

debdiff for precise

description: updated
description: updated
description: updated
description: updated
description: updated
description: updated

Hello Andreas, or anyone else affected,

Accepted landscape-client into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/landscape-client/12.05-0ubuntu1.12.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in landscape-client (Ubuntu Precise):
status: New → Fix Committed
tags: added: verification-needed
Andreas Hasenack (ahasenack) wrote :

I will start verification as soon as the other packages land in proposed. It's much easier to test them all at once against one single LDS server.

Andreas Hasenack (ahasenack) wrote :

Confirmed working for precise with the package from precise-proposed:

root@domU-12-31-39-01-5D-16:~# apt-cache policy landscape-client
landscape-client:
  Installed: 12.05-0ubuntu1.12.04
  Candidate: 12.05-0ubuntu1.12.04
  Version table:
 *** 12.05-0ubuntu1.12.04 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-proposed/main i386 Packages
        100 /var/lib/dpkg/status
     12.05-0ubuntu0.12.04 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates/main i386 Packages
     12.04.3-0ubuntu1 0
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise/main i386 Packages

Output of /tmp/lshw.log in the precise machine with the fixed client:

root@domU-12-31-39-01-5D-16:~# tail -f /tmp/lshw.log
Mon Oct 8 18:37:26 UTC 2012
Mon Oct 8 18:38:25 UTC 2012
Mon Oct 8 18:39:25 UTC 2012
Mon Oct 8 18:40:25 UTC 2012
Mon Oct 8 18:41:25 UTC 2012
Mon Oct 8 18:42:25 UTC 2012

For comparison, I registered a lucid client against the same server at the same time. This client is still buggy (no fixed version was uploaded yet to lucid-proposed). Here is /tmp/lshw.log for that buggy client:

root@ubuntu:~# tail -f /tmp/lshw.log
Mon Oct 8 18:38:47 UTC 2012
Mon Oct 8 18:38:47 UTC 2012
Mon Oct 8 18:38:47 UTC 2012
Mon Oct 8 18:38:47 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:48 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:49 UTC 2012
Mon Oct 8 18:38:50 UTC 2012
Mon Oct 8 18:38:50 UTC 2012
Mon Oct 8 18:38:50 UTC 2012

tags: added: verification-done-precise
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 12.05-0ubuntu1.12.04

---------------
landscape-client (12.05-0ubuntu1.12.04) precise-proposed; urgency=low

  * Added fix for lshw storm when the client was talking to an old
    Landscape server which was then upgraded (LP: #1053057).
 -- Andreas Hasenack <email address hidden> Tue, 25 Sep 2012 05:58:44 -0700

Changed in landscape-client (Ubuntu Precise):
status: Fix Committed → Fix Released
Brian Murray (brian-murray) wrote :

Hello Andreas, or anyone else affected,

Accepted landscape-client into oneiric-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/landscape-client/12.05-0ubuntu1.11.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in landscape-client (Ubuntu Oneiric):
status: New → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Andreas, or anyone else affected,

Accepted landscape-client into natty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/landscape-client/12.05-0ubuntu1.11.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in landscape-client (Ubuntu Natty):
status: New → Fix Committed
Changed in landscape-client (Ubuntu Lucid):
status: New → Fix Committed
Andreas Hasenack (ahasenack) wrote :

Verified the natty package from proposed:
# apt-cache policy landscape-client
landscape-client:
  Installed: 12.05-0ubuntu1.11.04
  Candidate: 12.05-0ubuntu1.11.04
  Version table:
 *** 12.05-0ubuntu1.11.04 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty-proposed/main i386 Packages
        100 /var/lib/dpkg/status

lds server switch:
2012-10-31 10:15:55,393 INFO [MainThread] Accepted types changed: +hardware-info computer-info operation-result memory-info mount-info text-message network-activity mount-activity custom-graph active-process-info reboot-required change-packages-result temperature apt-preferences test users reboot-required-info processor-info load-average packages unknown-package-hashes add-packages free-space action-info resynchronize package-reporter-result eucalyptus-info network-device distribution-info -hardware-inventory -client-uptime -package-locks -computer-uptime
2012-10-31 10:15:55,418 INFO [MainThread] Message exchange completed in 0.69s.
2012-10-31 10:16:25,424 INFO [MainThread] Ping indicates message available. Scheduling an urgent exchange.
2012-10-31 10:16:55,422 INFO [MainThread] Ping indicates message available. Scheduling an urgent exchange.
2012-10-31 10:16:55,552 INFO [MainThread] Starting urgent message exchange with https://ec2-23-22-241-133.compute-1.amazonaws.com/message-system.
2012-10-31 10:16:55,875 INFO [PoolThread-twisted.internet.reactor-1] Sent 5062 bytes and received 150 bytes in 0.32s.
2012-10-31 10:16:55,875 INFO [MainThread] Switching to normal exchange mode.
2012-10-31 10:16:55,877 INFO [MainThread] Message exchange completed in 0.33s.

lshw calls well behaved:
root@domU-12-31-39-14-1C-73:~# tail -f /tmp/lshw.log
(...)
Wed Oct 31 10:15:55 UTC 2012
Wed Oct 31 10:16:44 UTC 2012
Wed Oct 31 10:17:44 UTC 2012

Andreas Hasenack (ahasenack) wrote :

Verified the oneiric package from proposed:
root@domU-12-31-39-10-69-14:~# apt-cache policy landscape-client
landscape-client:
  Installed: 12.05-0ubuntu1.11.10
  Candidate: 12.05-0ubuntu1.11.10
  Version table:
 *** 12.05-0ubuntu1.11.10 0
        500 http://us.archive.ubuntu.com/ubuntu/ oneiric-proposed/main i386 Packages
        100 /var/lib/dpkg/status

LDS server switch:
2012-10-31 10:16:05,825 INFO [MainThread] Starting urgent message exchange with https://ec2-23-22-241-133.compute-1.amazonaws.com/message-system.
2012-10-31 10:16:06,127 INFO [PoolThread-twisted.internet.reactor-2] Sent 1560 bytes and received 1150 bytes in 0.30s.
2012-10-31 10:16:06,128 INFO [MainThread] Switching to normal exchange mode.
2012-10-31 10:16:06,130 INFO [MainThread] Accepted types changed: +hardware-info computer-info operation-result memory-info mount-info text-message network-activity mount-activity custom-graph active-process-info reboot-required change-packages-result temperature apt-preferences test users reboot-required-info processor-info load-average packages unknown-package-hashes add-packages free-space action-info resynchronize package-reporter-result eucalyptus-info network-device distribution-info -hardware-inventory -client-uptime -package-locks -computer-uptime
2012-10-31 10:16:06,135 INFO [MainThread] Message exchange completed in 0.31s.
2012-10-31 10:16:36,140 INFO [MainThread] Ping indicates message available. Scheduling an urgent exchange.
2012-10-31 10:17:06,140 INFO [MainThread] Ping indicates message available. Scheduling an urgent exchange.
2012-10-31 10:17:06,268 INFO [MainThread] Starting urgent message exchange with https://ec2-23-22-241-133.compute-1.amazonaws.com/message-system.
2012-10-31 10:17:06,473 INFO [PoolThread-twisted.internet.reactor-1] Sent 4937 bytes and received 150 bytes in 0.20s.
2012-10-31 10:17:06,474 INFO [MainThread] Switching to normal exchange mode.
2012-10-31 10:17:06,475 INFO [MainThread] Message exchange completed in 0.21s.

lshw calls well behaved:
root@domU-12-31-39-10-69-14:~# tail -f /tmp/lshw.log
Wed Oct 31 10:16:06 UTC 2012
Wed Oct 31 10:16:23 UTC 2012
Wed Oct 31 10:17:23 UTC 2012
Wed Oct 31 10:18:23 UTC 2012
Wed Oct 31 10:19:23 UTC 2012

Andreas Hasenack (ahasenack) wrote :

Verified the lucid package from proposed:
root@domU-12-31-39-02-25-D4:~# apt-cache policy landscape-client
landscape-client:
  Installed: 12.05-0ubuntu1.10.04
  Candidate: 12.05-0ubuntu1.10.04
  Version table:
 *** 12.05-0ubuntu1.10.04 0
        500 http://us.archive.ubuntu.com/ubuntu/ lucid-proposed/main Packages
        100 /var/lib/dpkg/status

LDS server switch:
2012-10-31 10:15:59,494 INFO [MainThread] Starting urgent message exchange with https://ec2-23-22-241-133.compute-1.amazonaws.com/message-system.
2012-10-31 10:15:59,893 INFO [PoolThread-twisted.internet.reactor-1] Sent 1396 bytes and received 1150 bytes in 0.40s.
2012-10-31 10:15:59,894 INFO [MainThread] Switching to normal exchange mode.
2012-10-31 10:15:59,897 INFO [MainThread] Accepted types changed: +hardware-info computer-info operation-result memory-info mount-info text-message network-activity mount-activity custom-graph active-process-info reboot-required change-packages-result temperature apt-preferences test users reboot-required-info processor-info load-average packages unknown-package-hashes add-packages free-space action-info resynchronize package-reporter-result eucalyptus-info network-device distribution-info -hardware-inventory -client-uptime -package-locks -computer-uptime
2012-10-31 10:15:59,930 INFO [MainThread] Message exchange completed in 0.44s.
2012-10-31 10:16:29,939 INFO [MainThread] Ping indicates message available. Scheduling an urgent exchange.
2012-10-31 10:16:59,937 INFO [MainThread] Ping indicates message available. Scheduling an urgent exchange.
2012-10-31 10:17:00,064 INFO [MainThread] Starting urgent message exchange with https://ec2-23-22-241-133.compute-1.amazonaws.com/message-system.
2012-10-31 10:17:00,331 INFO [PoolThread-twisted.internet.reactor-2] Sent 6049 bytes and received 150 bytes in 0.27s.
2012-10-31 10:17:00,331 INFO [MainThread] Switching to normal exchange mode.
2012-10-31 10:17:00,333 INFO [MainThread] Message exchange completed in 0.27s.

lshw calls well behaved:
root@domU-12-31-39-02-25-D4:~# tail -f /tmp/lshw.log
Wed Oct 31 10:15:59 UTC 2012
Wed Oct 31 10:16:43 UTC 2012
Wed Oct 31 10:17:43 UTC 2012
Wed Oct 31 10:18:43 UTC 2012
Wed Oct 31 10:19:43 UTC 2012
Wed Oct 31 10:20:43 UTC 2012

tags: removed: verification-needed
tags: added: verification-done
removed: verification-done-precise
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 12.05-0ubuntu1.11.10

---------------
landscape-client (12.05-0ubuntu1.11.10) oneiric-proposed; urgency=low

  * Added fix for lshw storm when the client was talking to an old
    Landscape server which was then upgraded (LP: #1053057).
 -- Andreas Hasenack <email address hidden> Tue, 25 Sep 2012 06:07:41 -0700

Changed in landscape-client (Ubuntu Oneiric):
status: Fix Committed → Fix Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 12.05-0ubuntu1.10.04

---------------
landscape-client (12.05-0ubuntu1.10.04) lucid-proposed; urgency=low

  * Added fix for lshw storm when the client was talking to an old
    Landscape server which was then upgraded (LP: #1053057).
 -- Andreas Hasenack <email address hidden> Tue, 25 Sep 2012 06:10:14 -0700

Changed in landscape-client (Ubuntu Lucid):
status: Fix Committed → Fix Released
Changed in landscape-client:
milestone: 12.09.1 → 12.11
Changed in landscape-client:
status: Fix Committed → Fix Released
Rolf Leggewie (r0lf) wrote :

natty has seen the end of its life and is no longer receiving any updates. Marking the natty task for this ticket as "Won't Fix".

Changed in landscape-client (Ubuntu Natty):
status: Fix Committed → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers