Sluggish performance with some status notifications enabled

[Jim MacKenzie]

screen-profiles results in sluggish performance when some status notification items are enabled. Default config results in 0.20-0.60 load factor (typically 0.30 when actively using apps within screen) on a PIII-1 GHz (dual CPU) 1 GiB RAM Ubuntu Intrepid system. Detaching from screen session causes load factor to drop to 0.00. (Plenty of free RAM available.)

At developer's suggestion I disabled certain status notifications and found that turning off "updates-available" causes load factor to drop to 0.01. Enabling "hostname" seems to cause increased load also. I have not experimented with other settings.

Bug reported at developer's request.

[Jim MacKenzie (jim-photojim)]

Status notification "logo" seems problematic too.

[Dustin Kirkland  (kirkland)]

What version specifically of screen-profiles are you experiencing this
problem with?

Are you connected over a network to the system? Do you have a
low-latency network connection to this system? I have perceived
screen-slowness in the past, but it's usually attributed to a crappy
wireless connection, or tethering through my cell phone. When
connected over a fast internet connection, or my home wired network,
it smokes. Even if I'm attached to a very diminutive netbook, or
virtual machine.

In the latest version in Jaunty, I have io nice'd and cpu nice'd all
status items.

updates-available is by far the most resource-intensive status
notification. It should only actually run once per hour, however.

I think you need a more scientific way of testing this performance.
The logo status item is trivial, and only runs once per day ;-)

:-Dustin

[Dustin Kirkland  (kirkland)]

I'll suggest the following for benchmarking the startup time:

 $ time screen /bin/true

That should launch a screen session, run /bin/true, and exit immediately.

On my system (with all status enabled):
real 0m0.728s

It's about 2 seconds longer, if updates-available is required to run (ie, if the cache file is over an hour out of date).

I turned everything off and I get:
real 0m0.122s

Perhaps you can use this to find out which status is performing so poorly?

Also, you can disable the 'nice' commands by editing /usr/bin/screen-profiles-status and commenting out these two lines:
 renice 10 $$ >/dev/null 2>&1 || true
 ionice -c3 -p $$ >/dev/null 2>&1 || true

See if that makes things faster for you.

:-Dustin

[Jim MacKenzie (jim-photojim)]

Running $ time screen /bin/true gives me a shell prompt (with screen-profiles running) and a comment that there is 1 zombie process. If I "exit" then I get a time report, but of course it's meaningless by this point. (The zombie process disappears when I exit the screen session.)

Version is 1.44-0ubuntu1.

Give guidance, great programmer. :) I can run whatever diagnostics are useful.

As for latency, the issue is identical irrespective of access. I normally ssh to the system in question directly via a public IP on its eth1 via cablemodem. I get similar performance and load issues if I connect via my LAN on 100BaseTX wire. I can connect via WiFi and 3G HSUPA or 2.5G EDGE if that provides additional diagnostic information, but the sluggishness and performance load issues seem local in origin (from the perspective of where screen-profiles is running).

I have root access to the machine in question, so can provide you an ssh login if you'd like to tinker yourself.

[Dustin Kirkland  (kirkland)]

Hmm, running:

 $ time screen /bin/true

Should start up screen, run /bin/true, and exit immediately, showing
how long that operation took.

You have something else funky going on.

What does your /usr/bin/screen shell script wrapper look like? Can
you attach it?

:-Dustin

[Jim MacKenzie (jim-photojim)]

I haven't edited it. This is a plain (and recent - maybe 3-month-old) Intrepid server install.

/usr/bin/screen:

#!/bin/sh
#
# screen wrapper script
# Copyright (C) 2008 Canonical Ltd.
#
# Authors: Dustin Kirkland <email address hidden>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.

# Create the screen-profiles directory, if it doesn't already exist
[ -d "$HOME/.screen-profiles" ] || mkdir -p "$HOME/.screen-profiles"

# If $HOME/.screenrc exists but $HOME/.screen-profiles/profile does not,
# this shows that the user has an existing custom screen configuration,
# and thus we will not bother them with a select-screen-profile prompt.
if [ -r "$HOME/.screenrc" -a ! -e "$HOME/.screen-profiles/profile" -a ! -h "$HOME/.screen-profiles/profile" ]; then
        exec /usr/bin/screen.real -c "$HOME/.screenrc" "$@"
        exit $?
fi

# Ensure that the user has selected a screen profile
[ -h "$HOME/.screen-profiles/profile" ] || /usr/bin/select-screen-profile
if [ ! -r "$HOME/.screen-profiles/profile" ]; then
        echo
        echo "Your selected profile is not accessible."
        echo
        echo "Either select a different profile:"
        echo " $ select-screen-profile"
        echo
        echo "Or install the extras package:"
        echo " $ sudo apt-get install screen-profiles-extras"
        echo
        exit 1
fi

# Ensure that their keybindings are seeded
[ -s "$HOME/.screen-profiles/keybindings" ] || echo "source /usr/share/screen-profiles/keybindings/common" > "$HOME/.screen-profiles/keybindings"

# Ensure that their default windows are seeded
[ -r "$HOME/.screen-profiles/windows" ] || cp "/usr/share/screen-profiles/windows/common" "$HOME/.screen-profiles/windows"

# Ensure that the user's .screenrc at least exists
[ -r "$HOME/.screenrc" ] || touch "$HOME/.screenrc"

# Now let's execute screen!
exec /usr/bin/screen.real -c "$HOME/.screen-profiles/profile" "$@"

Also, in case it's useful:

$ time screen /bin/true

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To access official Ubuntu documentation, please visit:
http://help.ubuntu.com/

  System information as of Tue Apr 14 13:00:01 CST 2009

  System load: 0.02 Memory usage: 30% Processes: 150
  Usage of /home: 26.5% of 182.88GB Swap usage: 0% Users logged in: 1

  => There is 1 zombie process.

  Graph this data and manage this system at https://landscape.canonical.com/
jim@adelaide:~$ ps aux | grep true
jim 10034 0.1 0.1 4084 1076 pts/9 S+ 13:05 0:00 /usr/bin/screen.real -c /home/jim/.screen-profiles/profile /bin/true
jim ...

[Dustin Kirkland  (kirkland)]

Aha!

Okay, please comment out any windows you have auto-opening in
~/.screen-profiles/windows. (At least just long enough to do your
benchmarking).

:-Dustin

[Dustin Kirkland  (kirkland)]

I've made some key changes that should drastically improve performance and startup times of screen.

The fixes are committed, and will be in 1.45. Please retest and leave some feedback then!

:-Dustin

[Dustin Kirkland  (kirkland)]

I was able to do a couple of low-risk things to greatly improve performance...

 1) Firstly, comment out the ionice and cpu nice calls in the screen-profiles-status wrapper. The idea is good, but this drastically slows down the initial launch of screen, which makes for a very bad experience, especially on low powered systems. I first noticed this playing with ARM. But also noted that on my high-powered laptop, screen's startup time was 1.5s vs. 0.5 seconds. I have a better approach for 'nice' on these status items in Karmic.

 2) The updates-available status is by far the slowest status item of all. I found a really neat, trivial approach to tremendously improve response time. Background the updates-available fetching and write these to a cache file, to be picked up at next run. We already support reading from cache file, so this is really a trivial fix.

 3) There was one superfluous call of lsb_release (surprisingly expensive) in the release script. Drop that.

All in all, very low-risk, high-yield fixes, IMHO.

:-Dustin

[Dustin Kirkland  (kirkland)]

Should definitely SRU, if it doesn't make the release.

:-Dustin

[Dustin Kirkland  (kirkland)]

SRU Justification:

 1. Impact: Significant impact on heavily loaded or under-powered systems; can take several seconds (up to a minute) to launch a screen session; these fixes can improve this by an order of magnitude
 2. How fixed: The fix drops the ionice and cpu nice calls which wrap all status scripts; this was added very late in the Jaunty release and is not desirable; cause screen to take a very long time to launch; also, we eliminate a couple of very expensive calls to lsb_release; finally, we drastically improve the most expensive status item (updates-available) by backgrounding the expensive run and writing that output to the cache file which we already read
 3. Patch: in the debdiff
 4. Test Case: Run 'time screen true'. On low power systems (like arm), or on heavily loaded systems, this can take 5 seconds - 60 seconds. Simply unacceptable. With this upload, all but the slowest systems should be able to launch screen within a couple of seconds (most taking less than a second).
 5. Regression potential: Low. I have tested this thoroughly. The ionice and cpu nice calls were a last minute addition and a poor choice; they don't actually do what we want. It's better to just let the system's normal scheduling mechanisms handle this.

:-Dustin

[Dustin Kirkland  (kirkland)]

Uploaded to jaunty-proposed.

:-Dustin

[Martin Pitt (pitti)]

This does not normally fall under the SRU criteria, but I ack this because (1) the package is new in Jaunty, and (2) the ionice/nice calls were added late, thus the package got tested without those during Jaunty development.

[Martin Pitt (pitti)]

Accepted screen-profiles into jaunty-proposed-proposed; please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Andreas Wenning (andreas-wenning)]

Even on my 2.0 Ghz single-core this has had a great impact. I have screen-profiles enabled on login; this would normally take ~3 seconds. After upgrading to screen-profiles in jaunty-proposed it takes less than a second.

[Dustin Kirkland  (kirkland)]

Excellent! Thanks for confirming.

:-Dustin

[Launchpad Janitor (janitor)]

This bug was fixed in the package screen-profiles - 1.44-0ubuntu1.1

---------------
screen-profiles (1.44-0ubuntu1.1) jaunty-proposed; urgency=low

  Cherry-picked fixes from upstream:
  * bin/cpu-freq: fix for powerpc LP: #352286, fix for arm LP: #362157,
    always report in GHz, round to nearest hundred
  * bin/uptime: don't print seconds, since we don't update often enough
  * Performance improvements, screen too slow to launch, LP: #360759
    - screen-profiles-status: use home bin scripts, if available first,
      then fallback to system-wide, allows for local overrides; comment
      out nice and ionice calls; this *really* slows down startup
    - bin/release: eliminate one expensive lsb_release call
    - bin/updates-available: background updates checking processes,
      let them write to cache, and let the next cache check pick up the
      changes

 -- Dustin Kirkland <email address hidden> Fri, 17 Apr 2009 12:40:29 -0500