memory threshold is too high

Bug #1576639 reported by Victor Tuson Palau on 2016-04-29
72
This bug affects 14 people
Affects Status Importance Assigned to Milestone
Canonical Device Images
Undecided
Unassigned
Canonical System Image
High
Bill Filler
webbrowser-app (Ubuntu)
High
Olivier Tilloy
Xenial
Undecided
Unassigned

Bug Description

When opening new tabs in the browser, it decides if it should close existing ones based on level of free memory available in the system. My understanding is that this is currenly set to 30% percentage of memory in the device.

There are two issues with this approach and setting:
1) a percentage means that the threshold changes from device to device. While theamount of memory needed to open a webpage is not device specific. For example at 30%, the browser would close open tabs if there less than 1.2Gb mem free in a 4Gb device, but will only need 300Mb free on a 1Gb device. This is a very large difference.

2) In M10 with 2Gb it requires 600mb free. From idle, just opening the gmail in the browser, takes the avilable system memory below this. This means that in 2gb devices the threshold is never met, and the browser never runs more than one tab at the same time. this impacts user experience, specially in desktop mode. For example is not possible to open a document in the browser while attending a hangout.

How to reproduce
open a tab in the browser, go to youtube and play any video
then open a new tab

expected:
music/sound in the video continues to play

actual in m10:
sound stops playing shortly after opening the tab

[Manual test case for verification]
On various devices (desktop with large amount of physical memory, tablet, phone, …), open the browser, open a number of tabs and browse to pages known to require a large amount of memory (Gmail, G+, hangouts, youtube, Google maps, webgl games, …).
Monitor the amount of available memory (e.g. using top) and verify that when the device is getting low on memory some cleanup is performed (background tabs are being unloaded) such that the OOM killer nevers kicks in, taking down the entire application.
Verify that the mechanism is not too aggressive, i.e. that not all background tabs get unloaded all the time (having a background tab playing some audio is a good idea to easily spot when it’s being unloaded).
As this mechanism is based on heuristics, it will never be 100% perfect, but the situation should have greatly improved.

Related branches

Changed in webbrowser-app (Ubuntu):
importance: Undecided → High
Olivier Tilloy (osomon) on 2016-04-29
summary: - memory treshhold is too high
+ memory threshold is too high
Changed in webbrowser-app (Ubuntu):
status: New → Confirmed
assignee: nobody → Olivier Tilloy (osomon)
Pat McGowan (pat-mcgowan) wrote :

Lets try with 100MB and get some mileage on proposed

Changed in canonical-devices-system-image:
assignee: nobody → Bill Filler (bfiller)
importance: Undecided → High
milestone: none → 11
status: New → Confirmed
Olivier Tilloy (osomon) wrote :

I’ve changed the threshold to 100MB and tested on my M10.
This is too low, the system OOM killer kicks in before the browser gets a chance to unload background tabs:

May 2 16:25:39 ubuntu-phablet kernel: [ 3173.400686] (0)[54:kswapd0]lowmemorykiller: Killing 'webbrowser-app' (7341), adj 1, score_adj 110,
May 2 16:25:39 ubuntu-phablet kernel: [ 3173.400686] to free 550600kB on behalf of 'kswapd0' (54) because
May 2 16:25:39 ubuntu-phablet kernel: [ 3173.400686] cache 65196kB is below limit 65536kB for oom_score_adj 12
May 2 16:25:39 ubuntu-phablet kernel: [ 3173.400686] Free memory is 4344kB above reserved

Note that in my tests I had several apps open (both native and webapps, 4 of them in total) plus the browser with a dozen tabs including a live hangout, a google doc being edited, google maps, google plus, facebook and a 1080p movie trailer playing in youtube.

The last "free memory" value reported by the browser before it was killed was 237980KB, i.e. 232MB.
Note that this value is not just the free RAM, it’s the sum of the following entries from /proc/meminfo:
  MemFree
  Buffers
  Cached

(https://bazaar.launchpad.net/~phablet-team/webbrowser-app/trunk/view/head:/src/app/meminfo.cpp#L83)

Olivier Tilloy (osomon) wrote :

I raised the threshold to 300MB, but this time the system OOM kicked in while the browser was reporting 345512KB free memory (337MB), which roughly translates to 17% free memory on that device.

Evandro P. Alves (evandro-pa) wrote :

Hi

I'm not the best person to comment on this issue except for the fact that this but also affects me, but shouldn't the kernel start swapping memory?

Olivier Tilloy (osomon) wrote :

The system OOM killer typically kicks in when it’s running out of memory (including swap).

Thomas Voß (thomas-voss) wrote :

First of all: I agree with the original bug report, relying on % of total system memory is likely to over- or under-estimate the actual memory situation. A short-term solution would be to select a fixed threshold in accordance with the OOM setup configured on our kernels, thus avoiding the issues presented in #2. Mid-term, I would propose two adjustments:

(1.) The system should let applications know if a low-on-memory situation is encountered, enabling the app to free up memory *before* the kernel OOM handler kicks in.
(2.) For the browser specifically: We should fine-tune how memory is freed up/tabs are torn down. "Least recently used" is an obvious choice, but we should also factor in if a tab is playing audio/video or actively accessing the camera/microphone and adjust likelihood of being killed accordingly.

(1.) and (2.) together should solve the issue in a way that handles memory pressure situations as gracefully as possible.

Olivier Tilloy (osomon) wrote :

An approach similar to what we currently do in the browser has been proposed for WebKit, as an alternative on configurations where memory cgroups are not available, see https://bugs.webkit.org/show_bug.cgi?id=155255.

However only MemFree is considered, and there are two (arbitrary) thresholds:
 - 300MB: non critical memory pressure
 - 100MB: critical memory pressure

I am going to experiment with this approach and will report back here.

Olivier Tilloy (osomon) wrote :

I did some extensive stress-testing on an M10, an MX4 and a E4.5, all running the latest rc-proposed image. I launched a number of applications (some native, some webapps) in the background, and then opened a number of tabs in the browser app (with the mechanism to unload tabs temporarily disabled).

I observed that even when under quite some pressure (with more than 10 tabs open and active, including memory-hungry apps like google plus, hangouts, google docs, google maps, twitter, facebook), the free memory (MemFree+Buffers+Cached) rarely seems to go below 200MB (more often on the E4.5 than on the other two devices, which is expected because the E4.5 has 1GB RAM whereas the other two have 2GB RAM).

I had to push the limits quite a bit to get the system OOM killer to kick in, and it never killed the browser app, always a background app.

Those numbers and observations seem to indicate that an arbitrary threshold of 200MB should work reasonably well, instead of a percentage of the available memory. I’ll experiment further with that value.

Olivier Tilloy (osomon) wrote :

And to follow up on previous comments: with the current approach we’ll never be able to fully prevent the OOM killer from kicking in (nor would it be desirable), what we need is an acceptable threshold that gives good results (prevents the browser from being killed while in the foreground most of the time) while being conservative enough to not close too many background tabs.

Changed in webbrowser-app (Ubuntu):
status: Confirmed → In Progress
Changed in canonical-devices-system-image:
status: Confirmed → In Progress
Olivier Tilloy (osomon) wrote :

Testing the linked branch on M10, and I got several occurrences of the OOM killer killing the browser app while it was the foreground app, before the browser got a chance to unload any background tab.

That happens more often when there are fewer apps in the background (because the OOM killer has fewer apps to select from).

That’s with a threshold of 200MB. I’ve bumped that value by increments of 50MB until it reached 400MB, where it seems to be stable enough (i.e. it anticipates correctly the OOM killer by killing background tabs).

Obviously a hardcoded value of 400MB isn’t realistic on the E4.5 that only has 1GB RAM, so I’m thinking of going back to a percentage. 400MB on the M10 is 20% (keep in mind this is not the amount of free RAM at any given time, it’s the sum of MemFree+Buffers+Cached).

I’ll update the branch and test it on all my devices.

Olivier Tilloy (osomon) wrote :

400MB works much better as a threshold on my M10 indeed. However I’m seeing something fishy: whenever the browser unloads a background tab to free up some memory, a small amount of memory is indeed reclaimed (usually less than 100MB), but the browser process continues to use up a lot of memory. When browsing pictures on 500px.com, eventually all other tabs get unloaded, and the only renderer process left active is consuming around 10% memory, but the browser process is consuming up to 30% memory. After a while left idle, it goes down to < 10%, which indicates that a lot of memory has been reclaimed at once (late garbage collection maybe?), which alleviates significantly the memory pressure.

My wild guess is that the memory used by the unloaded tabs is not garbage-collected by the QML engine right away when the tabs are unloaded. This seems to be confirmed by adding an explicit call to gc() in the unload() function of BrowserTab: unloading a tab now seems to reclaim a much larger chunk of memory, and the memory used by the browser process remains under control.

Olivier Tilloy (osomon) wrote :

And 200MB seems to work well on the E4.5, so 20% seems to be a reasonable threshold overall.

However I’m still seeing an issue with the memory pressure mechanism: on a tablet (wide layout) the least recently viewed tabs are unloaded first as expected, but on a phone (narrow layout), the most recently viewed tabs are unloaded first, which is the opposite of what should happen. Looking into that.

Olivier Tilloy (osomon) wrote :

That was an easy one: an incorrect type for the timestamp meant that the value was capped at MAX_INT and never changed, so the first tab in the list was always selected for unloading. This is now fixed in the linked branch.

Olivier Tilloy (osomon) on 2016-05-10
Changed in canonical-devices-system-image:
status: In Progress → Fix Committed
Changed in webbrowser-app (Ubuntu):
status: In Progress → Fix Committed
JOnathanJOnes (jonathanjones) wrote :

What if all unloaded tabs are saved as a binary file on the hard disk and loaded from the hard disk if the user switches back to that tab. The advantage would be that there is no need to reload the tab from the net.

Is that a way to solve it?

Olivier Tilloy (osomon) wrote :

@JOnathanJOnes: this is already the case: unloaded tabs are being serialized on disk. There are several other factors at play that might trigger some network activity when reloading a tab though, depending on the cache policy requested by the server, and more generally the dynamic nature of the page. Static content will probably not issue any requests to the network when reloaded, whereas a highly dynamic web app will do.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package webbrowser-app - 0.23+16.04.20160509.3-0ubuntu1

---------------
webbrowser-app (0.23+16.04.20160509.3-0ubuntu1) xenial; urgency=medium

  [ CI Train Bot ]
  * Resync trunk.

  [ Olivier Tilloy ]
  * Fine-tune the custom memory-pressure handler, from data gathered on
    several devices. (LP: #1576639)
  * Update translation template.

 -- Olivier Tilloy <email address hidden> Mon, 09 May 2016 17:56:03 +0000

Changed in webbrowser-app (Ubuntu):
status: Fix Committed → Fix Released
Changed in canonical-devices-system-image:
status: Fix Committed → Fix Released
Olivier Tilloy (osomon) on 2016-08-17
description: updated

Hello Victor, or anyone else affected,

Accepted webbrowser-app into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/webbrowser-app/0.23+16.04.20161028-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in webbrowser-app (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed
Olivier Tilloy (osomon) wrote :

Tested and verified in an up-to-date xenial amd64 VM, with webbrowser-app 0.23+16.04.20161028-0ubuntu1.

The VM has 784MB of memory, and after opening a number of memory-hungry tabs, I can see that the oldest ones are being unloaded to free up some memory, instead of the system taking down the entire application.

tags: added: verification-done
removed: verification-needed
Steve Langasek (vorlon) wrote :

Hello Victor, or anyone else affected,

Accepted webbrowser-app into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/webbrowser-app/0.23+16.04.20161028-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Olivier Tilloy (osomon) wrote :

Tested and verified in an up-to-date xenial amd64 VM, with webbrowser-app 0.23+16.04.20161028-0ubuntu2.

The VM has 768MB of memory, and after opening a number of memory-hungry tabs, I can see that the oldest ones are being unloaded to free up some memory, instead of the system taking down the entire application.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package webbrowser-app - 0.23+16.04.20161028-0ubuntu2

---------------
webbrowser-app (0.23+16.04.20161028-0ubuntu2) xenial; urgency=medium

  * SRU for selected bug fixes:
    - LP: #1565055: support for google hangouts
    - LP: #1573017: SAML detection logic broken in webapp container
    - LP: #1572673: invalid variable access error in webapp container
    - LP: #1466427: dynamic Ubuntu version in default UA string
    - LP: #1576639: fine-tune the custom memory-pressure handler
    - LP: #1581025: secure connection icon not showing
    - LP: #1580290: only load the current tab when the session is restored
    - LP: #1577806: browser and container open under the same instance
    - LP: #1565063: dynamic Chromium version in default UA string

 -- Olivier Tilloy <email address hidden> Fri, 28 Oct 2016 09:17:33 +0000

Changed in webbrowser-app (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for webbrowser-app has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.