Firefox utilizing 100% of station CPU

[Steve Smaldone]

At a high level, in certain instances we’re experiencing sustained abnormally high (100%) CPU utilization from Firefox. This utilization is competing with other critical application processes on the systems, starving them of resources, causing issues in the application processing. This happens, at times, when the browser should be idle, even.

We have not isolated this issue to be a bug with firefox itself or an issue with the webpage content being provided to the browser. Regardless, here are the symptoms:
    * CPU usage is extremely high for the FireFox process (>100% on average vs <10% average for healthy stations) even while the screen is sitting idle (no animations, user interactions, etc.)
   * We have isolated the thread within FireFox causing the most CPU usage to the Renderer thread with the SwComposite threads also consuming abnormally high CPU.
   * In the FireFox Task and Process Managers, we have validated there is no single tab or window consuming abnormally high CPU, just the FireFox master process.
   * Additionally, we have confirmed that the FireFox browser history (taken from the places.sqlite file) does not have any webpages being loaded during the times when CPU consumption experiences a step function change up or down.
   * We also have linux perf logs for the FireFox process on healthy stations and impacted stations while sitting idle on the login screen, which can provide some stack traces for the Renderer and other threads which may be more useful to someone with FireFox expertise.
   * The biggest difference is that the impacted workcells have the Renderer and Compositor 'tracks' very active while the healthy workcells see no usage of these tracks at all.
   * These 2 tracks also have supporting tracks such as SwComposite and WRRendererBackend#X. There were 2 of each of these, which may indicate the issue is happening across both FireFox windows. For reference, the stations always have 2 windows open: one for the screen UI and one for the projector on the pod face.
   * (Using the term 'track' here as defined by the open source FireFox profiler: https://profiler.firefox.com/. Basically a thread within the FireFox main process)
   * There are many memory copy and SW interrupt calls being made in these tracks.
   * This validates that FireFox is contributing to increased software interrupt load
   * Additionally, we found an interesting pair of calls occurring periodically in the Renderer track. We have not deep dived these yet, but on the off chance it matters, we're including it in this report:
   * viaduct_log_error - every 50 - 500ms
   * wgpu_render_pass_end_pipeline_statistics_query - every 50 - 1000ms

[Steve Smaldone (smaldone)]

Additional info:
Firefox is at version 100.0.2
It is installed as a deb package
We're running Ubuntu 20.04

[Sebastien Bacher (seb128)]

Thank you for your bug report. We don't really change the code over upstream and have limited resources and knowledge of the project compared to them, it would be a good idea to check if there is a similar issue reported on https://bugzilla.mozilla.org/ already and if not to open also there.

We might pick up the issue/work on it when our team capacity allows for it but upstream might help getting it resolved sooner.

[Amin Bandali (bandali)]

Hello,

The reported version of Firefox (100.0.2) is rather old at this point, and could be a potential security risk: to my knowledge, Mozilla don't backport bug fixes to older versions of Firefox (except for the current Firefox ESR series), and many security and non-security related issues have since been reported and fixed in later Firefox versions. Thus, I would highly recommend considering updating your system packages, including Firefox, to latest versions and then try again - your issue may already have been fixed in a newer version of Firefox.

If the issue still persists with up-to-date Firefox and system packages, I would echo Seb's recommendation of reporting the bug upstream at https://bugzilla.mozilla.org, and include a performance profile per https://profiler.firefox.com as well.

Searching through Bugzilla, https://bugzilla.mozilla.org/1733377 came up, but I'm not sure whether or not it's the same issue. Please feel free to comment on that and/or open a new issue.

[Nick Hummel (nhummel)]

Hi All,

I'm a developer with Steve's group. Thank you for taking a look at this and linking to that other bug! I took a look through the profiler summaries linked in the thread and it appears very similar to what we are observing. We also captured performance profiles for the current version of Firefox we're using (100.0.2), and we observed similar behavior from the Renderer, Compositor, and SwComposite tracks.

Since we are unable to change Firefox version quickly in our use case, we would like to understand if there are any mitigations or settings we could modify to prevent these symptoms. The last comment in the linked bug thread mentions that disabling and re-enabling the Compositor resolved symptoms, so we will try this out tomorrow. Longer term though, upgrading the version is absolutely something we're looking at.

In addition to testing changes with the Compositor settings, are there any parameters we could change in the about:config page? In our initial testing, we have observed CPU usage vary roughly proportionately with the layout.frame_rate parameter when these CPU throttling symptoms are present (default was -1, and when we set to 30 or 15 for example, we see the CPU usage decrease).

We can also collect additional Firefox profiler results or other logs if they would help clarify the issue/symptoms.

[Sebastien Bacher (seb128)]

@Nick, if the issue is easy to trigger in your setup would it be possible to at least test on a machine if installing a recent version of firefox would fix the issue?

as for your question about flag/profiler I don't have any suggestion offhand and would need to investigate but again you might have more eyes on the question if it's asked upstream so I would recommend adding a comment to the report Amin referenced