[snap] suggestion: alert users when the snap has been refreshed while running

Bug #1864901 reported by Olivier Tilloy on 2020-02-26
This bug affects 9 people
Affects Status Importance Assigned to Milestone
chromium-browser (Ubuntu)
Olivier Tilloy

Bug Description

When the chromium snap is being automatically refreshed to a newer revision while the app is running, undefined behaviour might happen, as well as data loss (because the profile folder is versioned). This issue is regularly being reported (see e.g. the duplicates of bug #1616650).

Jamie has an interesting suggestion to mitigate this situation, until refresh app awareness becomes default:

    « […] it would be possible for you to run something in the background that could occasionally see if the current symlink changed, and then alert to restart

    whenever refresh app awareness lands, it could be converted to see if something is pending, if so, prompt/alert to stop to have updates applied, or something

    simple idea is to nohup a script in a wrapper before invoking chromium proper

    iirc, firefox used to have UX built into it letting the user know it needed to be restarted, so there is some precedent for this sort of thing »

Olivier Tilloy (osomon) wrote :
Changed in chromium-browser (Ubuntu):
status: Triaged → In Progress
Olivier Tilloy (osomon) wrote :
Changed in chromium-browser (Ubuntu):
status: In Progress → Fix Committed
Adrien Jarthon (bigbourin) wrote :

I just noticed hundreds of chromium.launcher processes running on my machine with `sleep 60` and found that it's because of this fix. Looks like this background process is started but nerver stopped? nor does it checks if it's already running. This is especially visible in my case because I'm running automated tests starting chrome using capybara, and it'll be the same for everybody doing integration tests with chrome I guess.

Maybe a good fix would be to check if this background process is already running to not start it again?


Olivier Tilloy (osomon) wrote :

I can confirm this undesirable side effect. The background process that checks whether an update was installed should die when the corresponding chromium instance is terminated.

Changed in chromium-browser (Ubuntu):
status: Fix Committed → Triaged
Bèr Kessels (berkes) (ber) wrote :

It also occurs with a freshly installed chromium.

It can be easily reproduced:

$ pgrep chromium | xargs kill # kill any running chromium (beware!)
$ pgrep chromium | wc -l
0 # There are non running
$ /snap/bin/chromium --product-version # command quits immediately.
$ pgrep chromium | wc -l
$ /snap/bin/chromium # opens a browser.
# Quit browser manually.
$ pgrep chromium | wc -l
$ ps faux | grep chromium # check any processes running.
\_/bin/sh /snap/chromium/1100/bin/chromium.launcher --product-version
\_/bin/sh /snap/chromium/1100/bin/chromium.launcher

This shows that a process remains active after closing or running.
The `ps faux` shows the leftover running processes and clearly shows they no longer have a parent: it seems the parent process forks, then quits, but leaves its child running.

Above comment comes from a bug that this behaviour introduced in a project that uses chrome: https://github.com/titusfortner/webdrivers/issues/168

So, besides it leaving around running processes (slightly messy), it causes bugs (hanging) in scripts and tools that wait for chrome+children to exit.

Lothar Braun (constcast) wrote :

While it's not a beautiful solution, the attached patch does terminate the lingering process for me.

The sleep started by the symlink check is not killed, but will terminate after 60 seconds.

The attachment "1864901_termination_check_current_symlink.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Olivier Tilloy (osomon) wrote :

Thanks for your patch Lothar. I think your solution would work, although I went for a slightly different approach: https://git.launchpad.net/~chromium-team/chromium-browser/+git/snap-from-source/commit/?id=eb7aefe53b2c4710c030e04947ae66ced8fe9d3b.

(I am aware that pids can get recycled so it's not 100% correct, but it should work in most cases, and avoids loosing the exec call in the common code path (no temp profile)).

Comments and suggestions welcome, of course.

Changed in chromium-browser (Ubuntu):
status: Triaged → Fix Committed
Mike Gresens (msg-72) wrote :

I get a notification if snap updates chromium in background.
But clicking the notification does nothing - only the notifcation disappears.
No restart of all chromium instances - so not really useful...

Olivier Tilloy (osomon) wrote :

Indeed it's only an informative notification, it won't take any action by itself.

Adrien Jarthon (bigbourin) wrote :

For the record I recently noticed the 30 leaked processes started taking all my CPU (all at the same time, maybe when an update was downloaded). They were all running in infinite loop and I couldn't see the "sleep 60" in their children any more in htop. I didn't investigate more as I had to quickly kill them to recover my machine. This is really too experimental and dangerous.

Benjamin Schmid (benbuntu) wrote :

This feature caused me leaking processes.
See Bug: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1874973

Lothar Braun (constcast) wrote :

Olivier, I wasn't able to test your patch. But I like your approach better than mine.

Olivier Tilloy (osomon) wrote :

The problem with the numerous background sleep processes should be fixed in the stable channel now. Please shout here if you're still seeing it (note that existing background processes before refreshing to the latest version won't be cleaned up).

Changed in chromium-browser (Ubuntu):
status: Fix Committed → Fix Released
Alexey Bazhin (baz-irc) wrote :

I'm still seeing the problem.

tarantula!root# snap refresh
All snaps up to date.

tarantula!root# snap list | grep chromium
chromium 81.0.4044.122 1119 latest/stable canonical* -

Olivier Tilloy (osomon) wrote :

revision 1119 is not up-to-date, the latest for amd64 is now 1135.

Alexey Bazhin (baz-irc) wrote :

tarantula!root# snap refresh
All snaps up to date.

How do I get the "latest" ?

Olivier Tilloy (osomon) wrote :

That may sound like a dumb question, but are you connected to the network?

What's the output of the following command?

    snap info --abs-time chromium

Alexey Bazhin (baz-irc) wrote :

tarantula!root# snap refresh
core18 20200427 from Canonical✓ refreshed

tarantula!root# snap refresh
All snaps up to date.

tarantula!root# snap list | grep chromium
chromium 81.0.4044.122 1119 latest/stable canonical* -

tarantula!root# snap info --abs-time chromium
name: chromium
summary: Chromium web browser, open-source version of Chrome
publisher: Canonical✓
store-url: https://snapcraft.io/chromium
contact: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bugs?field.tag=snap
license: unset
description: |
  An open-source browser project that aims to build a safer, faster, and more stable way for all
  Internet users to experience the web.
  - chromium.chromedriver
  - chromium
snap-id: XKEcBqPM06H1Z7zGOdG5fbICuf8NWK5R
tracking: latest/stable
refresh-date: 2020-04-25T11:23:52+03:00
  latest/stable: 81.0.4044.129 2020-04-29T17:25:51Z (1135) 161MB -
  latest/candidate: 81.0.4044.129 2020-04-28T18:04:52Z (1135) 161MB -
  latest/beta: 83.0.4103.23 2020-04-25T02:07:58Z (1125) 163MB -
  latest/edge: 84.0.4128.3 2020-04-29T05:50:21Z (1136) 164MB -
installed: 81.0.4044.122 (1119) 161MB -

Alexey Bazhin (baz-irc) wrote :

tarantula!root# ps axu | grep chr
root 561745 0.0 0.0 6436 728 pts/2 S+ 15:20 0:00 grep chr

tarantula!root# snap refresh chromium
error: cannot refresh "chromium": snap "chromium" has running apps (chromium)

Had to restart snapd to update chromium snap.

Bèr Kessels (berkes) (ber) wrote :

The issues with dependant scripts like chromerunner are not solved either.

Were external scripts considered when building this "solution" at all? External scripts, running chromium, obviously, want the process they spawn to exit -including any children that that process might spawn-.

Breaking this, is bad.

If, for some reason, different launch-patterns are required, such as update processes, CLI-processes, scripts or user-interaction, could we not introduce *standalone* scripts for that?

E.g. "update-chromium.sh" that would do all the hackish-run-in-background-with-sleep. And avoid cluttering the generic runner with that?
Or, if that *has* to be included in the generic launching, can we then introduce a `chromium-cli` or somesuch that does *not* have all these hacks for during updates in place?

Olivier Tilloy (osomon) wrote :

The sleep process will outlive the launcher process at most 60 seconds, after which it will exit.

Note that this is only a temporary helper, until background refreshes of snaps are properly notified to running snaps by snapd through a well defined API.

Ebuzer Taha Kanat (ebuzer) wrote :

Here is a solution suggestion why not storing version independent files such as Downloads, profile and other in common directory since it doesn't change between versions and keep version dependent files in version directories?

Just like how virtual machines and containers work, storage is linked to new VM for files which scopes are beyond VM lifespan and per VM temporary files lives in VM itself.

Ebuzer Taha Kanat (ebuzer) wrote :

I tested and snap allows files in /home/user/snap/chromium/common to be accessed via chromium. What will be the down side of creating .config, Download, Documents, Videos etc. in that. From my previous experience i know that simply copying chromium directory in .config is okay to migrate profile to new chromium version.

This issue causes `ubuntu-bug chromium-browser` to hang.

Additionally, isn’t this an issue that’s best solved within snapd itself?

Olivier Tilloy (osomon) wrote :

@Ebuzer: Downloads are already stored by default outside of the versioned directory, in the user's XDG downloads directory (typically $HOME/Downloads). The rationale for versioning the profile directory is that the storage format isn't backwards compatible, so downgrading a snap to a previous revision would potentially render a profile unusable.

@Roman: this is indeed something that, longer-term, should be addressed in snapd itself. This is merely a stop-gap measure to mitigate the problem until this happens. Note that ubuntu-bug hanging sounds unrelated to this particular issue. Can you elaborate on what you're trying to do, and how it hangs?

Nick Grimshaw (nickgrim) wrote :

> The sleep process will outlive the launcher process at most 60 seconds, after which it will exit.

A problem I'm seeing is that the sleep process is sometimes *blocking* progression of other scripts e.g. if you pipe the output of a chromium process to something else:

$ snap refresh
All snaps up to date.

$ snap list | grep chromium
chromium 81.0.4044.138 1143 latest/stable canonical* -

$ time chromium --product-version

real 0m0.206s
user 0m0.107s
sys 0m0.110s

$ time chromium --product-version | tee /dev/null

real 1m0.161s # <--- !!!
user 0m0.122s
sys 0m0.078s

I came here from https://github.com/titusfortner/webdrivers/issues/168 as mentioned in #5, and the situation there is slightly worse, for reasons that aren't clear to me. That Ruby code is attempting to read the Chromium version through a pipe, very much like the following code, and this blocks *indefinitely*, not just for 60 seconds:
$ irb
irb(main):001:0> pipe = IO.popen(['chromium','--product-version'])
=> #<IO:fd 7>
irb(main):002:0> pipe.read
 … hangs indefinitely …

Olivier Tilloy (osomon) wrote :

I can confirm both Roman's observation in comment #25, and Nick's in comment #27.
Seeing that there are real problems with the implementation, I'm going to revert it for now, until I can work out a better solution (or, ideally, a better mechanism is implemented in snapd itself).

Adolfo Jayme (fitojb) on 2020-06-15
Changed in chromium-browser (Ubuntu):
status: Fix Released → Triaged
Olivier Tilloy (osomon) wrote :

Note that as a mitigation to bug #1616650, the chromium profile directory, which was previously versioned in $SNAP_USER_DATA is now under $SNAP_USER_COMMON, so problems should be much less visible to end users.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.