g_thread_new() fails with "creating thread 'gps thread': Error creating thread: Resource temporarily unavailable".

[Russell Grokett]

I ran into a bug causing foxtrotgps to crash due to out-of-threads error on Debian Buster when I happened to disconnect the GPS and let it run overnight (it can take hours before failure!). I see numerous GLib-CRITICAL messages concerning the gps_functions.c connecting to gpsd (not unexpected!) but then eventually, a GLib-ERROR g_thread_new() fails with "creating thread 'gps thread': Error creating thread: Resource temporarily unavailable".

This appears only in v1.2.2 using the new GLIB g_thread_new().

When I reverted to v1.2.1 with g_thread_create() I do not see this issue.

This appears to affect several (all?) threads being used as I was able to recreate with other threaded operations, like map tile download thread.

My best guess is that g_thread_exit() or g_thread_unref() isn't being called when the thread is no longer needed and every call to g_thread_new() is creating yet another thread.

Strangely, the GLIB docs say that the thread should free when the function returns, but that doesn't appear to me to be the case. (Maybe they mean it is free to be reused, but is not released and unrefed?)

I haven't worked with GLIB before, but I assume that wherever g_free() is being called, it may need g_thread_exit() or g_thread_unref()?

Reading GLIB info from:
https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-new

----- EXAMPLE ERROR MESSAGE -------

(foxtrotgps:18515): GLib-CRITICAL **: 11:22:28.267: Source ID 29030 was not found when attempting to remove it
connection to gpsd SUCCEEDED

(foxtrotgps:18515): GLib-CRITICAL **: 11:23:28.355: Source ID 29157 was not found when attempting to remove it

(foxtrotgps:18515): GLib-ERROR **: 11:23:28.355: creating thread 'gps thread': Error creating thread: Resource temporarily unavailable
Trace/breakpoint trap

$

Another...

(foxtrotgps:10300): GLib-ERROR **: 10:22:46.899: creating thread 'download thread': Error creating thread: Resource temporarily unavailable
Trace/breakpoint trap

$

gps_functions.c:
#if GLIB_CHECK_VERSION(2,34,0)
        g_thread_new("gps thread", &get_gps_thread, (gpointer)NULL);
#else
        g_thread_create(&get_gps_thread, NULL, FALSE, NULL);
#endif
}

Currently, I am testing the following change:

        g_thread_unref(g_thread_new("gps thread", &get_gps_thread, (gpointer)NULL));

But I don't know if this is an appropriate fix.

Ideas? Thanks greatly! And Thanks for a great program... for Raspberry Pi's. :-)

[Paul Wise (Debian) (pabs)]

I can't reproduce this on the x86 hardware that I have.

It sounds like you are running FoxtrotGPS on a Raspberry Pi?

Which operating system are you running?

Do you know what the thread limits are on your OS? Run this:

   $ cat /proc/sys/kernel/threads-max
   95400

As a workaround you can also increase the amount of threads:

   sudo sh -c 'echo 1000000 > /proc/sys/kernel/threads-max'

Are there any particular circumstances where it happens? Like just
after downloading a lot of tiles or something like that?

Please try to figure out which of the threads is being created but not
destroyed. I assume it isn't the gps thread, that is probably just the
final one that causes the limit to be reached. On my system I see this:

   $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status
   /proc/457081/task/457081/status:Name: foxtrotgps
   /proc/457081/task/457082/status:Name: gmain
   /proc/457081/task/457083/status:Name: dconf worker
   /proc/457081/task/457084/status:Name: gdbus
   /proc/457081/task/466976/status:Name: download thread
   /proc/457081/task/466977/status:Name: download thread
   /proc/457081/task/466978/status:Name: download thread
   /proc/457081/task/466982/status:Name: download thread
   /proc/457081/task/466984/status:Name: download thread
   /proc/457081/task/466985/status:Name: download thread

If you are able to bisect this issue to find out which commit causes
it, that would help in resolving the problem. I'm assuming it was
caused by the commit changing to the new GLib thread APIs.

--
bye,
pabs

https://wiki.debian.org/PaulWise

[Russell Grokett (rgrokett)]

 Thanks for the super fast response back!
I suspect the high thread-max you have is helping.

$ cat  /proc/sys/kernel/threads-max15137
OS is Raspbian Buster on RPi 4.

pi@raspix86TEST:~/foxtrotgps-1.2.2 $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status/proc/31606/task/31606/status:Name:     foxtrotgps/proc/31606/task/31607/status:Name:     gmain/proc/31606/task/31608/status:Name:     gdbus/proc/31606/task/31609/status:Name:     dconf worker
Using the g_thread_unref() below seems to have eliminated the issue as I have not been able to reproduce since adding.   I will try backing out the change and see if it reoccurs again and capture the threads.
g_thread_unref(g_thread_new("gps thread", &get_gps_thread,
  (gpointer)NULL));

 g_thread_unref(g_thread_new("download thread", &dl_thread, (gpointer *)tile_data));

Thanks!Russell

    On Monday, May 4, 2020, 08:50:36 PM EDT, Paul Wise (Debian) <email address hidden> wrote:

 I can't reproduce this on the x86 hardware that I have.

It sounds like you are running FoxtrotGPS on a Raspberry Pi?

Which operating system are you running?

Do you know what the thread limits are on your OS? Run this:

  $ cat  /proc/sys/kernel/threads-max
  95400

As a workaround you can also increase the amount of threads:

  sudo sh -c 'echo 1000000 > /proc/sys/kernel/threads-max'

Are there any particular circumstances where it happens? Like just
after downloading a lot of tiles or something like that?

Please try to figure out which of the threads is being created but not
destroyed. I assume it isn't the gps thread, that is probably just the
final one that causes the limit to be reached. On my system I see this:

  $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status
  /proc/457081/task/457081/status:Name:    foxtrotgps
  /proc/457081/task/457082/status:Name:    gmain
  /proc/457081/task/457083/status:Name:    dconf worker
  /proc/457081/task/457084/status:Name:    gdbus
  /proc/457081/task/466976/status:Name:    download thread
  /proc/457081/task/466977/status:Name:    download thread
  /proc/457081/task/466978/status:Name:    download thread
  /proc/457081/task/466982/status:Name:    download thread
  /proc/457081/task/466984/status:Name:    download thread
  /proc/457081/task/466985/status:Name:    download thread

If you are able to bisect this issue to find out which commit causes
it, that would help in resolving the problem. I'm assuming it was
caused by the commit changing to the new GLib thread APIs.

--
bye,
pabs

https://wiki.debian.org/PaulWise

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1876744

Title:
  g_thread_new() fails with "creating thread 'gps thread': Error
  creating thread: Resource temporarily unavailable".

Status in FoxtrotGPS:
  New

Bug description:
  I ran into a bug causing foxtrotgps to crash due to out-of-threads
  error on Debian Buster when I happened to disconnect the GPS and let
  it run overnight (it can take hours before failure!). I see numerous
  GLib-CRITICAL messages concerning the gps_functions.c connecting to
  gpsd (not unexpected!) but then eventually, a GLib-ERROR
  g_thread_new...

[Paul Wise (Debian) (pabs)]

Can you try increasing thread-max?

Your thread list doesn't look particularly long, hmm.

Looking at the GLib threads documentation it seems that you do need to
g_thread_unref if you aren't using g_thread_join indeed.

https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-new
https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-unref
https://codesearch.debian.net/search?q=g_thread_unref.*g_thread_new&literal=0

We have a number of calls to g_thread_new, so all of them will have to
get g_thread_unref added. If you would like to contribute a patch that
does that, that would be great, otherwise I can try to schedule some
time to work on it.

--
bye,
pabs

https://wiki.debian.org/PaulWise

[Jesse Gardner (jgardner7289)]

I am experiencing this bug on two x86_64 machines. On both, when downloading tiles at max depth, a crash occurs after about 15 minutes. The output before the crash is this several times (for different tiles):

  TILE DL PROBLEM: getaddrinfo() thread failed to start
  https://tile.openstreetmap.org/12/1279/1588.png|/home/jesse/Maps/OSM/12/1279/1588.png|/home/jesse/Maps/OSM/12/1279/

Followed by this:

  (foxtrotgps:68150): GLib-ERROR **: 03:37:17.434: creating thread 'download thread': Error creating thread: Resource temporarily unavailable
  Trace/breakpoint trap (core dumped)

The two machines are running Ubuntu 20.04, using foxtrotgps 1.2.2 from the official repos, and Manjaro, using foxtrotgps compiled from bzr. The thread-max for theses machines are 513321 and 62553, respectively.

On the manjaro machine, while downloading tiles, I ran this command given above every 2s until crash: grep Name /proc/$(pgrep foxtrotgps)/task/*/status

At any given time, there were 1 each of threads named foxtrotgps, gmain, dconf worker, gdbus, and a handful of download threads. About 10 seconds before crash, another (second) thread named foxtrotgps appeared. Not sure what that means, but hopefully it is helpful.

Edit: I was able to determine that this bug was introduced by commit 315. Compiling from 315 produced the same behavior, but compiling from 314 produced a build that was able to download tiles at max depth and stay running for over an hour, where previously it crashed consistently around the 15 minute mark.

[Paul Wise (Debian) (pabs)]

Jesse Gardner created, tested and submitted a patch that fixes this problem and I have now merged the patch into trunk.