g_thread_new() fails with "creating thread 'gps thread': Error creating thread: Resource temporarily unavailable".

Bug #1876744 reported by Russell Grokett on 2020-05-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
FoxtrotGPS
Undecided
Unassigned

Bug Description

I ran into a bug causing foxtrotgps to crash due to out-of-threads error on Debian Buster when I happened to disconnect the GPS and let it run overnight (it can take hours before failure!). I see numerous GLib-CRITICAL messages concerning the gps_functions.c connecting to gpsd (not unexpected!) but then eventually, a GLib-ERROR g_thread_new() fails with "creating thread 'gps thread': Error creating thread: Resource temporarily unavailable".

This appears only in v1.2.2 using the new GLIB g_thread_new().

When I reverted to v1.2.1 with g_thread_create() I do not see this issue.

This appears to affect several (all?) threads being used as I was able to recreate with other threaded operations, like map tile download thread.

My best guess is that g_thread_exit() or g_thread_unref() isn't being called when the thread is no longer needed and every call to g_thread_new() is creating yet another thread.

Strangely, the GLIB docs say that the thread should free when the function returns, but that doesn't appear to me to be the case. (Maybe they mean it is free to be reused, but is not released and unrefed?)

I haven't worked with GLIB before, but I assume that wherever g_free() is being called, it may need g_thread_exit() or g_thread_unref()?

Reading GLIB info from:
https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-new

----- EXAMPLE ERROR MESSAGE -------

(foxtrotgps:18515): GLib-CRITICAL **: 11:22:28.267: Source ID 29030 was not found when attempting to remove it
connection to gpsd SUCCEEDED

(foxtrotgps:18515): GLib-CRITICAL **: 11:23:28.355: Source ID 29157 was not found when attempting to remove it

(foxtrotgps:18515): GLib-ERROR **: 11:23:28.355: creating thread 'gps thread': Error creating thread: Resource temporarily unavailable
Trace/breakpoint trap

$

Another...

(foxtrotgps:10300): GLib-ERROR **: 10:22:46.899: creating thread 'download thread': Error creating thread: Resource temporarily unavailable
Trace/breakpoint trap

$

gps_functions.c:
#if GLIB_CHECK_VERSION(2,34,0)
        g_thread_new("gps thread", &get_gps_thread, (gpointer)NULL);
#else
        g_thread_create(&get_gps_thread, NULL, FALSE, NULL);
#endif
}

Currently, I am testing the following change:

        g_thread_unref(g_thread_new("gps thread", &get_gps_thread, (gpointer)NULL));

But I don't know if this is an appropriate fix.

Ideas? Thanks greatly! And Thanks for a great program... for Raspberry Pi's. :-)

I can't reproduce this on the x86 hardware that I have.

It sounds like you are running FoxtrotGPS on a Raspberry Pi?

Which operating system are you running?

Do you know what the thread limits are on your OS? Run this:

   $ cat /proc/sys/kernel/threads-max
   95400

As a workaround you can also increase the amount of threads:

   sudo sh -c 'echo 1000000 > /proc/sys/kernel/threads-max'

Are there any particular circumstances where it happens? Like just
after downloading a lot of tiles or something like that?

Please try to figure out which of the threads is being created but not
destroyed. I assume it isn't the gps thread, that is probably just the
final one that causes the limit to be reached. On my system I see this:

   $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status
   /proc/457081/task/457081/status:Name: foxtrotgps
   /proc/457081/task/457082/status:Name: gmain
   /proc/457081/task/457083/status:Name: dconf worker
   /proc/457081/task/457084/status:Name: gdbus
   /proc/457081/task/466976/status:Name: download thread
   /proc/457081/task/466977/status:Name: download thread
   /proc/457081/task/466978/status:Name: download thread
   /proc/457081/task/466982/status:Name: download thread
   /proc/457081/task/466984/status:Name: download thread
   /proc/457081/task/466985/status:Name: download thread

If you are able to bisect this issue to find out which commit causes
it, that would help in resolving the problem. I'm assuming it was
caused by the commit changing to the new GLib thread APIs.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Russell Grokett (rgrokett) wrote :
Download full text (5.4 KiB)

 Thanks for the super fast response back!
I suspect the high thread-max you have is helping.

$ cat  /proc/sys/kernel/threads-max15137
OS is Raspbian Buster on RPi 4.

pi@raspix86TEST:~/foxtrotgps-1.2.2 $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status/proc/31606/task/31606/status:Name:     foxtrotgps/proc/31606/task/31607/status:Name:     gmain/proc/31606/task/31608/status:Name:     gdbus/proc/31606/task/31609/status:Name:     dconf worker
Using the g_thread_unref() below seems to have eliminated the issue as I have not been able to reproduce since adding.   I will try backing out the change and see if it reoccurs again and capture the threads.
g_thread_unref(g_thread_new("gps thread", &get_gps_thread,
  (gpointer)NULL));

 g_thread_unref(g_thread_new("download thread", &dl_thread, (gpointer *)tile_data));

Thanks!Russell

    On Monday, May 4, 2020, 08:50:36 PM EDT, Paul Wise (Debian) <email address hidden> wrote:

 I can't reproduce this on the x86 hardware that I have.

It sounds like you are running FoxtrotGPS on a Raspberry Pi?

Which operating system are you running?

Do you know what the thread limits are on your OS? Run this:

  $ cat  /proc/sys/kernel/threads-max
  95400

As a workaround you can also increase the amount of threads:

  sudo sh -c 'echo 1000000 > /proc/sys/kernel/threads-max'

Are there any particular circumstances where it happens? Like just
after downloading a lot of tiles or something like that?

Please try to figure out which of the threads is being created but not
destroyed. I assume it isn't the gps thread, that is probably just the
final one that causes the limit to be reached. On my system I see this:

  $ grep Name /proc/$(pgrep foxtrotgps)/task/*/status
  /proc/457081/task/457081/status:Name:    foxtrotgps
  /proc/457081/task/457082/status:Name:    gmain
  /proc/457081/task/457083/status:Name:    dconf worker
  /proc/457081/task/457084/status:Name:    gdbus
  /proc/457081/task/466976/status:Name:    download thread
  /proc/457081/task/466977/status:Name:    download thread
  /proc/457081/task/466978/status:Name:    download thread
  /proc/457081/task/466982/status:Name:    download thread
  /proc/457081/task/466984/status:Name:    download thread
  /proc/457081/task/466985/status:Name:    download thread

If you are able to bisect this issue to find out which commit causes
it, that would help in resolving the problem. I'm assuming it was
caused by the commit changing to the new GLib thread APIs.

--
bye,
pabs

https://wiki.debian.org/PaulWise

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1876744

Title:
  g_thread_new() fails with "creating thread 'gps thread': Error
  creating thread: Resource temporarily unavailable".

Status in FoxtrotGPS:
  New

Bug description:
  I ran into a bug causing foxtrotgps to crash due to out-of-threads
  error on Debian Buster when I happened to disconnect the GPS and let
  it run overnight (it can take hours before failure!). I see numerous
  GLib-CRITICAL messages concerning the gps_functions.c connecting to
  gpsd (not unexpected!) but then eventually, a GLib-ERROR
  g_thread_new...

Read more...

Paul Wise (Debian) (pabs) wrote :

Can you try increasing thread-max?

Your thread list doesn't look particularly long, hmm.

Looking at the GLib threads documentation it seems that you do need to
g_thread_unref if you aren't using g_thread_join indeed.

https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-new
https://developer.gnome.org/glib/stable/glib-Threads.html#g-thread-unref
https://codesearch.debian.net/search?q=g_thread_unref.*g_thread_new&literal=0

We have a number of calls to g_thread_new, so all of them will have to
get g_thread_unref added. If you would like to contribute a patch that
does that, that would be great, otherwise I can try to schedule some
time to work on it.

--
bye,
pabs

https://wiki.debian.org/PaulWise

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers