TensorFlow application crashes after glibc upgrade from 14.04 to 16.04 version
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
glibc (Ubuntu) |
New
|
Undecided
|
Taco Screen team |
Bug Description
== Comment: #0 - Brian Hart - 2016-10-25 11:38:18 ==
---Problem Description---
TensorFlow application crashes after glibc upgrade from 14.04 to 16.04 version
My team is building and running Google's TensorFlow deep learning framework application. We've observed that TensorFlow (v 0.9.0, 0.10.0, and latest master) work on Ubuntu 14.04, but crash on Ubuntu 16.04.
We done various isolation experiments and have found that the same application binary that runs on 14.04 can be made to fail by updating just the libc packages on the 14.04 system. So we're suspecting either some regression in glibc, or perhaps some incorrect application code that worked by accident under the older glibc and now fails.
The failure takes various forms, usually one of:
a) crash due to gcc's "stack smashing" detection
b) segfault due to dereferencing a pointer that was damaged while it was on the stack
c) python type exception due to mismatched list sizes
---uname output---
Linux p10a102 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:45:41 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
---Additional Hardware Info---
Application uses NVIDIA GPU. Problem is seen with any of: PCI-attached K80 and M40, NVLINK-attached P100
Machine Type = Minsky
---Steps to Reproduce---
Train an inception network () on the ILSVRC2012 dataset using TensorFlow.
Inception network model:
$ git clone https:/
$ cd inception
$ bazel build inception/
$ bazel-bin/
Userspace tool common name: TensorFlow
The userspace tool has the following bit modes: 64-bit
Userspace tool obtained from project website: TensorFlow v0.9.0, built for example from: https:/
== Comment: #4 - Brian Hart - 2016-10-25 16:05:03 ==
Tulio,
Thank you; to respond to your points...
[W]ere you able to restrict to a smaller code sample?
=> Not yet.
Are you able to run the same training without GPU? i.e. using just the POWER
processor.
=> This is on our list of tests to run, but haven't tried it yet.
Have you run this code on valgrind's memcheck tool?
=> Running this now. So far at startup, I get several complaints about read-after-free on the part of python 2.7--mostly in things like python's realloc and GC code paths. I'm not going to worry about those at the moment.
Could you generate a coredump? e.g. running 'ulimit -c unlimited' before
training the network.
=> We'll try.
I wouldn't expect glibc from Ubuntu 16.04 to run on Ubuntu 14.04.
Is this issue appearing on an Ubuntu 16.04 install?
=> Yes, the crash happens with a straight 16.04 install. But we have been able to use newer libc packages (a constellation of about 6 packages, covering several libraries--libc, libm, pthreads, etc.). We find:
14.04 w/ glibc 2.19 - no crash
14.04 w/ glibc 2.21 (0ubuntu4) - no crash
14.04 w/ glibc 2.21 (0ubuntu4.3) - no crash
14.04 w/ glibc 2.23 - crashes
14.04 w/ glibc 2.24 - crashes
Were there any problematic changes between 2.21 and 2.23?
== Comment: #5 - Brian Hart - 2016-10-25 16:47:13 ==
Just got a core file from a crash. Moving it (~11GB) to a system where I can set up access.
The crash scenario was kind of interesting...
We crashed dereferencing the stack pointer:
(gdb) x/4i $pc-8
0x3fff7f4420a8 <_ZN10tensorflo
0x3fff7f4420ac <_ZN10tensorflo
=> 0x3fff7f4420b0 <_ZN10tensorflo
0x3fff7f4420b4 <_ZN10tensorflo
Which currently holds a bad value. But it looks like the stack pointer value
would be valid except that the high 16-bits has been changed from 0x0000 to
0x0001:
(gdb) info registers r1
r1 0x13bffa77fd4d0
If we drop the 0x0001 we're left with a pointer to a sane-looking stack
frame, with a saved LR that would put us in at least a nearby routine:
(gdb) x/8g 0x3bffa77fd4d0
0x3bffa77fd4d0: 0x00003bffa77fd500 0x00003bffa77fd5a8
0x3bffa77fd4e0: 0x00003fff7f442148 0x00003fff8931b428
0x3bffa77fd4f0: 0x00003bffa77fd928 0x00003bffa77fd500
0x3bffa77fd500: 0x00003bffa77fd650 0x00003fff00002200
(gdb) x/i 0x00003fff7f442
0x3fff7f442144 <_ZN10tensorflo
In another case we anaylzed we segfaulted because we dereferenced via r31,
which had been damaged in the same way (high 16-bits changed from 0x0000 to
0x0001). In that case, r31 was an alias for the stack pointer (because we built
with "-fstack-
_while sitting on the stack_ during a call to glibc's free(). (Caller had
dereferenced r31 just before the call to free(), so it was fine then, and we
could see that the damaged value was still present in the now-defunct free()
stackframe when free() caller ultimately dereferenced r31.
The current case is probably more interesting because the cases where r1 would
be lying around in memory to be damaged should be narrower than for other
registers. Basically when the thread is switched out.
A stray write by any other thread in the process might be the cause of the
problem here. But then why do we only see it at recent glibc versions? Were
there any changes to, say, the pthreads context switching code lately (does
pthreads even _do_ any context switching, or does it leave that all to the
kernel)?
== Comment: #6 - Brian Hart - 2016-10-25 20:13:33 ==
A couple of further comments...
Re: Valgrind - After the startup batch of complaints about python itself, valgrind is silent while the app is running. (And starting up python under valgrind without the app generates a similar set of complaints.)
We made the core file available to Tulio; sent access info out of band.
== Comment: #7 - Tulio Magno Quites Machado Filho - 2016-10-26 09:17:28 ==
(In reply to comment #5)
> A stray write by any other thread in the process might be the cause of the
> problem here. But then why do we only see it at recent glibc versions?
glibc 2.23 enabled -fstack-
So, there is a chance the problem was already there, but glibc 2.23 started to catch and report it.
For the record, there has been an issue reported to the tensorflow community: https:/
It was closed due to the lack of information.
> Were there any changes to, say, the pthreads context switching code lately (does
> pthreads even _do_ any context switching, or does it leave that all to the
> kernel)?
That's all kernel code.
== Comment: #8 - Brian Hart - 2016-10-26 12:29:00 ==
I'm not sure the stack protection setting change in glibc 2.23 can be the explanation here.
My understanding of the stack protection is that the compiler emits some additional code to stash a sentinal value on the stack, and to verify the sentinal value later (either just at the end of the routine, or after every child function call in the case of stack-protector
But we're seeing crashes that are sensitive to glibc levels, in a common binary that was built with the older toolchain. So the application binary doesn't have any additional stack protection checks in the 2.23 case compared to the 2.21 case. The newer glibc itself might have been built with the newer toolchain, but the stack protection hits we do see are occurring in the app object rather than the glibc objects.
And several of the crashes we see aren't stack protector catches.
Thank you for the pointer to the TensorFlow bug report; we're looking at that to see if it contains anything that might help with problem isolation.
tags: | added: architecture-ppc64le bugnameltc-147893 severity-high targetmilestone-inin--- |
Changed in ubuntu: | |
assignee: | nobody → Taco Screen team (taco-screen-team) |
affects: | ubuntu → glibc (Ubuntu) |
------- Comment From <email address hidden> 2016-11-11 18:14 EDT-------
A few notes on the debug effort since the previous update:
- We opened an issue with the TensorFlow project in github: /github. com/tensorflow/ tensorflow/ issues/ 5482
https:/
- Crash occurs with any of TensorFlow 0.9, 0.10, and recent master
(e.g. 2cbb9b5 of Oct 26th)
- Crash occurs while training inception model on ILSVRC 2012 dataset. Model is /github. com/tensorflow/ models. git inception/
from: https:/
- Crash is independent of CUDA / GPU; occurs even if we compile without CUDA
and GPU support
- Crash occurs after varying run times. Appears to occur sooner with higher
thread counts. Very possibly a race of some sort.
Our team continues to try to isolate and simplify the problem. We'll try
buildling on a 14.04 system with AT 9. That setup uses GLIBC 2.22, and
so may allow us to narrow the problem to either 2.21 -> 2.22 or 2.22 -> 2.23.
Any debug advice or suggestions are appreciated.