I have a python with gdb support, and have a nice stack trace:


(gdb) pystack
/opt/stack/nova/nova/utils.py (745): to_primitive
/opt/stack/nova/nova/utils.py (736): to_primitive
/opt/stack/nova/nova/utils.py (751): to_primitive
/opt/stack/nova/nova/utils.py (730): to_primitive
/opt/stack/nova/nova/utils.py (745): to_primitive
/opt/stack/nova/nova/utils.py (736): to_primitive
/opt/stack/nova/nova/utils.py (751): to_primitive
/opt/stack/nova/nova/utils.py (736): to_primitive
/opt/stack/nova/nova/utils.py (751): to_primitive
/opt/stack/nova/nova/utils.py (730): to_primitive
/opt/stack/nova/nova/utils.py (736): to_primitive
/opt/stack/nova/nova/notifier/api.py (118): notify
/opt/stack/nova/nova/exception.py (136): wrapped
/opt/stack/nova/nova/compute/manager.py (141): decorated_function
/opt/stack/nova/nova/exception.py (112): wrapped
/opt/stack/nova/nova/compute/manager.py (394): _shutdown_instance_even_if_deleted
/opt/stack/nova/nova/compute/manager.py (443): _run_instance
/opt/stack/nova/nova/compute/manager.py (644): run_instance
/opt/stack/nova/nova/compute/manager.py (170): decorated_function
/opt/stack/nova/nova/exception.py (112): wrapped
/opt/stack/nova/nova/rpc/amqp.py (250): _process_data
/opt/stack/nova/nova/exception.py (112): wrapped
/usr/lib/python2.7/dist-packages/eventlet/greenpool.py (80): _spawn_n_impl
/usr/lib/python2.7/dist-packages/eventlet/hubs/epolls.py (59): do_poll
/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py (97): wait
/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py (226): run
/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py (177): switch
/usr/lib/python2.7/dist-packages/eventlet/event.py (116): wait
/usr/lib/python2.7/dist-packages/eventlet/greenthread.py (166): wait
/opt/stack/nova/nova/service.py (131): wait
/opt/stack/nova/nova/service.py (411): wait
/opt/stack/nova/bin/nova-compute (54): <module>


So this does look like bug 934575.  I merged trunk to get the patch, and no more crashes.  Thanks Russell for the help + for the fix.

Here are my notes on how to get a stack trace with Python & GDB:

# I'm presuming root, I think...
sudo bash

# You probably want gdb :-)
sudo apt-get install gdb

# Useful symbols
sudo apt-get install python2.7-dbg
sudo apt-get install libc6-dbg
sudo apt-get install python-greenlet-dbg

#Grrr... from http://wiki.python.org/moin/DebuggingWithGdb :
#Also, with gcc 4.5.2 on Ubuntu (at least) the macros fail because the call_function routine appears to be between PyEval_EvalFrameEx and PyEval_EvalCodeEx so the macro fails with No symbol "co" in current context..
#Recompiling python with make "CFLAGS=-g -fno-inline -fno-strict-aliasing" solves this problem.

# So let's build our own python...
sudo apt-get install dpkg-dev
sudo apt-get build-dep python2.7

mkdir -p ~/apts
pushd ~/apts
apt-get source python2.7
cd python2.7-2.7.2/
#Edit configure.in; add -f -fno-inline to the -fno-strict-aliasing
vim configure.in
Change:
      BASECFLAGS="$BASECFLAGS -fno-strict-aliasing"
To:
      BASECFLAGS="$BASECFLAGS -fno-strict-aliasing -fno-inline -g"

# Build the package, skip tests
export DEB_BUILD_OPTIONS="nocheck notest nobench parallel=8" 
# That should work, but I haven't actually tried with parallel
export DEB_BUILD_OPTIONS="nocheck notest nobench" 
dpkg-buildpackage

cd ..
dpkg -i python2.7_2.7.2-5ubuntu1_amd64.deb
dpkg -i python2.7-dbg_2.7.2-5ubuntu1_amd64.deb
dpkg -i libpython2.7_2.7.2-5ubuntu1_amd64.deb
dpkg -i python2.7-minimal_2.7.2-5ubuntu1_amd64.deb

# Should now show today's date...
python

popd


#Become root
sudo bash

cd ~
wget "http://svn.python.org/view/*checkout*/python/branches/release27-maint/Misc/gdbinit"
mv gdbinit .gdbinit

# Change mode so gdb will trust it
chmod 700 .gdbinit
sudo chown root .gdbinit

# Shouldn't show any warnings about gdb init
sudo gdb
quit


ps -ef | grep compute

stack@openstack1:~$ ps -ef | grep comp
root     17372  5350  0 11:01 pts/10   00:00:00 sg libvirtd /opt/stack/nova/bin/nova-compute
stack    17373 17372 10 11:01 pts/10   00:00:00 python /opt/stack/nova/bin/nova-compute
stack    17379  4561  0 11:01 pts/0    00:00:00 grep --color=auto comp

sudo gdb python 17373
pystack

# Should have a python stack