Heat:Heat api exited with trace back.It did not restart after that .Had to manually restart

[Sandip Dey]

Build:R3.02 35 kilo

Logs saved at :http://10.204.216.50/Docs/bugs/<bug-id>

Had the below setup.Its a solution testbed running for a week after upgrade

On nodei35, heat engine exted with the below traceback

heat-api-supervisor-stdout.log
================================
2016-05-14 21:27:21.852 27251 INFO eventlet.wsgi.server [-] (27251) accepted ('192.168.1.6', 58103)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
  File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 376, in _single_run
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 828, in server
  File "/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 117, in waitall
AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.

upstart/supervisor-openstack.log
-----------------------------------------------------------
2016-05-14 21:27:23,096 INFO exited: heat-api (exit status 0; expected)^M
2016-05-15 09:07:55,817 INFO spawned: 'heat-api' with pid 28537^M

setup
------
host1 = 'root@10.204.217.139'
host2 = 'root@10.204.217.140'
host3 = 'root@10.204.217.147'
host4 = 'root@10.204.217.144'
host5 = 'root@10.204.217.147'
host6 = 'root@10.204.217.148'
host7 = 'root@10.204.217.149'
host8 = 'root@10.204.217.150'
host9 = 'root@10.204.217.210'
host10 = 'root@10.204.217.217'
host11 = 'root@10.204.217.218'
host12 = 'root@10.204.217.220'
host13 = 'root@10.204.217.247'
host14 = 'root@10.204.217.248'
host15 = 'root@10.204.217.249'
host16 = 'root@10.204.217.118'
host17 = 'root@10.204.217.119'
host18 = 'root@10.204.217.120'
host19 = 'root@10.204.217.121'
host20 = 'root@10.204.217.122'
host21 = 'root@10.204.217.123'
host22 = 'root@10.204.217.124'
host23 = 'root@10.204.217.131'

ext_routers = [('blr-mx2', '10.204.216.245')]
router_asn = 64512
public_vn_rtgt = 30001
#public_vn_subnet = "10.204.219.72/29"

host_build = 'vjoshi@10.204.216.56'

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6,host7, host8, host9, host10, host11, host12, host13, host14, host15,host16,host17,host18,host19,host20,host21,host22,host23],
    'cfgm': [host1, host2, host3],
    'openstack': [host4, host5, host6],
    'webui': [host1, host2, host3],
    'control': [host1, host2, host3],
    'compute': [host7, host8, host9, host10, host11, host12, host13, host14, host15,host16,host17,host18,host19,host20,host21,host22,host23], 'collector': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodei27', 'nodei28', 'nodei35', 'nodei32', 'nodei35', 'nodei36', 'nodei37', 'nodei38', 'nodel4', 'nodel7', 'nodel8', 'nodel9', 'nodel10', 'nodel11', 'nodel12','nodei6','nodei7','nodei8','nodei9','nodei10','nodei11','nodei12','nodei19']
}