n-api fails to start on juno grenade

[gordon chung]

possibly related to this: https://github.com/openstack/nova/commit/a657582c5cd8a39580c44ad401fd3e69870068b1

grenade tests are not passing due to n-api not starting

logs here: http://logs.openstack.org/56/161256/1/check/check-grenade-dsvm/6fb9bfa/logs/

from n-api log:

2015-03-05 14:04:34.874 10501 INFO nova.openstack.common.service [-] Started child 10508
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
TypeError: server() got an unexpected keyword argument 'socket_timeout'
2015-03-05 14:04:34.876 10501 INFO nova.openstack.common.service [-] Started child 10509
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
TypeError: server() got an unexpected keyword argument 'socket_timeout'
2015-03-05 14:04:34.878 10501 INFO nova.openstack.common.service [-] Started child 10510
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
TypeError: server() got an unexpected keyword argument 'socket_timeout'

[Matt Riedemann (mriedem)]

Started on 3/5:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVHlwZUVycm9yOiBzZXJ2ZXIoKSBnb3QgYW4gdW5leHBlY3RlZCBrZXl3b3JkIGFyZ3VtZW50ICdzb2NrZXRfdGltZW91dCdcIiBBTkQgdGFnczpcInNjcmVlbi1uLWFwaS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyNTU4NDU4NTE2Nn0=

Due to this merge:

https://github.com/openstack/nova/commit/a657582c5cd8a39580c44ad401fd3e69870068b1

When that merged, it was tested against eventlet 0.15.2:

http://logs.openstack.org/11/138811/2/gate/gate-tempest-dsvm-full/ac4e165/logs/pip-freeze.txt.gz

When it fails in stable/juno, the 'old' icehouse side of grenade is running with eventlet 0.13.0 for some reason:

http://logs.openstack.org/56/161256/1/check/check-grenade-dsvm/6fb9bfa/logs/pip-freeze.txt.gz

Even though that goes against what's in stable/juno's global-requiements:

https://github.com/openstack/requirements/blob/stable/juno/global-requirements.txt#L23

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/161863

[Matt Riedemann (mriedem)]

We can't increase the minimum required version of eventlet in global-requirements on stable/icehouse b/c that will break distros that have shipped their supported versions of eventlet for icehouse, so we just can't have this change on icehouse since we could be using a version of eventlet that doesn't support the socket_timeout kwarg.

[Joe Gordon (jogo)]

Icehouse job on precise:

http://logs.openstack.org/11/138811/2/gate/gate-tempest-dsvm-full/ac4e165/logs/devstacklog.txt.gz#_2015-03-05_00_29_06_531

python-eventlet all 0.9.16-1ubuntu4.2 [283 kB]

Icehouse -> Juno Grenade job in Trusty:

http://logs.openstack.org/56/161256/1/check/check-grenade-dsvm/6fb9bfa/logs/old/devstacklog.txt.gz#_2015-03-05_14_00_27_638

/python-eventlet_0.13.0-1ubuntu2_all.deb

icehouse global-requirments: 'eventlet': 'eventlet>=0.13.0,<0.16.0'

On Trusty, we just use the deb based eventlet, but on precise its too old so we need a newer version so pip installs

2015-03-05 00:30:22.732 | Collecting eventlet<0.16.0,>=0.13.0 (from keystone==2014.1.4.dev15)
2015-03-05 00:30:22.813 | Downloading http://pypi.region-b.geo-1.openstack.org/packages/2.7/e/eventlet/eventlet-0.15.2-py2.py3-none-any.whl (150kB)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161863
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=07ec12c40cdeea8bf4d8617d408cbf6189f76e2b
Submitter: Jenkins
Branch: stable/icehouse

commit 07ec12c40cdeea8bf4d8617d408cbf6189f76e2b
Author: Matt Riedemann <email address hidden>
Date: Thu Mar 5 19:53:22 2015 +0000

    Revert "Eventlet green threads not released back to pool"

    This reverts commit a657582c5cd8a39580c44ad401fd3e69870068b1

    The minimum required version of eventlet does not work for the
    socket_timeout usage here:

    eventlet>=0.13.0,<0.16.0

    Closes-Bug: #1428821

    Change-Id: Iece034c6a055c70bc2f617a1eee1a8541ff65db1