Spice console is freezing

[Bjoern]

Currently we do encounter intermittent spice console lock ups on individual nodes.
The spice console is still accepting new TCP connections but not serving any HTML5/CSS. No errors in the logs,
it just stops reporting the internal pool statistics (alive/zombie threads).

From a debugging perspective :

- File descriptions are not exhausted
  (It had only < 8 TCP connections in ESTABLISHED state but was queuing on RECV tcp packets)

- Strace showed only one system call (no forking seem to have happened)

select(0, null, null, null, {60, 0}) = 0 (timeout)

- Few TCP connections where in state SYN_RECV

A simple service restart fixed it it needs to be investigated.

Websockify is already 0.6.0

Spice-html5proxy is used from the vanilla ubuntu 14.04 package

[Bjoern (bjoern-t)]

open files during this state:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nova-spic 531 nova cwd DIR 252,2 4096 62522368 /usr/share/spice-html5
nova-spic 531 nova rtd DIR 252,2 4096 47980974 /
nova-spic 531 nova txt REG 252,2 3349512 48634653 /usr/bin/python2.7
nova-spic 531 nova mem REG 252,2 58378 62002287 /usr/local/lib/python2.7/dist-packages/netifaces.so
nova-spic 531 nova mem REG 252,2 66632 59776747 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
nova-spic 531 nova mem REG 252,2 46768 61872199 /usr/lib/python2.7/lib-dynload/bz2.x86_64-linux-gnu.so
nova-spic 531 nova mem REG 252,2 31792 59776813 /lib/x86_64-linux-gnu/librt-2.19.so
nova-spic 531 nova mem REG 252,2 27962 62001089 /usr/local/lib/python2.7/dist-packages/markupsafe/_speedups.so
nova-spic 531 nova mem REG 252,2 11152 61872218 /usr/lib/python2.7/lib-dynload/resource.x86_64-linux-gnu.so
nova-spic 531 nova mem REG 252,2 33448 61872203 /usr/lib/python2.7/lib-dynload/_multiprocessing.x86_64-linux-gnu.so
nova-spic 531 nova mem REG 252,2 2083636 62134332 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
nova-spic 531 nova mem REG 252,2 135727 62134602 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
nova-spic 531 nova mem REG 252,2 7632 61872229 /usr/lib/python2.7/lib-dynload/future_builtins.x86_64-linux-gnu.so
nova-spic 531 nova mem REG 252,2 4797429 62134735 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
nova-spic 531 nova mem REG 252,2 4182588 62134739 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
nova-spic 531 nova mem REG 252,2 102780 62134711 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
nova-spic 531 nova mem REG 252,2 625241 62134439 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
nova-spic 531 nova mem REG 252,2 2237183 62134577 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
nova-spic 531 nova mem REG 252,2 7452854 62134560 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
nova-spic 531 nova mem REG 252,2 170064 59776740 /lib/x86_64-linux-gnu/libexpat.so.1.6.0
nova-spic 531 nova mem REG 252,2 69200 61872197 /usr/lib/python2.7/lib-dynload/pyexpat.x86_64-linux-gnu.so
nova-spic 531 nova mem REG 252,2 54064 61872220 /usr/lib/python2.7/lib-dynload/_json.x86_64-linux-gnu.so
nova-spic 531 nova mem REG 252,2 164466 62002907 /usr/local/lib/python2.7/dist-packages/simplejson/_speedups.so
nova-spic 531 nova mem REG 252,2 18936 59776804 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
nova-spic 531 nova mem REG 252,2 136232 61872210 /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so
nova-spic 531 nova mem REG 252,2 2373019 62001327 /usr/local/lib/python2.7/dist-packages/cryptography/_Cryptography_cffi_36a40ff0x2bad1bae.so
nova-spic 531 nova mem REG 252,2 30944 61999434 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
nova-...

[Bjoern (bjoern-t)]

open FDs:

lrwx------ 1 nova nova 64 Apr 9 13:08 0 -> /dev/null
lrwx------ 1 nova nova 64 Apr 9 13:08 1 -> /dev/pts/1
lrwx------ 1 nova nova 64 Apr 9 13:08 2 -> /dev/pts/1
l-wx------ 1 nova nova 64 Apr 9 13:08 3 -> /var/log/nova/nova-spicehtml5proxy.log
lrwx------ 1 nova nova 64 Apr 9 13:08 4 -> socket:[596562780]
lrwx------ 1 nova nova 64 Apr 9 13:08 5 -> anon_inode:[eventpoll]
lrwx------ 1 nova nova 64 Apr 9 13:08 6 -> socket:[598785845]
lr-x------ 1 nova nova 64 Apr 9 13:08 8 -> /dev/urandom

[Andy McCrae (andrew-mccrae)]

Are there any steps to recreate this issue consistently, or is it transient?

I'm not too sure what I can add to resolve this since I can't recreate the problem.

[Bjoern (bjoern-t)]

There is no way I can reproduce it. It just comes and goes intermittently

[Kevin Carter (kevin-carter)]

Marked invalid , cant reproduce the issue.