Users reported they were unable to connect to instance consoles via either Horizon or direct URL. Upon investigation we found errors suggesting the address and port were in use:
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] WebSocket server settings:
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Flash security policy server
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file)
2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address already in use
2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last):
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/bin/nova-novncproxy", line 10, in <module>
2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main())
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main
2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port)
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy
2017-08-23 14:51:56.249 1355081 ERROR nova RequestHandlerClass=websocketproxy.NovaProxyRequestHandler
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in start_server
2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=self.tcp_keepintvl)
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket
2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4])
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/socket.py", line 224, in meth
2017-08-23 14:51:56.249 1355081 ERROR nova return getattr(self._sock,name)(*args)
2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in use
2017-08-23 14:51:56.249 1355081 ERROR nova
This lead us to the discovery of a stuck nova-novncproxy process after stopping the service. Once we sent a kill -HUP to that process, we were able to start the nova-novncproxy and restore service to the users.
This was not the first time we have had to restart nova-novncproxy services after users reported that were unable to connect with VNC. This time, as well as at least 2 other times, we have seen the following errors in the nova-novncproxy.log during the time frame of the issue:
gaierror: [Errno -8] Servname not supported for ai_socktype
which seems to correspond to a log entries for connection strings with an invalid port ('port': u'-1'). As well as a bunch of:
Users reported they were unable to connect to instance consoles via either Horizon or direct URL. Upon investigation we found errors suggesting the address and port were in use:
2017-08-23 14:51:56.248 1355081 INFO nova.console. websocketproxy [-] WebSocket server settings: websocketproxy [-] - Listen on 0.0.0.0:6080 websocketproxy [-] - Flash security policy server websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc websocketproxy [-] - No SSL/TLS support (no cert file) nova-novncproxy ", line 10, in <module> python2. 7/dist- packages/ nova/cmd/ novncproxy. py", line 41, in main vnc.novncproxy_ port) python2. 7/dist- packages/ nova/cmd/ baseproxy. py", line 73, in proxy lass=websocketp roxy.NovaProxyR equestHandler python2. 7/dist- packages/ websockify/ websocket. py", line 909, in start_server self.tcp_ keepintvl) python2. 7/dist- packages/ websockify/ websocket. py", line 698, in socket addrs[0] [4]) python2. 7/socket. py", line 224, in meth self._sock, name)(* args)
2017-08-23 14:51:56.248 1355081 INFO nova.console.
2017-08-23 14:51:56.248 1355081 INFO nova.console.
2017-08-23 14:51:56.248 1355081 INFO nova.console.
2017-08-23 14:51:56.248 1355081 INFO nova.console.
2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address already in use
2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last):
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/bin/
2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main())
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/
2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/
2017-08-23 14:51:56.249 1355081 ERROR nova RequestHandlerC
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/
2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/
2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(
2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/
2017-08-23 14:51:56.249 1355081 ERROR nova return getattr(
2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in use
2017-08-23 14:51:56.249 1355081 ERROR nova
This lead us to the discovery of a stuck nova-novncproxy process after stopping the service. Once we sent a kill -HUP to that process, we were able to start the nova-novncproxy and restore service to the users.
This was not the first time we have had to restart nova-novncproxy services after users reported that were unable to connect with VNC. This time, as well as at least 2 other times, we have seen the following errors in the nova-novncproxy.log during the time frame of the issue:
gaierror: [Errno -8] Servname not supported for ai_socktype
which seems to correspond to a log entries for connection strings with an invalid port ('port': u'-1'). As well as a bunch of:
error: [Errno 104] Connection reset by peer