Activity log for bug #1715254

Date Who What changed Old value New value Message
2017-09-05 22:58:58 Graham Burgess bug added bug
2017-09-05 23:03:36 Graham Burgess bug added subscriber The Canonical Sysadmins
2017-10-03 16:54:16 Jill Rouleau affects nova (Ubuntu) charm-nova-cloud-controller
2017-10-05 13:40:03 James Page bug task added nova (Ubuntu)
2017-10-05 13:40:12 James Page bug task added websockify (Ubuntu)
2017-10-05 13:40:17 James Page charm-nova-cloud-controller: status New Invalid
2017-10-06 00:13:20 Launchpad Janitor nova (Ubuntu): status New Confirmed
2017-10-06 00:13:20 Launchpad Janitor websockify (Ubuntu): status New Confirmed
2017-10-20 11:51:03 James Page nova (Ubuntu): importance Undecided Medium
2017-10-20 11:51:05 James Page websockify (Ubuntu): importance Undecided Medium
2017-10-24 02:22:26 Seyeong Kim websockify (Ubuntu): assignee Seyeong Kim (xtrusia)
2017-10-24 02:22:46 Seyeong Kim bug added subscriber Seyeong Kim
2017-10-24 03:48:17 Seyeong Kim bug task added cloud-archive
2017-10-24 03:48:45 Seyeong Kim bug task deleted nova (Ubuntu)
2017-10-24 04:07:29 Seyeong Kim description Users reported they were unable to connect to instance consoles via either Horizon or direct URL. Upon investigation we found errors suggesting the address and port were in use: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] WebSocket server settings: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Flash security policy server 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file) 2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last): 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/bin/nova-novncproxy", line 10, in <module> 2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main()) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main 2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy 2017-08-23 14:51:56.249 1355081 ERROR nova RequestHandlerClass=websocketproxy.NovaProxyRequestHandler 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in start_server 2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=self.tcp_keepintvl) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket 2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4]) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/socket.py", line 224, in meth 2017-08-23 14:51:56.249 1355081 ERROR nova return getattr(self._sock,name)(*args) 2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova This lead us to the discovery of a stuck nova-novncproxy process after stopping the service. Once we sent a kill -HUP to that process, we were able to start the nova-novncproxy and restore service to the users. This was not the first time we have had to restart nova-novncproxy services after users reported that were unable to connect with VNC. This time, as well as at least 2 other times, we have seen the following errors in the nova-novncproxy.log during the time frame of the issue: gaierror: [Errno -8] Servname not supported for ai_socktype which seems to correspond to a log entries for connection strings with an invalid port ('port': u'-1'). As well as a bunch of: error: [Errno 104] Connection reset by peer [Impact] affected - UCA Mitaka - Xenial need to test - UCA Icehouse - Trusty TBD [Test case] 1. Deploy openstack 2. Creating instances 3. open console in browser with auto refresh extension ( set 5 seconds ) 4. after several hours connection rejected TBD [Regression Potential] Components that using websockify, escpecially nova-novncproxy, will be affected by this patch. However, After upgrading this and refreshing test above mentioned for 2 days without restarting any services, no hang happens. I tested this test in my local simple environment, so need to be considered possibility in different circumstances. [Others] related commits - https://github.com/novnc/websockify/pull/226 - https://github.com/novnc/websockify/pull/219 [Original Description] Users reported they were unable to connect to instance consoles via either Horizon or direct URL. Upon investigation we found errors suggesting the address and port were in use: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] WebSocket server settings: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Flash security policy server 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file) 2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last): 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/bin/nova-novncproxy", line 10, in <module> 2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main()) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main 2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy 2017-08-23 14:51:56.249 1355081 ERROR nova RequestHandlerClass=websocketproxy.NovaProxyRequestHandler 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in start_server 2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=self.tcp_keepintvl) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket 2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4]) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/socket.py", line 224, in meth 2017-08-23 14:51:56.249 1355081 ERROR nova return getattr(self._sock,name)(*args) 2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova This lead us to the discovery of a stuck nova-novncproxy process after stopping the service. Once we sent a kill -HUP to that process, we were able to start the nova-novncproxy and restore service to the users. This was not the first time we have had to restart nova-novncproxy services after users reported that were unable to connect with VNC. This time, as well as at least 2 other times, we have seen the following errors in the nova-novncproxy.log during the time frame of the issue: gaierror: [Errno -8] Servname not supported for ai_socktype which seems to correspond to a log entries for connection strings with an invalid port ('port': u'-1'). As well as a bunch of: error: [Errno 104] Connection reset by peer
2017-10-24 04:07:53 Seyeong Kim attachment added lp1715254-mitaka.debdiff https://bugs.launchpad.net/cloud-archive/+bug/1715254/+attachment/4990058/+files/lp1715254-mitaka.debdiff
2017-10-24 04:08:06 Seyeong Kim attachment added lp1715254-xenial.debdiff https://bugs.launchpad.net/cloud-archive/+bug/1715254/+attachment/4990059/+files/lp1715254-xenial.debdiff
2017-10-24 04:20:45 Ubuntu Foundations Team Bug Bot tags canonical-bootstack canonical-bootstack patch
2017-10-24 04:20:50 Ubuntu Foundations Team Bug Bot bug added subscriber Ubuntu Sponsors Team
2017-10-26 19:25:38 Corey Bryant nominated for series cloud-archive/kilo
2017-10-26 19:25:38 Corey Bryant bug task added cloud-archive/kilo
2017-10-26 19:25:38 Corey Bryant nominated for series cloud-archive/icehouse
2017-10-26 19:25:38 Corey Bryant bug task added cloud-archive/icehouse
2017-10-26 19:25:38 Corey Bryant nominated for series cloud-archive/mitaka
2017-10-26 19:25:38 Corey Bryant bug task added cloud-archive/mitaka
2017-10-26 19:26:06 Corey Bryant cloud-archive/kilo: status New Triaged
2017-10-26 19:26:10 Corey Bryant cloud-archive/icehouse: status New Triaged
2017-10-26 19:26:11 Corey Bryant cloud-archive/icehouse: importance Undecided Medium
2017-10-26 19:26:13 Corey Bryant cloud-archive/kilo: importance Undecided Medium
2017-10-26 19:26:18 Corey Bryant cloud-archive/mitaka: importance Undecided Medium
2017-10-26 19:26:20 Corey Bryant cloud-archive: status New Invalid
2017-10-26 19:26:25 Corey Bryant cloud-archive/mitaka: status New Triaged
2017-10-26 19:26:50 Corey Bryant nominated for series Ubuntu Trusty
2017-10-26 19:26:50 Corey Bryant bug task added websockify (Ubuntu Trusty)
2017-10-26 19:26:50 Corey Bryant nominated for series Ubuntu Xenial
2017-10-26 19:26:50 Corey Bryant bug task added websockify (Ubuntu Xenial)
2017-10-26 19:26:57 Corey Bryant websockify (Ubuntu Trusty): status New Triaged
2017-10-26 19:26:59 Corey Bryant websockify (Ubuntu Xenial): status New Triaged
2017-10-26 19:27:01 Corey Bryant websockify (Ubuntu Trusty): importance Undecided Medium
2017-10-26 19:27:03 Corey Bryant websockify (Ubuntu Xenial): importance Undecided Medium
2017-10-26 19:27:19 Corey Bryant websockify (Ubuntu Trusty): assignee Seyeong Kim (xtrusia)
2017-10-26 19:27:28 Corey Bryant websockify (Ubuntu Xenial): assignee Seyeong Kim (xtrusia)
2017-10-26 19:27:34 Corey Bryant websockify (Ubuntu): status Confirmed Invalid
2017-10-26 19:27:36 Corey Bryant websockify (Ubuntu): assignee Seyeong Kim (xtrusia)
2017-10-26 19:29:35 Corey Bryant websockify (Ubuntu): importance Medium Undecided
2017-10-27 20:43:29 Corey Bryant bug added subscriber Ubuntu Stable Release Updates Team
2017-10-28 08:13:40 Seyeong Kim attachment added lp1715254-kilo.debdiff https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+attachment/4998576/+files/lp1715254-kilo.debdiff
2017-10-30 06:51:28 Seyeong Kim description [Impact] affected - UCA Mitaka - Xenial need to test - UCA Icehouse - Trusty TBD [Test case] 1. Deploy openstack 2. Creating instances 3. open console in browser with auto refresh extension ( set 5 seconds ) 4. after several hours connection rejected TBD [Regression Potential] Components that using websockify, escpecially nova-novncproxy, will be affected by this patch. However, After upgrading this and refreshing test above mentioned for 2 days without restarting any services, no hang happens. I tested this test in my local simple environment, so need to be considered possibility in different circumstances. [Others] related commits - https://github.com/novnc/websockify/pull/226 - https://github.com/novnc/websockify/pull/219 [Original Description] Users reported they were unable to connect to instance consoles via either Horizon or direct URL. Upon investigation we found errors suggesting the address and port were in use: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] WebSocket server settings: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Flash security policy server 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file) 2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last): 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/bin/nova-novncproxy", line 10, in <module> 2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main()) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main 2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy 2017-08-23 14:51:56.249 1355081 ERROR nova RequestHandlerClass=websocketproxy.NovaProxyRequestHandler 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in start_server 2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=self.tcp_keepintvl) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket 2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4]) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/socket.py", line 224, in meth 2017-08-23 14:51:56.249 1355081 ERROR nova return getattr(self._sock,name)(*args) 2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova This lead us to the discovery of a stuck nova-novncproxy process after stopping the service. Once we sent a kill -HUP to that process, we were able to start the nova-novncproxy and restore service to the users. This was not the first time we have had to restart nova-novncproxy services after users reported that were unable to connect with VNC. This time, as well as at least 2 other times, we have seen the following errors in the nova-novncproxy.log during the time frame of the issue: gaierror: [Errno -8] Servname not supported for ai_socktype which seems to correspond to a log entries for connection strings with an invalid port ('port': u'-1'). As well as a bunch of: error: [Errno 104] Connection reset by peer [Impact] affected - UCA Mitaka, Kilo - Xenial not affected - UCA Icehouse - Trusty ( log symptom is different, there is no reaing(which is errata) zombie... etc) [Test case] 1. Deploy openstack 2. Creating instances 3. open console in browser with auto refresh extension ( set 5 seconds ) 4. after several hours connection rejected [Regression Potential] Components that using websockify, escpecially nova-novncproxy, will be affected by this patch. However, After upgrading this and refreshing test above mentioned for 2 days without restarting any services, no hang happens. I tested this test in my local simple environment, so need to be considered possibility in different circumstances. [Others] related commits - https://github.com/novnc/websockify/pull/226 - https://github.com/novnc/websockify/pull/219 [Original Description] Users reported they were unable to connect to instance consoles via either Horizon or direct URL. Upon investigation we found errors suggesting the address and port were in use: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] WebSocket server settings: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Flash security policy server 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file) 2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last): 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/bin/nova-novncproxy", line 10, in <module> 2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main()) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main 2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy 2017-08-23 14:51:56.249 1355081 ERROR nova RequestHandlerClass=websocketproxy.NovaProxyRequestHandler 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in start_server 2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=self.tcp_keepintvl) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket 2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4]) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/socket.py", line 224, in meth 2017-08-23 14:51:56.249 1355081 ERROR nova return getattr(self._sock,name)(*args) 2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova This lead us to the discovery of a stuck nova-novncproxy process after stopping the service. Once we sent a kill -HUP to that process, we were able to start the nova-novncproxy and restore service to the users. This was not the first time we have had to restart nova-novncproxy services after users reported that were unable to connect with VNC. This time, as well as at least 2 other times, we have seen the following errors in the nova-novncproxy.log during the time frame of the issue: gaierror: [Errno -8] Servname not supported for ai_socktype which seems to correspond to a log entries for connection strings with an invalid port ('port': u'-1'). As well as a bunch of: error: [Errno 104] Connection reset by peer
2017-10-30 06:56:03 Seyeong Kim description [Impact] affected - UCA Mitaka, Kilo - Xenial not affected - UCA Icehouse - Trusty ( log symptom is different, there is no reaing(which is errata) zombie... etc) [Test case] 1. Deploy openstack 2. Creating instances 3. open console in browser with auto refresh extension ( set 5 seconds ) 4. after several hours connection rejected [Regression Potential] Components that using websockify, escpecially nova-novncproxy, will be affected by this patch. However, After upgrading this and refreshing test above mentioned for 2 days without restarting any services, no hang happens. I tested this test in my local simple environment, so need to be considered possibility in different circumstances. [Others] related commits - https://github.com/novnc/websockify/pull/226 - https://github.com/novnc/websockify/pull/219 [Original Description] Users reported they were unable to connect to instance consoles via either Horizon or direct URL. Upon investigation we found errors suggesting the address and port were in use: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] WebSocket server settings: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Flash security policy server 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file) 2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last): 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/bin/nova-novncproxy", line 10, in <module> 2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main()) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main 2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy 2017-08-23 14:51:56.249 1355081 ERROR nova RequestHandlerClass=websocketproxy.NovaProxyRequestHandler 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in start_server 2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=self.tcp_keepintvl) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket 2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4]) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/socket.py", line 224, in meth 2017-08-23 14:51:56.249 1355081 ERROR nova return getattr(self._sock,name)(*args) 2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova This lead us to the discovery of a stuck nova-novncproxy process after stopping the service. Once we sent a kill -HUP to that process, we were able to start the nova-novncproxy and restore service to the users. This was not the first time we have had to restart nova-novncproxy services after users reported that were unable to connect with VNC. This time, as well as at least 2 other times, we have seen the following errors in the nova-novncproxy.log during the time frame of the issue: gaierror: [Errno -8] Servname not supported for ai_socktype which seems to correspond to a log entries for connection strings with an invalid port ('port': u'-1'). As well as a bunch of: error: [Errno 104] Connection reset by peer [Impact] affected - UCA Mitaka, Kilo - Xenial not affected - UCA Icehouse - Trusty ( log symptom is different, there is no reaing(which is errata) zombie... etc) When number of connections are many or frequently reconnecting to console, nova-novncproxy daemon is stuck because websockify is hang. [Test case] 1. Deploy openstack 2. Creating instances 3. open console in browser with auto refresh extension ( set 5 seconds ) 4. after several hours connection rejected [Regression Potential] Components that using websockify, escpecially nova-novncproxy, will be affected by this patch. However, After upgrading this and refreshing test above mentioned for 2 days without restarting any services, no hang happens. I tested this test in my local simple environment, so need to be considered possibility in different circumstances. [Others] related commits - https://github.com/novnc/websockify/pull/226 - https://github.com/novnc/websockify/pull/219 [Original Description] Users reported they were unable to connect to instance consoles via either Horizon or direct URL. Upon investigation we found errors suggesting the address and port were in use: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] WebSocket server settings: 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Flash security policy server 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc 2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] - No SSL/TLS support (no cert file) 2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last): 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/bin/nova-novncproxy", line 10, in <module> 2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main()) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main 2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy 2017-08-23 14:51:56.249 1355081 ERROR nova RequestHandlerClass=websocketproxy.NovaProxyRequestHandler 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in start_server 2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=self.tcp_keepintvl) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket 2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4]) 2017-08-23 14:51:56.249 1355081 ERROR nova File "/usr/lib/python2.7/socket.py", line 224, in meth 2017-08-23 14:51:56.249 1355081 ERROR nova return getattr(self._sock,name)(*args) 2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in use 2017-08-23 14:51:56.249 1355081 ERROR nova This lead us to the discovery of a stuck nova-novncproxy process after stopping the service. Once we sent a kill -HUP to that process, we were able to start the nova-novncproxy and restore service to the users. This was not the first time we have had to restart nova-novncproxy services after users reported that were unable to connect with VNC. This time, as well as at least 2 other times, we have seen the following errors in the nova-novncproxy.log during the time frame of the issue: gaierror: [Errno -8] Servname not supported for ai_socktype which seems to correspond to a log entries for connection strings with an invalid port ('port': u'-1'). As well as a bunch of: error: [Errno 104] Connection reset by peer
2017-10-31 06:11:13 Dominique Poulain bug added subscriber Dominique Poulain
2017-11-02 23:41:33 Brian Murray websockify (Ubuntu Xenial): status Triaged Fix Committed
2017-11-02 23:41:36 Brian Murray bug added subscriber SRU Verification
2017-11-02 23:41:41 Brian Murray tags canonical-bootstack patch canonical-bootstack patch verification-needed verification-needed-xenial
2017-11-03 10:23:36 Seyeong Kim tags canonical-bootstack patch verification-needed verification-needed-xenial canonical-bootstack patch verification-done-xenial verification-needed
2017-11-15 00:43:08 Chris Halse Rogers removed subscriber Ubuntu Stable Release Updates Team
2017-11-15 00:43:17 Launchpad Janitor websockify (Ubuntu Xenial): status Fix Committed Fix Released
2017-11-20 15:49:16 Corey Bryant cloud-archive/mitaka: status Triaged Fix Committed
2017-11-20 15:49:18 Corey Bryant tags canonical-bootstack patch verification-done-xenial verification-needed canonical-bootstack patch verification-done-xenial verification-mitaka-needed verification-needed
2017-12-04 06:24:26 Seyeong Kim tags canonical-bootstack patch verification-done-xenial verification-mitaka-needed verification-needed canonical-bootstack patch verification-done-xenial verification-mitaka-done verification-needed
2017-12-04 15:11:03 James Page cloud-archive/mitaka: status Fix Committed Fix Released
2017-12-05 05:02:24 Seyeong Kim cloud-archive/kilo: assignee Seyeong Kim (xtrusia)
2017-12-05 05:02:26 Seyeong Kim cloud-archive/mitaka: assignee Seyeong Kim (xtrusia)
2017-12-05 05:02:28 Seyeong Kim websockify (Ubuntu Trusty): assignee Seyeong Kim (xtrusia)
2017-12-11 14:09:46 Seyeong Kim bug task deleted websockify (Ubuntu Trusty)
2017-12-11 14:24:03 Edward Hope-Morley bug task deleted cloud-archive/icehouse
2017-12-18 21:03:36 Corey Bryant cloud-archive/kilo: status Triaged Fix Committed
2017-12-18 21:03:38 Corey Bryant tags canonical-bootstack patch verification-done-xenial verification-mitaka-done verification-needed canonical-bootstack patch verification-done-xenial verification-kilo-needed verification-mitaka-done verification-needed
2017-12-21 02:40:02 Seyeong Kim tags canonical-bootstack patch verification-done-xenial verification-kilo-needed verification-mitaka-done verification-needed canonical-bootstack patch verification-done-xenial verification-kilo-done verification-mitaka-done verification-needed
2018-01-02 19:01:02 Corey Bryant cloud-archive/kilo: status Fix Committed Fix Released
2018-01-04 11:54:48 Edward Hope-Morley tags canonical-bootstack patch verification-done-xenial verification-kilo-done verification-mitaka-done verification-needed canonical-bootstack patch sts sts-sru-done verification-done verification-done-xenial verification-kilo-done verification-mitaka-done