Unable to connect to: ws://<maas IP>:/MAAS/ws
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| MAAS |
High
|
Anthony Ochoa | ||
| 1.8 |
High
|
Blake Rouse | ||
| 1.9 |
High
|
Blake Rouse | ||
| Trunk |
High
|
Blake Rouse | ||
| apache2 (Ubuntu) |
High
|
Dave Chiluk | ||
| Trusty |
High
|
Dave Chiluk |
Bug Description
[Impact]
* The Maas webui shows "Unable to connect to: ws://<maas IP>:/MAAS/ws" instead of listing the available machines.
* This is because apache proxy is attempting to reuse a socket for an ssl connection to the maas web server sitting on 5240. This is causing the maas server to throw an error 500.
* Similar results should be received from other web services that make use of the mod_proxy_wstunnel services and ssl.
* The upload fixes the bug by backporting an upstream commit that resolves this. It functions by preventing re-use of the socket connection.
[Test Case]
* Create a maas server, access webui repeatedly. Eventually the connections will get reused, and the error will be seen.
[Regression Potential]
* There is a potential performance regression for non-ssl proxy work, but that should be minimal.
* Patch has existed in vivid, wily, and xenial for a long while.
[Other Info]
* Upstream commit: https:/
-------
I have a set of servers that were transferred over to another maas servers so that the power credentials were no longer valid when back on our maas server. While to trying to get them working, I would see that check power state hangs because of the credentials being no longer valid. These would eventually time out and error out as expected.
However, after clicking on Nodes view and or trying to view nodes from a Zone, I would observe this error: "Unable to connect to: ws://<maas IP>:/MAAS/ws", then the view would eventually start load slowly while showing some nodes but not all of them. In one case it was stuck loading for > 10 minutes and it did not seem like it would load without me trying to reload from the browser.
I was initially doing this on firefox, then when I tried to start a session with chromium after problem was recreated, I got the same error followed by the nodes view being stuck while loading.
After a while, reloading the page from the browser would seem0 to work and everything would look ok.
I didn't see anything in the logs anything pointing to any issue with the cluster.
Related branches
- Andres Rodriguez (community): Approve on 2015-08-19
-
Diff: 135 lines (+23/-18)2 files modifiedsrc/maasserver/static/js/angular/factories/region.js (+4/-4)
src/maasserver/static/js/angular/factories/tests/test_region.js (+19/-14)
- Gavin Panella (community): Approve on 2015-08-20
-
Diff: 135 lines (+23/-18)2 files modifiedsrc/maasserver/static/js/angular/factories/region.js (+4/-4)
src/maasserver/static/js/angular/factories/tests/test_region.js (+19/-14)
- Andres Rodriguez (community): Approve on 2015-08-25
-
Diff: 18 lines (+9/-2)1 file modifieddebian/changelog (+9/-2)
- Andres Rodriguez (community): Approve on 2015-09-24
- Gavin Panella (community): Approve on 2015-09-24
-
Diff: 41 lines (+19/-1)2 files modifiedsrc/maasserver/static/js/angular/factories/region.js (+6/-1)
src/maasserver/static/js/angular/factories/tests/test_region.js (+13/-0)
- Blake Rouse (community): Approve on 2015-09-25
-
Diff: 41 lines (+19/-1)2 files modifiedsrc/maasserver/static/js/angular/factories/region.js (+6/-1)
src/maasserver/static/js/angular/factories/tests/test_region.js (+13/-0)
- MAAS Maintainers: Pending requested 2015-09-25
-
Diff: 4031 lines (+3119/-47) (has conflicts)40 files modifiedMakefile (+7/-0)
docs/changelog.rst (+72/-0)
docs/conf.py (+1/-1)
docs/index.rst (+1/-0)
docs/os-support.rst (+124/-0)
src/maasserver/api/tests/test_maas.py (+78/-0)
src/maasserver/clusterrpc/boot_images.py (+59/-0)
src/maasserver/clusterrpc/tests/test_boot_images.py (+62/-0)
src/maasserver/dns/tests/test_config.py (+4/-3)
src/maasserver/fields.py (+30/-0)
src/maasserver/migrations/0139_power_parameters_and_state_updated_field.py (+453/-0)
src/maasserver/models/tests/test_node.py (+5/-0)
src/maasserver/node_action.py (+13/-0)
src/maasserver/plugin.py (+14/-0)
src/maasserver/rpc/nodes.py (+31/-0)
src/maasserver/rpc/tests/test_nodes.py (+12/-1)
src/maasserver/rpc/tests/test_regionservice.py (+15/-0)
src/maasserver/static/css/maas-styles.css (+4/-0)
src/maasserver/static/img/icons/tag-close.svg (+18/-0)
src/maasserver/static/partials/node-details.html (+67/-0)
src/maasserver/static/partials/node-events.html (+4/-0)
src/maasserver/static/partials/nodes-list.html (+10/-0)
src/maasserver/static/scss/maas/components/_placeholders.scss (+4/-0)
src/maasserver/testing/factory.py (+17/-0)
src/maasserver/tests/test_fields.py (+38/-0)
src/maasserver/tests/test_node_action.py (+88/-32)
src/maasserver/tests/test_plugin.py (+22/-0)
src/maasserver/utils/osystems.py (+27/-0)
src/maasserver/utils/tests/test_osystems.py (+47/-0)
src/maasserver/views/tests/test_rpc.py (+13/-2)
src/maasserver/websockets/handlers/node.py (+32/-0)
src/provisioningserver/drivers/power/__init__.py (+4/-0)
src/provisioningserver/plugin.py (+7/-0)
src/provisioningserver/power/__init__.py (+170/-0)
src/provisioningserver/power/tests/test_power.py (+1001/-0)
src/provisioningserver/tests/test_plugin.py (+20/-8)
src/provisioningserver/utils/fs.py (+4/-0)
src/provisioningserver/utils/tests/test_fs.py (+14/-0)
src/provisioningserver/utils/tests/test_isc.py (+279/-0)
utilities/maas-region-support-dump (+248/-0)
- Mike Pontillo (community): Approve on 2015-12-22
-
Diff: 12 lines (+1/-1)1 file modifiedcontrib/maas-http.conf (+1/-1)
- Mike Pontillo (community): Approve on 2015-12-22
-
Diff: 12 lines (+1/-1)1 file modifiedcontrib/maas-http.conf (+1/-1)
description: | updated |
description: | updated |
Larry Michel (lmic) wrote : | #1 |
Larry Michel (lmic) wrote : | #2 |
The scenario to recreate this may just be a coincidence and the recreates may be somewhat random. I do see the issue recreated just by clicking on Zone, then the link provided for servers without having checked power state prior to that.
But, one thing I did observe is that if I changed power settings while checking power state was going on, then my changes made during the check power state were lost after I navigated away from the window. In those cases, save changes looked like it completed successfully. If it didn't work, I should have gotten an error and I didn't see any error other than the check power state one.
Larry Michel (lmic) wrote : | #3 |
I am seeing some strange behavior with the browser not updating. I am thinking it's related to this so adding to this bug. So, I started commissioning system from UI:
Aug 14 05:28:26 maas-trusty-
Aug 14 05:28:26 maas-trusty-
Aug 14 05:28:26 maas-trusty-
Aug 14 05:28:28 maas-trusty-
The logs shows it is commissioning but it's still in READY state, and power is on in the UI! If I try to commission it again from the UI, it fails as expected but without the log there was not telling that commissioning had worked (attached screen capture)..
I saw something similar with deleting a node where node was still there as if the delete hadn't worked, so when I tried to delete again, it would fail with message: "The delete action for 1 node failed with error: node-84864d62-
From another session, I couldn't find the node, so the delete had worked. But the log showed 3 separate delete attempts but no error:
Aug 13 22:03:49 maas-trusty-
Aug 13 22:03:58 maas-trusty-
Aug 13 22:04:44 maas-trusty-
Gavin Panella (allenap) wrote : | #4 |
The following from regiond.conf (repeated a lot) seems pertinent:
Closing connection: <STATUSES=
Opening connection with IPv4Address(TCP, '10.172.68.84', 54762)
Gavin Panella (allenap) wrote : | #5 |
In reverse order :)
Gavin Panella (allenap) wrote : | #6 |
Verified with Andres: please connect to MAAS via port 5240 instead of the default (80). There seems to be a CSRF issue affecting all connections via Apache.
Andres Rodriguez (andreserl) wrote : | #7 |
Marking this bug as Incomplete for the time being as I'm unable to reproduce anymore. Blake says this might just require a log out and logging back in through the webui!
Changed in maas: | |
status: | New → Incomplete |
Andreas Hasenack (ahasenack) wrote : | #8 |
We are seeing it as well with 1.8.1:
192.168.24.45 - - [18/Aug/
tags: | added: cisco landscape |
Blake Rouse (blake-rouse) wrote : | #9 |
Andreas,
That is showing a 500 error. A traceback should be in regiond.log. That seems like a different bug.
Changed in maas: | |
status: | Incomplete → Confirmed |
importance: | Undecided → High |
milestone: | none → 1.9.0 |
Changed in maas: | |
status: | Confirmed → Fix Committed |
Changed in maas: | |
assignee: | nobody → Blake Rouse (blake-rouse) |
Larry Michel (lmic) wrote : | #10 |
I have been testing the patch from attached branch since last week and seems to work so far. On the maas server without the patch, we're now hitting it on a regular basis.
Jason Hobbs (jason-hobbs) wrote : | #11 |
I hit this on MAAS 1.8.2.
Jason Hobbs (jason-hobbs) wrote : | #12 |
We're still hitting this on 1.8.2 - a fresh install of 1.8.2.
Jason Hobbs (jason-hobbs) wrote : | #13 |
The error is still "Unable to connect to: ws://10.
Blake Rouse (blake-rouse) wrote : | #14 |
The issue is the extra ":" at the end. I will look into why its being appended.
Changed in maas: | |
status: | Fix Committed → Confirmed |
summary: |
- Unable to connect to: ws://<maas IP>:/MAAS/ws after check power state - slow responding on bad power credentials + Unable to connect to: ws://<maas IP>:/MAAS/ws |
Changed in maas: | |
status: | Confirmed → Triaged |
Changed in maas: | |
status: | Triaged → In Progress |
Changed in maas: | |
status: | In Progress → Fix Committed |
Jason Hobbs (jason-hobbs) wrote : | #15 |
I just hit this in MAAS 1.8.3.
Larry Michel (lmic) wrote : | #16 |
Also hitting on 1.8.3 on both firefox and chromium after not having seen it since upgrade last tuesday from 1.8.2.
nate (nate-sellers) wrote : | #17 |
Hitting this on a fresh install of 1.8.3
ii maas 1.8.3+bzr4053-
nate (nate-sellers) wrote : | #18 |
2015-11-17 11:16:17 [-] 127.0.0.1 - - [17/Nov/
2015-11-17 11:16:17 [HTTPChannel,
Traceback (most recent call last):
File "/usr/lib/
return func(*args,**kw)
File "/usr/lib/
why = selectable.doRead()
File "/usr/lib/
return self._dataRecei
File "/usr/lib/
rval = self.protocol.
--- <exception caught here> ---
File "/usr/lib/
self.
File "/usr/lib/
for opcode, data, fin in _parseFrames(
File "/usr/lib/
raise _WSException(
maasserver.
Ray Wang (raywang) wrote : | #19 |
also hit on this bug, not sure it's relevant, but I saw errors in /var/log/
<snip>
[Fri Nov 27 09:15:26.672654 2015] [proxy_
[Fri Nov 27 09:15:33.530217 2015] [proxy_
</snip>
Mike Pontillo (mpontillo) wrote : | #20 |
Have you customized your Apache configuration at all? This appears to be an issue with that.
Can you try hitting http://<maas-ip-
Larry Michel (lmic) wrote : | #21 |
We haven't customized our Apache2 configuration at all Mike. We hit this issue on our prod environment and were able to recreate on a maas environment. http://<maas-ip-
Larry Michel (lmic) wrote : | #22 |
Correction/
Ray Wang (raywang) wrote : | #23 |
I also saw this bug on MAAS 1.9 rc4 a lot of times, if it's fixed by maas 1.9.0, it might be a regression bug?
Changed in maas: | |
status: | Fix Committed → Fix Released |
Billy Olsen (billy-olsen) wrote : | #24 |
Marked this as also affects apache2 to get the root cause fixed in apache. Upstream bug for this is https:/
From the looks of it, this only needs to be fixed in trusty as of this time, since vivid already includes the updated code and 2.2.x series on precise does not have support for mod_proxy_wstunnel:
wolsen@
apache2 | 2.2.22-1ubuntu1 | precise | source, amd64, armel, armhf, i386, powerpc
apache2 | 2.2.22-1ubuntu1.10 | precise-security | source, amd64, armel, armhf, i386, powerpc
apache2 | 2.2.22-1ubuntu1.10 | precise-updates | source, amd64, armel, armhf, i386, powerpc
apache2 | 2.4.7-1ubuntu4 | trusty | source, amd64, arm64, armhf, i386, powerpc, ppc64el
apache2 | 2.4.7-1ubuntu4.5 | trusty-security | source, amd64, arm64, armhf, i386, powerpc, ppc64el
apache2 | 2.4.7-1ubuntu4.8 | trusty-updates | source, amd64, arm64, armhf, i386, powerpc, ppc64el
apache2 | 2.4.10-9ubuntu1 | vivid | source, amd64, arm64, armhf, i386, powerpc, ppc64el
apache2 | 2.4.10-9ubuntu1.1 | vivid-security | source, amd64, arm64, armhf, i386, powerpc, ppc64el
apache2 | 2.4.10-9ubuntu1.1 | vivid-updates | source, amd64, arm64, armhf, i386, powerpc, ppc64el
apache2 | 2.4.12-2ubuntu2 | wily | source, amd64, arm64, armhf, i386, powerpc, ppc64el
apache2 | 2.4.17-3ubuntu1 | xenial | source, amd64, arm64, armhf, i386, powerpc, ppc64el, s390x
Dave Chiluk (chiluk) wrote : | #25 |
I'll work on getting the usptream fix SRU'd.
Dave Chiluk (chiluk) wrote : | #26 |
Xenial, and vivid already have the solution for this.
Here is the debdiff with the solution for trusty.
I also took the liberty of uploading a build with this debdiff here in the hopes that I might get some additional testing.
https:/
Changed in apache2 (Ubuntu): | |
status: | New → In Progress |
assignee: | nobody → Dave Chiluk (chiluk) |
milestone: | none → trusty-updates |
description: | updated |
Changed in apache2 (Ubuntu): | |
importance: | Undecided → High |
Dave Chiluk (chiluk) wrote : | #27 |
Tested that the apache fix appears to resolve the issue even after removing the workaround from wolsen.
Dave Chiluk (chiluk) wrote : | #28 |
Updated dep3 header information per arges' review.
Dave Chiluk (chiluk) wrote : | #29 |
@blake-rouse
After this fix is committed the maas team should strongly reconsider their stance to disable the apache proxy on port 80.
Hello Larry, or anyone else affected,
Accepted apache2 into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
Changed in apache2 (Ubuntu Trusty): | |
status: | New → Fix Committed |
tags: | added: verification-needed |
Dave Chiluk (chiluk) wrote : | #31 |
Verification complete. My test Apache is still up for maas after 2 days.
tags: |
added: verification-done removed: verification-needed |
Changed in apache2 (Ubuntu): | |
status: | In Progress → Fix Released |
Launchpad Janitor (janitor) wrote : | #32 |
This bug was fixed in the package apache2 - 2.4.7-1ubuntu4.9
---------------
apache2 (2.4.7-1ubuntu4.9) trusty; urgency=medium
* Force disablereuse on for mod_proxy_wstunnel. Fixes "Unable to connect to:
ws://<maas IP>:/MAAS/ws" errors with maas, and other proxy applications.
https:/
(LP: #1484696).
-- Dave Chiluk <email address hidden> Wed, 13 Jan 2016 15:34:51 -0600
Changed in apache2 (Ubuntu Trusty): | |
status: | Fix Committed → Fix Released |
Chris J Arges (arges) wrote : Update Released | #33 |
The verification of the Stable Release Update for apache2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
Changed in apache2 (Ubuntu): | |
status: | Fix Released → Invalid |
Changed in apache2 (Ubuntu Trusty): | |
milestone: | none → trusty-updates |
Changed in apache2 (Ubuntu): | |
milestone: | trusty-updates → none |
Changed in apache2 (Ubuntu Trusty): | |
importance: | Undecided → High |
assignee: | nobody → Dave Chiluk (chiluk) |
no longer affects: | maas/1.10 |
Changed in maas: | |
assignee: | Blake Rouse (blake-rouse) → Anthony Ochoa (anthony1985) |
Here are the logs.