[2.4] INTERNAL_SERVER_ERROR on various operations without crash logs, which seemingly continue to work (e.g. release, allocate)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
High
|
Blake Rouse |
Bug Description
There's two situations where logs are being suppressed:
# As per Jasons bug.
In maas 2.4, we're seeing lots of 500 responses to allocate requests. In those requests, maas is setting the node to ALLOCATED but is returning a 500 code.
Here's two examples from the same machine and same run:
http://
We've seen this on:
maas_2.
and on:
2.4.0~beta2-
We're also seeing 500 responses on "release" operations and metadata posts in the same runs.
We believe these 500 errors are causing issues with juju not being able to recognize which machine's it's successfully allocated.
# As per CI.
=======
ERROR: maas-integratio
-------
testtools.
stderr for maas maas ['rack-
retcode for maas maas ['rack-
stdout for maas maas ['rack-
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://
<html xmlns="http://
<head>
<title>Error: Internal server error</title>
</head>
<body>
<h2>
Internal server error.
</h2>
</body>
</html>
}}}
Traceback (most recent call last):
File "/tmp/autopkgte
result = func(*args, **kwargs)
File "/tmp/autopkgte
rack_id = self.get_
File "/tmp/autopkgte
rack_
File "/usr/lib/
return _default_
File "/usr/lib/
obj, end = self.raw_decode(s)
File "/usr/lib/
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.
Related branches
- Andres Rodriguez (community): Approve
- MAAS Lander: Approve
-
Diff: 1341 lines (+378/-223)10 files modifiedsrc/maasserver/context_processors.py (+2/-0)
src/maasserver/djangosettings/demo.py (+1/-1)
src/maasserver/djangosettings/settings.py (+24/-25)
src/maasserver/djangosettings/tests/test_settings.py (+0/-8)
src/maasserver/middleware.py (+70/-22)
src/maasserver/tests/test_auth.py (+2/-2)
src/maasserver/tests/test_macaroon_auth.py (+2/-4)
src/maasserver/tests/test_middleware.py (+151/-101)
src/maasserver/utils/tests/test_views.py (+18/-40)
src/maasserver/utils/views.py (+108/-20)
Changed in maas: | |
milestone: | none → 2.4.0beta2 |
importance: | Undecided → Critical |
status: | New → Triaged |
Changed in maas: | |
milestone: | 2.4.0beta2 → 2.4.0beta3 |
importance: | Critical → High |
summary: |
- [2.4] INTERNAL_SERVER_ERROR has no logs in /var/log/maas/*.log + [2.4] INTERNAL_SERVER_ERROR on various operations without crash logs and + seemingly continue to work |
summary: |
- [2.4] INTERNAL_SERVER_ERROR on various operations without crash logs and - seemingly continue to work + [2.4] INTERNAL_SERVER_ERROR on various operations without crash logs, + which seemingly continue to work (e.g. release, allocate) |
tags: | added: cdo-qa cdo-qa-blocker foundations-engine |
description: | updated |
Changed in maas: | |
assignee: | nobody → Blake Rouse (blake-rouse) |
tags: | added: cdo-release-blocker |
Changed in maas: | |
status: | Triaged → In Progress |
Changed in maas: | |
status: | In Progress → Fix Committed |
Changed in maas: | |
milestone: | 2.4.0rc1 → 2.4.0beta3 |
Changed in maas: | |
status: | Fix Committed → Fix Released |
Ok, I can confirm this is the case. For example, we see a lot of INTERNAL_ SERVER_ ERROR on the regiond.log but we dont see tracebacks:
10.244. 40.30/var/ log/maas/ regiond. log:2018- 04-19 04:57:21 regiond: [info] 10.244.40.32 POST /MAAS/api/ 2.0/machines/ ?op=allocate HTTP/1.1 --> 500 INTERNAL_ SERVER_ ERROR (referrer: -; agent: Go-http-client/1.1) 40.30/var/ log/maas/ regiond. log:2018- 04-19 04:57:32 regiond: [info] 10.244.40.32 POST /MAAS/api/ 2.0/machines/ ?op=allocate HTTP/1.1 --> 500 INTERNAL_ SERVER_ ERROR (referrer: -; agent: Go-http-client/1.1) 40.30/var/ log/maas/ regiond. log:2018- 04-19 04:58:52 regiond: [info] 10.244.40.32 POST /MAAS/api/ 2.0/machines/ ?op=release HTTP/1.1 --> 500 INTERNAL_ SERVER_ ERROR (referrer: -; agent: Go-http-client/1.1) 40.30/var/ log/maas/ regiond. log:2018- 04-19 04:59:06 regiond: [info] 10.244.40.32 POST /MAAS/api/ 2.0/machines/ ?op=allocate HTTP/1.1 --> 500 INTERNAL_ SERVER_ ERROR (referrer: -; agent: Go-http-client/1.1) 40.31/var/ log/maas/ regiond. log:2018- 04-19 04:16:29 regiond: [info] 10.244.40.32 POST /MAAS/metadata/ 2012-03- 01/ HTTP/1.1 --> 500 INTERNAL_ SERVER_ ERROR (referrer: -; agent: Python-urllib/3.5) 40.31/var/ log/maas/ regiond. log:2018- 04-19 04:26:26 regiond: [info] 10.244.40.32 POST /MAAS/metadata/ 2012-03- 01/ HTTP/1.1 --> 500 INTERNAL_ SERVER_ ERROR (referrer: -; agent: Python-urllib/3.5)
10.244.
10.244.
10.244.
10.244.
10.244.