Activity log for bug #1761530

Date Who What changed Old value New value Message
2018-04-05 15:34:14 Andres Rodriguez bug added bug
2018-04-05 15:34:35 Andres Rodriguez maas: milestone 2.4.0beta2
2018-04-05 15:34:37 Andres Rodriguez maas: importance Undecided Critical
2018-04-05 15:34:38 Andres Rodriguez maas: status New Triaged
2018-04-11 23:55:21 Andres Rodriguez maas: milestone 2.4.0beta2 2.4.0beta3
2018-04-11 23:55:23 Andres Rodriguez maas: importance Critical High
2018-04-19 14:33:18 Andres Rodriguez maas: milestone 2.4.0beta3 2.4.0rc1
2018-04-19 14:33:36 Andres Rodriguez summary [2.4] Internal Server error has no logs in /var/log/maas/*.log [2.4] INTERNAL_SERVER_ERROR has no logs in /var/log/maas/*.log
2018-04-19 15:40:21 Andres Rodriguez summary [2.4] INTERNAL_SERVER_ERROR has no logs in /var/log/maas/*.log [2.4] INTERNAL_SERVER_ERROR on various operations without crash logs and seemingly continue to work
2018-04-19 15:40:58 Andres Rodriguez summary [2.4] INTERNAL_SERVER_ERROR on various operations without crash logs and seemingly continue to work [2.4] INTERNAL_SERVER_ERROR on various operations without crash logs, which seemingly continue to work (e.g. release, allocate)
2018-04-19 15:41:49 Jason Hobbs tags cdo-qa cdo-qa-blocker foundations-engine
2018-04-19 15:42:09 Andres Rodriguez description ====================================================================== ERROR: maas-integration.TestMAASIntegration.test_region_rack_connected ---------------------------------------------------------------------- testtools.testresult.real._StringException: Empty attachments: stderr for maas maas ['rack-controllers', 'read'] retcode for maas maas ['rack-controllers', 'read']: {{{2}}} stdout for maas maas ['rack-controllers', 'read']: {{{ <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>Error: Internal server error</title> </head> <body> <h2> Internal server error. </h2> </body> </html> }}} Traceback (most recent call last): File "/tmp/autopkgtest.W1TiDa/build.0iu/real-tree/debian/tests/timeout.py", line 22, in wrapper result = func(*args, **kwargs) File "/tmp/autopkgtest.W1TiDa/build.0iu/real-tree/debian/tests/common.py", line 682, in test_region_rack_connected rack_id = self.get_master_rack()['system_id'] File "/tmp/autopkgtest.W1TiDa/build.0iu/real-tree/debian/tests/common.py", line 663, in get_master_rack rack_controllers = loads(output) File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads return _default_decoder.decode(s) File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode obj, end = self.raw_decode(s) File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) There's two situations where logs are being suppressed: # As per Jasons bug. In maas 2.4, we're seeing lots of 500 responses to allocate requests. In those requests, maas is setting the node to ALLOCATED but is returning a 500 code. Here's two examples from the same machine and same run: http://paste.ubuntu.com/p/PDnYvnhmrP/ We've seen this on: maas_2.4.0~beta3-6863-ge11fd5e-0ubuntu1~18.04.1~20180418~ubuntu18.04.1 and on: 2.4.0~beta2-6865-gec43e47e6-0ubuntu1 We're also seeing 500 responses on "release" operations and metadata posts in the same runs. We believe these 500 errors are causing issues with juju not being able to recognize which machine's it's successfully allocated. # As per CI. ====================================================================== ERROR: maas-integration.TestMAASIntegration.test_region_rack_connected ---------------------------------------------------------------------- testtools.testresult.real._StringException: Empty attachments:   stderr for maas maas ['rack-controllers', 'read'] retcode for maas maas ['rack-controllers', 'read']: {{{2}}} stdout for maas maas ['rack-controllers', 'read']: {{{ <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">   <head>     <title>Error: Internal server error</title>   </head>   <body>     <h2>       Internal server error.     </h2>   </body> </html> }}} Traceback (most recent call last):   File "/tmp/autopkgtest.W1TiDa/build.0iu/real-tree/debian/tests/timeout.py", line 22, in wrapper     result = func(*args, **kwargs)   File "/tmp/autopkgtest.W1TiDa/build.0iu/real-tree/debian/tests/common.py", line 682, in test_region_rack_connected     rack_id = self.get_master_rack()['system_id']   File "/tmp/autopkgtest.W1TiDa/build.0iu/real-tree/debian/tests/common.py", line 663, in get_master_rack     rack_controllers = loads(output)   File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads     return _default_decoder.decode(s)   File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode     obj, end = self.raw_decode(s)   File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode     return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2018-04-19 16:09:33 Andres Rodriguez maas: assignee Blake Rouse (blake-rouse)
2018-04-19 17:49:25 Chris Gregan tags cdo-qa cdo-qa-blocker foundations-engine cdo-qa cdo-qa-blocker cdo-release-blocker foundations-engine
2018-04-20 17:34:43 Blake Rouse merge proposal linked https://code.launchpad.net/~blake-rouse/maas/+git/maas/+merge/343710
2018-04-20 17:34:50 Blake Rouse maas: status Triaged In Progress
2018-04-20 21:13:26 MAAS Lander maas: status In Progress Fix Committed
2018-05-03 07:45:22 Andres Rodriguez maas: milestone 2.4.0rc1 2.4.0beta3
2018-05-03 07:58:02 Andres Rodriguez maas: status Fix Committed Fix Released