[2.2] rackd errors after fresh install

[John George]

Solutions QA CI failure caused by unavailable rack controller after fresh install of maas_2.2.1-6078-g2a6d96e-0ubuntu1~16.04.1

https://solutions.qa.canonical.com/#/qa/testRun/dd7949c8-0ce9-4f2b-a817-1a24c997aa12

rackd.log has errors.

Logs available at:
http://10.245.161.162/swift/v1/solutions-qa/dd7949c8-0ce9-4f2b-a817-1a24c997aa12/cloud_stack_417/maas-logs.tgz

[Lee Trager (ltrager)]

According to your logs the rack is trying to connect to the region at 127.0.0.1, is that correct? Could you please post the value of maas_url from /etc/maas/rackd.conf and /etc/maas/regiond.conf?

[John George (jog)]

This was a transient CI run and did not recreate on the next attempt. I'll capture the configs from etc when we hit it again.

[Andres Rodriguez (andreserl)]

@John, Chris,

Is this still an issue/transient failure?

It seems that with 2.2.2 the issue is not present.

[John George (jog)]

We have not hit it again yet.

MAAS deployed in the same environment has these maas_url values, requested in an earlier comment:
/etc/maas/rackd.conf:maas_url: http://10.245.208.33/MAAS
/etc/maas/regiond.conf:maas_url: http://10.245.208.33/MAAS

[Andres Rodriguez (andreserl)]

Thanks for the update. I'll mark this as invalid. If it comes up again, please re-open.

[Jason Hobbs (jason-hobbs)]

Here's logs, including /etc, of a recreate

[Greg Lutostanski (lutostag)]

hit against maas_2.2.3-6106-g314b2b2-0ubuntu1~16.04.1

[Greg Lutostanski (lutostag)]

Logs avaiable at https://10.245.162.101/artifacts/cc0e90f1-e706-4abb-8ec8-544f48bf043f/cloud_stack_750/maas-logs.tgz

[Blake Rouse (blake-rouse)]

Can you provide a full log of all the SQL queries? I know that will be a lot of information. But from the logs I can tell that the rack controller registered at one point and it was given that system_id, but then some how that system_id is deleted or no longer a rack controller.

So I think that some how the machine gets converted back to just a region controller instead of a region and rack controller.

Maybe the output of `nodes read` would help provide more information as well.

[Jason Hobbs (jason-hobbs)]

We hit this again last night, I've attached logs.

There is a rack controller error in regiond shortly before we hit this:

http://paste.ubuntu.com/25616610/

[Jason Hobbs (jason-hobbs)]

Blake, we hit this intermittently on a setup that runs lots of tests every day. We could turn on SQL query logging but it would add a very large load in terms of disk space and I'm not sure how well that would work out.

Is there additional logging that can be added to MAAS to help identify the source of the failure?

[Jason Hobbs (jason-hobbs)]

if the theory is a controller is turning from a rack controller into just a region controller, can you add logging everywhere that happens? That seems important enough to log and it doesn't seem like it would happen often enough to add much noise/size to the logs.

[Chris Gregan (cgregan)]

@Blake
Perhaps a more reasonable fix here is to add logging to MAAS to announce when controllers change spec? Our testing only exposes this failure once a week...maybe. Turning in global sql logging would likely have other resource impacts and greatly effect our other activities.

[Andres Rodriguez (andreserl)]

Hi guys!

What do you mean by "turning from a rack controller into just a region controller" and "announce when controllers change spec" ?

[Chris Gregan (cgregan)]

@andres

"The announce when controllers change" is referring to the comment Blake made about what seems to have happened "from the logs I can tell that the rack controller registered at one point and it was given that system_id, but then some how that system_id is deleted or no longer a rack controller"

If we had more detailed logging here it would allow is to properly triage this issue.

[Chris Gregan (cgregan)]

Attach cdoqa-system-test code for this process

[Chris Gregan (cgregan)]

@Andres

Per your request, this is the script running when the error occurs: https://bazaar.launchpad.net/~cdo-qa/cdoqa-system-tests/trunk/view/head:/cdoqa/set_maas_dhcp.py

[Greg Lutostanski (lutostag)]

This causes there to be no rack controller at all. If this occurs. There is no retry if this error path is hit. (re-occurrence misfiled as a dupe https://bugs.launchpad.net/maas/+bug/1721302).

Offer for maas team to poke active reproduction (after it occurred).