MAAS

Bug #1718016
Comment #14

Comment 14 for bug 1718016

Revision history for this message

Jason Hobbs (jason-hobbs) wrote on 2017-10-09: Re: [Bug 1718016] Re: MAAS failed to respond to POST'd deploy request but still deployed node

#14

doh, meant haproxy when i said hacluster :) No we haven't tried that. It's
set to 30 seconds right now. We can enable more logging in haproxy and try
more to reproduce.

On Mon, Oct 9, 2017 at 3:36 PM, Blake Rouse <email address hidden>
wrote:

> I would bet that is the case, have you tried increasing the timeout on
> haproxy to see if that helps this issue?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1718016
>
> Title:
> MAAS failed to respond to POST'd deploy request but still deployed
> node
>
> Status in MAAS:
> Triaged
> Status in MAAS 2.2 series:
> New
>
> Bug description:
> maas version: 2.2.2-6099-g8751f91-0ubuntu1~16.04.1
> juju version: 2.2.4-0ubuntu1~16.04.1~juju1
>
> =Description=
> When using juju to deploy a bundle to MAAS, juju received an EOF/no
> response from the MAAS server when POST'ing to MAAS to deploy the node:
>
> logsink.log:24cbf3cb-bad6-4d28-8377-4ffe01d2a25c: machine-0 2017-09-18
> 17:13:58 WARNING juju.provisioner provisioner_task.go:747 failed to
> start instance (unexpected: Post
> http://10.245.208.33/MAAS/api/2.0/machines/ab6sax/?op=deploy: EOF),
> retrying in 10s (10 more attempts)
>
> MAAS did not log this POST request in regiond.log. However, the node
> started its deploying process and eventually reach deployed. Since
> juju got no response from MAAS when trying to deploy the node, juju
> considered the node in an error state and the deployment of the bundle
> failed.
>
> Here's the summarized event log for the node:
> http://paste.ubuntu.com/25567525/
>
> MAAS and juju logs will be attached. This is an HA setup with 3
> region controllers and 2 rack controllers. There are quite a "few
> could not serialize access due to concurrent update" errors in the
> logs.
>
> =Expected behavior=
> MAAS should always return a response to the client.
>
> =Reproduction steps=
> I don't have a good way to reproduce the reliably. I saw similar
> behavior in bug 1717301 - it seems like they could be the same root cause.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1718016/+subscriptions
>

doh, meant haproxy when i said hacluster :) No we haven't tried that. It's
set to 30 seconds right now.  We can enable more logging in haproxy and try
more to reproduce.

On Mon, Oct 9, 2017 at 3:36 PM, Blake Rouse <blake.rouse@canonical.com>
wrote:

> I would bet that is the case, have you tried increasing the timeout on
> haproxy to see if that helps this issue?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1718016
>
> Title:
>   MAAS failed to respond to POST'd deploy request but still deployed
>   node
>
> Status in MAAS:
>   Triaged
> Status in MAAS 2.2 series:
>   New
>
> Bug description:
>   maas version: 2.2.2-6099-g8751f91-0ubuntu1~16.04.1
>   juju version: 2.2.4-0ubuntu1~16.04.1~juju1
>
>   =Description=
>   When using juju to deploy a bundle to MAAS, juju received an EOF/no
> response from the MAAS server when POST'ing to MAAS to deploy the node:
>
>   logsink.log:24cbf3cb-bad6-4d28-8377-4ffe01d2a25c: machine-0 2017-09-18
>   17:13:58 WARNING juju.provisioner provisioner_task.go:747 failed to
>   start instance (unexpected: Post
>   http://10.245.208.33/MAAS/api/2.0/machines/ab6sax/?op=deploy: EOF),
>   retrying in 10s (10 more attempts)
>
>   MAAS did not log this POST request in regiond.log. However, the node
>   started its deploying process and eventually reach deployed.  Since
>   juju got no response from MAAS when trying to deploy the node, juju
>   considered the node in an error state and the deployment of the bundle
>   failed.
>
>   Here's the summarized event log for the node:
>   http://paste.ubuntu.com/25567525/
>
>   MAAS and juju logs will be attached.  This is an HA setup with 3
>   region controllers and 2 rack controllers.  There are quite a "few
>   could not serialize access due to concurrent update" errors in the
>   logs.
>
>   =Expected behavior=
>   MAAS should always return a response to the client.
>
>   =Reproduction steps=
>   I don't have a good way to reproduce the reliably. I saw similar
> behavior in bug 1717301 - it seems like they could be the same root cause.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1718016/+subscriptions
>