django.db.utils.OperationalError: could not serialize access due to concurrent update during "migrate-conflicting-options"

[Jason Hobbs]

During an HA install, we hit this error:

https://pastebin.canonical.com/p/wt7qq5RDkn/

This is with 2.3.2-6485-ge93e044-0ubuntu1~16.04.1.

[Andres Rodriguez (andreserl)]

Hi Jason,

Can you provide an:

1. apt.log and history.log
2. Were you installing the 'maas-region-controller' package in two different machines at the same time?
3. Was the database up-and-running when this happened?

Thanks.

[Andres Rodriguez (andreserl)]

After a closer look, it seems you are running these commands manually which wans't clear from the description on your bug report.

In that case, can you clarify something:

1. How do you install MAAS ? (e.g. do you install maas-region-controller in one server, and all others you install maas-region-api, or do you install maas-region-api on all?)
2. Do all install at the same time (e.g. are commands being run in parallel in each machine?)
3. Were the MAAS migrations had already being run or were they in progress?
3. More importantly, was postgresql fully installed, with migrations complete, and up-and-running?

[Andres Rodriguez (andreserl)]

[10.244.40.32] run: sudo maas-region edit_named_options --migrate-conflicting-options
[10.244.40.31] run: sudo maas-region edit_named_options --migrate-conflicting-options
[10.244.40.31] out: Traceback (most recent call last):

So the above command seems to be happening twice. This is trying to update the database twice at the same time, obviously causing the error.

[Jason Hobbs (jason-hobbs)]

On Fri, Apr 20, 2018 at 11:39 AM, Andres Rodriguez
<email address hidden> wrote:
> [10.244.40.32] run: sudo maas-region edit_named_options --migrate-conflicting-options
> [10.244.40.31] run: sudo maas-region edit_named_options --migrate-conflicting-options
> [10.244.40.31] out: Traceback (most recent call last):
>
> So the above command seems to be happening twice. This is trying to
> update the database twice at the same time, obviously causing the error.

No, it's running on two separate hosts, at about the same time. This
is an HA setup with multiple region controllers.

[Jason Hobbs (jason-hobbs)]

On Fri, Apr 20, 2018 at 11:28 AM, Andres Rodriguez
<email address hidden> wrote:
> After a closer look, it seems you are running these commands manually
> which wans't clear from the description on your bug report.
>
> In that case, can you clarify something:
>
> 1. How do you install MAAS ? (e.g. do you install maas-region-controller in one server, and all others you install maas-region-api, or do you install maas-region-api on all?)

maas-region-controller on one, then after it's done, maas-region-api
on the others

> 2. Do all install at the same time (e.g. are commands being run in parallel in each machine?)

maas-region-controller on one, then after it's done, maas-region-api
on the others in parallel.

> 3. Were the MAAS migrations had already being run or were they in progress?

They were already done.

> 3. More importantly, was postgresql fully installed, with migrations complete, and up-and-running?

Yes.

You can see the whole log of the install here:

https://10.245.162.101/artifacts/d7dbba5b-f47f-4e14-a045-87b1a47c3b83/make_foundation_908/make_foundation_908_console.out

[Andres Rodriguez (andreserl)]

@Jason,

So the underlying issue is that in your both regions, you are trying to update the same field at the same time via the CLI (which communicates directly with the DB).

As such, I suggest you drop usage of --migrate-conflicting-options and just use 'maas-region edit_named_options'. (the docs are being updated to reflect this change).

Also, note that in 2.4 --migrate-conflicting-options has been deprecated and has no effect. Also, 'maas-region edit_named_options' is no longer needed to be run due to the latest packaging changes and maas-region-api will run this command for you on initial install. (the docs are also being updated to reflect this).

[Andres Rodriguez (andreserl)]

Documentation has now been updated to correctly reflect the options for 2.3 or the drop of such option for 2.4.

[Andres Rodriguez (andreserl)]

Docs are updated, but since we dont track docs bugs here, the bug status reflects:

1. For 2.4, maas-dns has been dropped and you no longer need to run "maas-region edit_named_options".
2. For 2.3, you dont need to run with "--migrate-conflicting-options", which effectively unblocks this bug, however, from the MAAS side we are keeping the bug open to improve the handling in the near future.

[Andres Rodriguez (andreserl)]

This issue has been fixed as part of bug #1513775. Marking as Fix committed.