Failed to deploy Disco on Power8 with MAAS 2.5.2

[Po-Hsu Lin]

The deployment works on the very same Power8 node "modoc" 8 days ago (Jul.10)

But for now it complaining about this on the MAAS UI when trying to deploy Disco:

curtin: Installation started. (18.1-59-g0f993084-0ubuntu1~18.04.1)
third party drivers not installed or necessary.
Failed to exclusively open path: /dev/dm-0
Device holders with exclusive access: ['dm-1', 'dm-2']
Device mounts: []
Possible users of /dev/dm-0:
None
Failed to exclusively open path: /dev/dm-0
Device holders with exclusive access: ['dm-1', 'dm-2']
Device mounts: []
Possible users of /dev/dm-0:
None
Failed to exclusively open path: /dev/dm-0
Device holders with exclusive access: ['dm-1', 'dm-2']
Device mounts: []
Possible users of /dev/dm-0:
None
Failed to exclusively open path: /dev/dm-0
Device holders with exclusive access: ['dm-1', 'dm-2']
Device mounts: []
Possible users of /dev/dm-0:
None
[Errno 16] Device or resource busy: '/dev/dm-0'
curtin: Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'block-meta', 'custom']
Exit code: 3
Reason: -
Stdout: Failed to exclusively open path: /dev/dm-0
        Device holders with exclusive access: ['dm-1', 'dm-2']
        Device mounts: []
        Possible users of /dev/dm-0:
        None
        Failed to exclusively open path: /dev/dm-0
        Device holders with exclusive access: ['dm-1', 'dm-2']
        Device mounts: []
        Possible users of /dev/dm-0:
        None
        Failed to exclusively open path: /dev/dm-0
        Device holders with exclusive access: ['dm-1', 'dm-2']
        Device mounts: []
        Possible users of /dev/dm-0:
        None
        Failed to exclusively open path: /dev/dm-0
        Device holders with exclusive access: ['dm-1', 'dm-2']
        Device mounts: []
        Possible users of /dev/dm-0:
        None
        [Errno 16] Device or resource busy: '/dev/dm-0'

Stderr: ''

Tried to power cycle node "modoc" from bos01-b-07-fa1-15 / bos01-b-07-fb1-15, but no luck.

I can't deploy Disco on another Power8 node "entei" as well.

Not sure if this has something to do with the cloud image, since I don't think we have upgrade our MAAS server.

MAAS for node modoc - 2.5.2 (7523-ge4ecbd54d-0ubuntu1~18.04.1)
MAAS for node entei - 2.5.3 (7533-g65952b418-0ubuntu1~18.04.1)

[Po-Hsu Lin (cypressyew)]

On node "entei" it just failed with:
  "Installation was aborted."

[Frank Heimes (fheimes)]

Since modoc is a Tuleta system and Entei is a Firestone system, it does not seem to be dependent on a particular P8 model, hence also assuming it's MAAS and/or image related - hence involving the MAAS team.

[Frank Heimes (fheimes)]

@Po-Hsu Lin Can you confirm that deployments of Ubuntu releases other than disco still works?
Hence that this issue is limited to disco?

[Po-Hsu Lin (cypressyew)]

Tried to re-sync the Disco image on our MAAS server, no luck.

node "modoc" deployed with the following releases today:
Xenial - OK
Bionic - OK
Cosmic - OK
Disco - NOT OK
Eoan - NOT OK

[Manoj Iyer (manjo)]

I was able to deploy Disco on dryden Power8

ubuntu@dryden:~$ uname -a
Linux dryden 5.0.0-20-generic #21-Ubuntu SMP Mon Jun 24 09:31:42 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
ubuntu@dryden:~$ cat /etc/issue
Ubuntu 19.04 \n \l

ubuntu@dryden:~$

[Andrew Cloke (andrew-cloke)]

Adding curtin as possibly impacted - as the description contains an error log emanating from curtin.

I also wonder if this could be associated with multipath (only a thought).

[Ryan Harper (raharper)]

Can you get the curtin debug logs?

https://discourse.maas.io/t/getting-curtin-debug-logs/169

[Manoj Iyer (manjo)]

I was able to deploy Power8 Wichita (Tuleta) with Disco

ubuntu@wichita:~$ uname -a
Linux wichita 5.0.0-20-generic #21-Ubuntu SMP Mon Jun 24 09:31:42 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
ubuntu@wichita:~$ cat /etc/issue
Ubuntu 19.04 \n \l

ubuntu@wichita:~$

[Manoj Iyer (manjo)]

Kernel team,

We may need further debug information from Modoc as to why its failing install, please respond to server teams request for logs, I don't have access to modoc since its not in Power MAAS. Also, if this is blocking Power8 kernel tests please use Wichita temporarily while the issue with modoc is being worked on.

[Ryan Harper (raharper)]

I suspect this is related to multipath-tools being installed and present in disco+ images.

The lastest curtin sru has enabled additional multipath support;
it's likely there's a bug in handling multipath on these systems.

If you can, run a debug (verbose=True) deployment on systems with multipath
so we can see what's failing.

[Blake Rouse (blake-rouse)]

If you are running the latest MAAS, curtin is already setup to run in debug mode by default.

[Po-Hsu Lin (cypressyew)]

Log for "modoc" collected with instruction in comment #7

Thanks Manoj, I will use that wichita for the test temporarily.

[Po-Hsu Lin (cypressyew)]

Here is the config.

[Ryan Harper (raharper)]

The curtin used is a bit old, actually prior to multipath clearing
support; can you test
with what's in bionic-updates?

curtin | 19.1-7-g37a7a0f4-0ubuntu1~18.04.1 | bionic-updates |
source, all

This looks to be a duplicate of this:

https://bugs.launchpad.net/curtin/+bug/1813228

On Mon, Jul 22, 2019 at 9:25 PM Po-Hsu Lin <email address hidden> wrote:

> Here is the config.
>
> ** Attachment added: "curtin-config.log"
>
> https://bugs.launchpad.net/maas/+bug/1837042/+attachment/5278594/+files/curtin-config.log
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1837042
>
> Title:
> Failed to deploy Disco on Power8 with MAAS 2.5.2
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1837042/+subscriptions
>

[Po-Hsu Lin (cypressyew)]

Hello Ryan,

I tried to update the curtin-common and python3-curtin
$ dpkg -l | grep curtin
ii curtin-common 19.1-7-g37a7a0f4-0ubuntu1~18.04.1 all Library and tools for curtin installer
ii python3-curtin 19.1-7-g37a7a0f4-0ubuntu1~18.04.1 all Library and tools for curtin installer

And the Disco deployment for "modoc" works now.

Maybe this solution can be apply to the power8-maas too?

[Andrew Cloke (andrew-cloke)]

My understanding is that modoc is managed by the kernel team's (QA?) MAAS server, and that MAAS will use the curtin version present on that MAAS server during node deployment.

Is it possible to update the version of the curtin package on the kernel team's MAAS server, and does that remedy this problem?

[Manoj Iyer (manjo)]

The power8-maas has the latest curtin installed.

ubuntu@maas-dev:~$ dpkg -l "*curtin*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii curtin-common 19.1-726-gb8 all Library and tools for curtin inst
ii python3-curtin 19.1-726-gb8 all Library and tools for curtin inst
ubuntu@maas-dev:~$

[Andrew Cloke (andrew-cloke)]

Note that Power8 nodes deployed by the server team's power8-maas do not exhibit this issue.

[Dan Watkins (oddbloke)]

It sounds to me like this issue has already been addressed in curtin, and some environments need updating to a version in which it is fixed.

I'm going to mark this Invalid, but please feel free to set it back to New[0] if my understanding is incorrect (with a comment explaining why, please!).

[0] We see New bugs in our daily triage report, hence that status rather than any other.

[Po-Hsu Lin (cypressyew)]

@Andrew,
yes that solves the problem. I think the issue for entei might be different.

@Dan,
yes this can be closed.

Thanks all for your help.