restore-storage-configuration fails when a raid device is part of a bcache

[Jason Hobbs]

This is with maas 2.5.0.

To reproduce, configure a raid device for a machine, and then make that raid device part of a bcache (in my test case, I made it the backing device).

Then use the 'restore-storage-configuration' API call to try to restore the storage configuration to default. It will fail, with an error:

ubuntu@dratini:~$ maas root machine restore-storage-configuration drt3nq
Cannot delete block device because its part of a Bcache.

The expected behavior is that maas will restore the configuration regardless of its current state.

[Jason Hobbs (jason-hobbs)]

The only workaround I'm aware of for this is to re-implement restore-storage-configuration outside of MAAS through API calls. That is a huge complex mess because we have to be very careful about the order in which storage devices are removed, and it is extremely slow, doubling the time of reconfiguring nodes in maas.

[Jason Hobbs (jason-hobbs)]

sub'd to field-high

[Blake Rouse (blake-rouse)]

This is most likely due to the nested level of the virtual block devices. Can you provide a stack trace of the error so I can see exactly where in the code it is occurring?

[Jason Hobbs (jason-hobbs)]

There is no traceback, just this:

2019-02-08 00:31:02 regiond: [info] 10.245.208.5 POST /MAAS/api/2.0/machines/drt3nq/?op=restore_storage_configuration HTTP/1.1 --> 400 BAD_REQUEST (referrer: -; agent: Python-httplib2/0.9.2 (gzip))

But you can reproduce this anywhere, so it shouldn't be hard to reproduce on your side.

[Blake Rouse (blake-rouse)]

Strange that it shows as a 400 error. I thought it would be a 500 error that is why I asked for the trace. Will reproduce manually, thanks!

[Andres Rodriguez (andreserl)]

Setting back as incomplete until this can be reproduced!

[Jason Hobbs (jason-hobbs)]

Do you need anything else from me? I've provided complete directions on how to reproduce. Have you attempted to reproduce and failed?

[Jason Hobbs (jason-hobbs)]

Set this back to "New" as "Reporter needs to provide more information" is inaccurate unless you actually need more information from me.

[Andres Rodriguez (andreserl)]

Hi Jason,

Sorry I wasn’t clear, just setting it to incomplete until Blake can
reproduce locally and confirm your reproduction steps yield the same result
for him.

He should mark this confirmed once he is able to confirm.

On Thu, Feb 14, 2019 at 10:25 AM Jason Hobbs <email address hidden>
wrote:

> Do you need anything else from me? I've provided complete directions on
> how to reproduce. Have you attempted to reproduce and failed?
>
> ** Changed in: maas
> Status: Incomplete => New
>
> ** Changed in: maas/2.5
> Status: Incomplete => New
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1815091
>
> Title:
> restore-storage-configuration fails when a raid device is part of a
> bcache
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1815091/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; milestone=2.6.0; status=New;
> importance=Critical; <email address hidden>;
> Launchpad-Bug: product=maas; productseries=2.5; milestone=2.5.2;
> status=New; importance=Critical; <email address hidden>;
> Launchpad-Bug-Tags: cdo-qa cpe-onsite foundations-engine
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl blake-rouse jason-hobbs
> Launchpad-Bug-Reporter: Jason Hobbs (jason-hobbs)
> Launchpad-Bug-Modifier: Jason Hobbs (jason-hobbs)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

[Dmitrii Shcherbakov (dmitriis)]

Tried this on 2.5.2 - MAAS returns 200 and the configs remain as they were.

https://pastebin.canonical.com/p/QpQKdCnQ3W/

Versions (2.5.2 from -proposed ppa on all 3 nodes):
https://pastebin.canonical.com/p/BczQgM6wK6/

 maas root machine restore-storage-configuration sgtp6b | grep bcache
                    "fstype": "bcache-backing",
    "bcaches": [
                        "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-cache",
                "fstype": "bcache-cache",
                        "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-backing",
                "fstype": "bcache-cache",
                "fstype": "bcache-cache",

2019-03-20 11:04:24 regiond: [info] 192.0.2.18 POST /MAAS/api/2.0/machines/sgtp6b/?op=restore_storage_configuration HTTP/1.1 --> 200 OK (referrer: -; agent: Python-
httplib2/0.9.2 (gzip))
2019-03-20 11:04:25 regiond: [info] 192.0.2.18 POST /MAAS/api/2.0/machines/ambatm/?op=restore_storage_configuration HTTP/1.1 --> 200 OK (referrer: -; agent: Python-
httplib2/0.9.2 (gzip))
2019-03-20 11:04:26 regiond: [info] 192.0.2.18 GET /MAAS/api/2.0/machines/?hostname=control-7&domain=maas HTTP/1.1 --> 200 OK (referrer: -; agent: Python-httplib2/0
.9.2 (gzip))
2019-03-20 11:04:26 regiond: [info] 192.0.2.18 GET /MAAS/api/2.0/nodes/sgtp6b/blockdevices/ HTTP/1.1 --> 200 OK (referrer: -; agent: Python-httplib2/0.9.2 (gzip))
2019-03-20 11:04:28 regiond: [info] 192.0.2.18 GET /MAAS/api/2.0/machines/?hostname=compute-3&domain=maas HTTP/1.1 --> 200 OK (referrer: -; agent: Python-httplib2/0

[Dmitrii Shcherbakov (dmitriis)]

Code path:

https://paste.ubuntu.com/p/xHpvdvRXGP/

So MAAS clears the storage by applying a default storage layout.

Tracing shows that self.skip_storage is set to True (I used that option during re-commissioning once or twice but it should not affect this operation):

https://paste.ubuntu.com/p/sDS2xjcn3k/

3214 -> if self.skip_storage:
3215 return
3216 storage_layout = Config.objects.get_config("default_storage_layout")

(Pdb) n
> /usr/lib/python3/dist-packages/maasserver/models/node.py(3215)set_default_storage_layout()
-> return

So MAAS simply returns 200 skipping storage reconfiguration.

[Dmitrii Shcherbakov (dmitriis)]

Filed a separate bug for #10 and #11:

https://bugs.launchpad.net/maas/+bug/1820998