[2.4] Creating bcache partitions fails in some situations

[Ashley Lai]

MAAS version: 2.3.2 (6485-ge93e044-0ubuntu1~16.04.1)

A disk with multiple partitions and if one of the partition is bcache, MAAS umount / mount point. It has to do with the order when the bcache partitions are created. The workaround is to create the non-bcache partition and mount it at /. Then create the rest of the partions with bcache.

create sda_part1
create sda_part2_bcache
mount sda_part1 / <=== maas will umount this right after it is mounted

work around:
create sda_part1
mount sda_part1 /
create sda_part2_bcache
create sda_part3_bcache

2018-05-29 20:00:59 maasserver.websockets.protocol: [critical] Error on request (175) machine.update_disk: <ul class="errorlist"><li>__all__<ul class="errorlist nonfield"><li>Cannot format block device with a partition table.</li></ul></li></ul>

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 842, in worker
    return target()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 875, in callInContext
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 232, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 754, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 561, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 551, in update_disk
    params.get('mount_options', ''))
    params.get('mount_options', ''))
  File "/usr/lib/python3/dist-packages/maasserver/websockets/handlers/machine.py", line 497, in update_blockdevice_filesystem
    raise HandlerError(form.errors)
maasserver.websockets.base.HandlerError: <ul class="errorlist"><li>__all__<ul class="errorlist nonfield"><li>Cannot format block device with a partition table.</li></ul></li></ul>

[Andres Rodriguez (andreserl)]

Another work around i have tested in 2.4 is:

1. create unformatted partition 1 (sda-part1)
2. Create unformatted partition 2 (sda-part2)
3. Select sda-part2 and 'Create bcache' and mount in /var/lib/lxc
4. Select sda-part1 and 'Edit' and mount to /

[Blake Rouse (blake-rouse)]

There is no workaround for 2.3 or 2.4. This is already resolved in 2.5.

Landing commit from 2.5 to 2.4 and 2.3 will solve the issue.

[Ashley Lai (alai)]

MAAS is upgraded with ppa:maas/proposed and the node still fail the deployment.

Here's the curtin config: https://pastebin.canonical.com/p/N49rHGS83W/

ubuntu@dcs1-clm-inf1:~$ dpkg -l | grep maas
ii maas-cli 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all MAAS client and command-line interface
ii maas-common 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all MAAS server common files
ii maas-dhcp 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all MAAS DHCP server
ii maas-dns 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all MAAS DNS server
ii maas-proxy 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all Rack Controller for MAAS
ii maas-region-api 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all Region Controller for MAAS
ii python3-django-maas 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.3.4-6504-gac6a15b-0ubuntu1~16.04.1 all MAAS server provisioning libraries (Python 3)
ubuntu@dcs1-clm-inf1:~$

[Ashley Lai (alai)]

Note that the root file system is on sda and boot option for sda is checked. Also sda has the root filesystem / on it by default after commission.

http://people.canonical.com/~alai/docapostNode11a.png
http://people.canonical.com/~alai/docapostNode11b.png

[Andres Rodriguez (andreserl)]

Ashley,

Please attach the installation log (under the logs tab), the rsyslog log (/var/log/maas/rsyslog/<machine-name>/<date>/messages and the latest events of the node event log (events tab)

[Ashley Lai (alai)]

Hi Andres,

Here's the installation log from the logs tab:
https://pastebin.canonical.com/p/2yXhG4wsBp/

The event tab can be found at
http://people.canonical.com/~alai/maasNode11EventTab.png

The rsyslog messages file is attached.

Please let me know if you need anything else.

[Blake Rouse (blake-rouse)]

The storage config is correct and the installation was successful. We need to determine why the machine didn't boot after it installed to the disk.

Do you have a console log of the machine deploying? Specifically after the installation to disk and its rebooting for the first time into the newly installed system.

[Ashley Lai (alai)]

http://people.canonical.com/~alai/maasNoMBRmagic.png

It looks similar to this bug:
https://bugs.launchpad.net/maas/+bug/1671605

[Andres Rodriguez (andreserl)]

Ashley,

That looks to me as if the disk on the boot order of the machine (BIOS) is different from the disk that / was installed. Can you ensure the firmware has the specific disk as the boot disk in the boot order?

e.g. The boot order should be:

1. PXE. 2. The disk where you installed /

[Andres Rodriguez (andreserl)]

FWIW, alternatively, you can use EFI instead of Legacy booting, which
wouldn’t have this problem.

On Thu, Jun 7, 2018 at 8:00 AM Andres Rodriguez <email address hidden>
wrote:

> Ashley,
>
> That looks to me as if the disk on the boot order of the machine (BIOS)
> is different from the disk that / was installed. Can you ensure the
> firmware has the specific disk as the boot disk in the boot order?
>
> e.g. The boot order should be:
>
> 1. PXE. 2. The disk where you installed /
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1774058
>
> Title:
> [2.4] Creating bcache partitions fails in some situations
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1774058/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; status=Fix Released; importance=High;
> assignee=None;
> Launchpad-Bug: product=maas; productseries=2.3; milestone=2.3.4;
> status=Fix Committed; importance=Critical; assignee=
> <email address hidden>;
> Launchpad-Bug: product=maas; productseries=2.4; milestone=2.4.1;
> status=Fix Committed; importance=Critical; assignee=
> <email address hidden>;
> Launchpad-Bug-Tags: cdo-qa cdo-qa-blocker cpe-onsite
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: alai andreserl blake-rouse
> Launchpad-Bug-Reporter: Ashley Lai (alai)
> Launchpad-Bug-Modifier: Andres Rodriguez (andreserl)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer