CentOS 6 deployment fails to deploy with Bionic

[Anthony Yarbrough]

* Operating system: Ubuntu 18.04
* MAAS version: 2.4.0 (6981-g011e51b7a-0ubuntu1~18.04.1)

root@maas1:~# ls -lah /var/log/maas/
total 16M
drwxr-xr-x 4 maas maas 4.0K May 23 06:25 .
drwxrwxr-x 18 root syslog 4.0K Jun 6 06:25 ..
-rw-r--r-- 1 syslog syslog 932K Jun 6 15:41 maas.log
drwxr-x--- 2 proxy proxy 4.0K Jun 6 06:25 proxy
-rw-r--r-- 1 maas maas 5.0M Jun 6 15:42 rackd.log
-rw-r--r-- 1 maas maas 9.3M Jun 6 15:49 regiond.log
-rw-r--r-- 1 maas maas 391K May 23 06:24 regiond.log.1.gz
drwxr-xr-x 6 syslog syslog 4.0K May 31 13:56 rsyslog

# dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-======================================-============-=================================================
ii maas 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server common files
ii maas-dhcp 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS DHCP server
un maas-dns <none> <none> (no description available)
ii maas-proxy 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Rack Controller for MAAS
ii maas-region-api 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server provisioning libraries (Python 3)

I'm running into an issue related to CentOS 6 deployments failing with multiple MAAS installations using different network configurations (one simple and one complex) for the machines being deployed. I thought it might have been related to the complex network config however I've confirmed the same problem exists with what I would consider a simple network config. Below is the error output found in the installation log once the CentOS 6 deployment fails:

----------------------------------------------------------------------------------

Traceback (most recent call last):

  File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 393, in <module>

    main()

  File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 376, in main

    grub_root = get_grub_root(target)

  File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 208, in get_grub_root

    return regex.groups()[0]

AttributeError: 'NoneType' object has no attribute 'groups'

Unexpected error while running command.

Command: ['/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks']

Exit code: 1

Reason: -

Stdout: ''

Stderr: ''

curtin: Installation failed with exception: Unexpected error while running command.

Command: ['curtin', 'curthooks']

Exit code: 3

Reason: -

Stdout: Traceback (most recent call last):

          File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 393, in <module>

            main()

          File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 376, in main

            grub_root = get_grub_root(target)

          File "/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks.py", line 208, in get_grub_root

            return regex.groups()[0]

        AttributeError: 'NoneType' object has no attribute 'groups'

        Unexpected error while running command.

        Command: ['/tmp/tmpzkdjc8hq/target/curtin/curtin-hooks']

        Exit code: 1

        Reason: -

        Stdout: ''

        Stderr: ''

Stderr: ''

----------------------------------------------------------------------------------

Here are the steps to recreate the problem:

1.) Enlist the server into MAAS and rename it to the appropriate server name
2.) Mark the machine as broken and configure the network interfaces as follows:
    - eth0 - private subnet used for MAAS deployments
    - eth1 - public /29 subnet dedicated to specific server (gateway to internet here)
3.) Mark the machine as fixed and then commission the machine ensuring to select the commissioning option to retain the current network config
4.) Once the server has been successfully commissioned, acquire the server (admin user) and deploy with CentOS 6.
5.) The server indicates the installation has begun and CentOS 6 cloud installer image is downloaded for install/extraction however the error listed above is produced immediately after the image download then the status changes to 'Failed Deployment'

I have no problems successfully installing via MAAS operating systems Ubuntu 14.04, Ubuntu 16.04, Ubuntu 18.04 and CentOS 7 using the same exact steps listed above.

If you need any additional information or have any questions please let me know and I'll be happy to oblige.

Thanks!

[Andres Rodriguez (andreserl)]

Hi there,

I've deployed centos6 and i cannot reproduce your issue https://pastebin.ubuntu.com/p/t7H9hbQMKZ/

Could you confirm a few things:

1. Which image have you used?
2. Is there any customization done, or just straight forward deployment?

[Anthony Yarbrough (virento)]

I am using the stock image provided by maas.io and no customizations have been done, everything was left at default.

[Andres Rodriguez (andreserl)]

Can you provide the output of your machine:

maas 'user' machine read <system-id>

and:

maas 'user' machine get-curtin-config <system_id>

[Lee Trager (ltrager)]

I was able to reproduce this when I had my commissioning operating system set to Bionic, could you please try 16.04 Xenial?

[Daniel Souza (danielsouzasp)]

Hello,

I'm facing this same error with MaaS 2.3.5 to deploy CentOS 6, any other OS works fine.

The node has 1x NVMe 1 TB + 2x SSD 500 GB, I'm trying this config:
Name |Model | Serial|BootDevice typeUsed forHealth
nvme0n1 |Physical | MBR |partitioned with 1 partition Ok
nvme0n1-part1 |Partition| ext4 |formatted filesystem mounted at /

Error:
Traceback (most recent call last):
  File "/tmp/tmph5qpnwg3/target/curtin/curtin-hooks.py", line 393, in <module>
    main()
  File "/tmp/tmph5qpnwg3/target/curtin/curtin-hooks.py", line 376, in main
    grub_root = get_grub_root(target)
  File "/tmp/tmph5qpnwg3/target/curtin/curtin-hooks.py", line 208, in get_grub_root
    return regex.groups()[0]
AttributeError: 'NoneType' object has no attribute 'groups'
Unexpected error while running command.
Command: ['/tmp/tmph5qpnwg3/target/curtin/curtin-hooks']
-

If I ran "maas myadmin machine get-curtin-config" there is no custom storage config.

Does someone know it is known bug?

[Andres Rodriguez (andreserl)]

Hi Daniel,

Could you please confirm where do these images come from ? Note that MAAS 2.3.5 doesn't support storage configuration for CentOS, this support is only available in 2.5+.

Also, please make sure you have xenial as the commissioning image.

[Daniel Souza (danielsouzasp)]

Hi Andres,

I'm using default MaaS repository and I also set xenial as commissioning image, so I'll think of update to 2.5.

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]

[Fabio Augusto Miranda Martins (fabio.martins)]

I can also reproduce this problem with MAAS 2.4.2 + Ubuntu 18.04.2 + kernel 4.15.0-45. I'm using the CentOS 6 image synced from maas.io through the regular MAAS Web GUI (no curtin customizations):

https://pastebin.ubuntu.com/p/n235yRgvdk/

[Fabio Augusto Miranda Martins (fabio.martins)]

Log for the failure

[Ryan Harper (raharper)]

The source of the issue is around the ext4 filesystem and grub 0.97[1]. In Bionic, ext4 has 64bit and metadata_csum features which grub 0.97 in Centos6 cannot handle read. There are couple of workarounds here:

A) Use Xenial to deploy Centos6
B) If you use bionic, then you'll need custom storage where you create a /boot partition smaller than 512Mb and put either ext2 or ext3 as the filesystem format.

1. https://bugzilla.redhat.com/show_bug.cgi?id=1099237

[Ryan Harper (raharper)]

Scratch separate /boot, the curtin-hooks in centos6 do not support separate /boot. So, switching the filesystem format to ext3 should work fine.

[Stefan Miktov Ivanov (tension183)]

I have this issue only wen i use software RAID here it is the messeges.

[Björn Tillenius (bjornt)]

So, as I understand it, the workaround is to set Xenial to be the commissioning environment, but we need to make it so that Xenial is used when deploying Centos6.

[Jose Delarosa (jdelaros1)]

I am seeing the exact same problem, even when commissioning with Xenial (both ga and hwe kernels). Was there ever a more permanent solution?

[Adam Collard (adam-collard)]

We plan to address this as a feature that explicitly models the dependencies between target deployment image and the ephemeral image that MAAS uses to run curtin in (and install that image).

This will, for example, give MAAS the knowledge that CentOS 6 requires Xenial, CentOS 7 requires Focal etc.

Internally, see PF-3237