New curtin regresses older centos images due to AttributeError: module 'curtin.commands.curthooks' has no attribute 'write_files'

[john]

MAAS 2.2.2 no longer boots custom centos image

I upgraded my my server from 2.2.0.
apt-get update
apt-get upgrade

I created a new custom image as I did previously with 2.2.0
maas admin boot-resources create name=custom/centos7-netfix title="CentOS 7 - netfix" architecture=amd64/generic content@=/tmp/centos7-root-netfix2.tgz

I tried to deploy it: failed with no such file or directory
See attached image screen shot of console window

The maas snapshot directory looks like this
root@MAAS:/var/lib/maas/boot-resources/snapshot-20171112-004041/custom/amd64/generic/centos7-netfix/uploaded# ls
root-tgz
root@MAAS:/var/lib/maas/boot-resources/snapshot-20171112-004041/custom/amd64/generic/centos7-netfix/uploaded#

[Andres Rodriguez (andreserl)]

John,

Could you please provide the curtin version you are using? "dpkg -l | grep curtin"

Could you please attach logs:

/var/log/maas/*.log

Also attach the output of the machine that's failing to deploy:

maas admin boot-resources read
maas admin machine read id=<system-id of the machine>
maas admin rack-controller read id=<system-id of rack controller>

And lastly, please also provide the installation log. I think this is the correct result_type otherwise you can do it via the UI.

maas admin node-results read system_id=sa4whm result_type=1 | grep data | cut -d'"' -f4 | base64 --decode

[john (g-john-p)]

missing attrtibute found in deployment log:

2017-11-12 00:44:39 (46.5 MB/s) - written to stdout [340052866/340052866]

Traceback (most recent call last):
  File "/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks.py", line 595, in <module>
    main()
  File "/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks.py", line 537, in main
    curthooks.write_files(cfg, target)
AttributeError: module 'curtin.commands.curthooks' has no attribute 'write_files'
Unexpected error while running command.
Command: ['/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
curtin: Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: Traceback (most recent call last):
          File "/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks.py", line 595, in <module>
            main()
          File "/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks.py", line 537, in main
            curthooks.write_files(cfg, target)
        AttributeError: module 'curtin.commands.curthooks' has no attribute 'write_files'
        Unexpected error while running command.
        Command: ['/tmp/tmpa_i9n7t_/target/curtin/curtin-hooks']
        Exit code: 1
        Reason: -
        Stdout: ''
        Stderr: ''

Stderr: ''

[john (g-john-p)]

boot-resources read : http://paste.ubuntu.com/25947773/

/var/log/maas.log http://paste.ubuntu.com/25947789/

/var/log/rackd.log: http://paste.ubuntu.com/25947800/

/var/log/regiond.log : http://paste.ubuntu.com/25947806/

[john (g-john-p)]

root@MAAS:/var/log/maas# dpkg -l | grep curtin
ii curtin-common 0.1.0~bzr532-0ubuntu1~16.04.1 all Library and tools for curtin installer
ii python3-curtin 0.1.0~bzr532-0ubuntu1~16.04.1 all Library and tools for curtin installer

[john (g-john-p)]

maas admin machines read | jq '.[] | {hostname:.hostname, system_id: .system_id, status:.status}' --compact-output
{"hostname":"MACH1","system_id":"t8fqk7","status":6}
{"hostname":"MACH5","system_id":"daey3c","status":4}
{"hostname":"MACH4","system_id":"7atrke","status":4}
{"hostname":"MACH3","system_id":"hdxmfp","status":4}
{"hostname":"MACH2","system_id":"te68d8","status":11}
{"hostname":"MACH6","system_id":"mwsdrn","status":4}

maas admin machines read http://paste.ubuntu.com/25947937/

maas admin rack-controllers read http://paste.ubuntu.com/25947950/

[john (g-john-p)]

note-result output log: http://paste.ubuntu.com/25948012/

[john (g-john-p)]

New update.
I created a whole new maas environment from scratch; nothing was upgraded, installed from latest repo.
MAAS version: 2.2.2 (6099-g8751f91-0ubuntu1~16.04.1
The same problem exists installing the custom centos image. same error message:
  File "/tmp/tmpngh0fprp/target/curtin/curtin-hooks.py", line 595, in <module>
            main()
          File "/tmp/tmpngh0fprp/target/curtin/curtin-hooks.py", line 537, in main
            curthooks.write_files(cfg, target)
        AttributeError: module 'curtin.commands.curthooks' has no attribute 'write_files'

So this problem is in the repo itself.

my colleague is running 2.2.2 installed a month ago - he is not seeing this problem.

This must have been an update in the last 3 weeks that broke this - if this helps.

[john (g-john-p)]

The problem exist in curtin 532

dpkg -l | grep curtin
ii curtin-common 0.1.0~bzr532-0ubuntu1~16.04.1 all Library and tools for curtin installer
ii python3-curtin 0.1.0~bzr532-0ubuntu1~16.04.1 all Library and tools for curtin installer

but does not exists in 505:
 curtin-common 0.1.0~bzr505-0ubuntu1~16.04.1 all Library and tools for curtin installer
 python3-curtin 0.1.0~bzr505-0ubuntu1~16.04.1 all Library and tools for curtin installer
root@maas:~#

[john (g-john-p)]

problem is with curtin-common and python3-curtin version 532

[john (g-john-p)]

the image was created using instruction from Michael Latrou:
-- snip from his email to us -- 2016 ---

It was brought to my attention that you have opened a bug and you are immediately affected by it:
https://bugs.launchpad.net/juju/+bug/1646847

This issue is actually related to MAAS, and the handling of multiple interfaces for non-Ubuntu images (Ubuntu images work as expected).

We have a workaround for you, until the issue is resolved upstream.

You will need to build a custom image for CentOS and import it to MAAS. Here is the process:

On the MAAS server, we will download the upstream CentOS image and we will modify the curtin config, which is responsible for handling the network interfaces configuration. We will then repack the image and manually import it to MAAS.

    wget https://people.canonical.com/~iatrou/curtin-hooks-20170110.py.gz -O - | gunzip > curtin-hooks.py
    wget http://images.maas.io/ephemeral-v2/daily/centos70/amd64/20160801_01/root-tgz -O centos7-root.tgz
    mkdir centos7-root && cd centos7-root
    sudo tar -zxvf ../centos7-root.tgz
    sudo cp ../curtin-hooks.py curtin/
    sudo chown root: curtin/curtin-hooks.py
    sudo chmod 755 curtin/curtin-hooks.py
    sudo tar -zcvf ../centos7-root-netfix.tgz .
    sudo maas admin boot-resources create name=custom/centos7-netfix title="CentOS 7 - netfix" architecture=amd64/generic content@=../centos7-root-netfix.tgz

At this point you should be able to provision a machine from MAAS, using the imported image, and by having a second interface configured on it.

--- end snip --

[Andres Rodriguez (andreserl)]

@John,

On your comment above, please keep in mind that MAAS with newer version of curtin now support custom network configuration and as such, you no longer need to generate a custom image.

That said, curtin will continue to be update to support writing files to remain backward compatible.

[john (g-john-p)]

@ Andres,
thanks for comment #12. Can you point me to any docs where on how to do this? Was speaking with Narinder today and he said that you don't support Centos w/ multiple network configs. We are just at a loss on how to proceed. if there is a better way to support centos than providing our own image, that would be ideal.

[Andres Rodriguez (andreserl)]

@John,

Network support for CentOS is officially introduced with MAAS 2.3, but not to worry, as it will work with MAAS 2.2 just the same.

There's nothing that you really need to do, other than configure your interfaces in the machine in MAAS the same way you would do with Ubuntu.

[Andres Rodriguez (andreserl)]

Oh, note that you need the latest version of curtin (532)

[john (g-john-p)]

@Andres,

We did test the standard Centos7 and are running into other problems. The network interfaces do come up, however, the machine was created with no swap. there maybe other issues, but this is the next in the line. Is there a way to customize this image or tell MAAS to create a swap for the VM.

[Ryan Harper (raharper)]

I'm not sure if the centos curthooks handle it, but curtin does have
configuration options for swap files

http://curtin.readthedocs.io/en/latest/topics/config.html#swap

swap:
  filename: swap.img
  size: None
  maxsize: 4GB

On Wed, Nov 15, 2017 at 5:33 PM, john <email address hidden> wrote:

> @Andres,
>
> We did test the standard Centos7 and are running into other problems.
> The network interfaces do come up, however, the machine was created with
> no swap. there maybe other issues, but this is the next in the line.
> Is there a way to customize this image or tell MAAS to create a swap for
> the VM.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1731709
>
> Title:
> New curtin regresses older centos images due to AttributeError: module
> 'curtin.commands.curthooks' has no attribute 'write_files'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1731709/+subscriptions
>

[john (g-john-p)]

@Ryan,

We were able to successfully deploy Oracle RAC using the CentOS7 Image provided by MAAS 2.2.2 with curtin 532. As described Centos at 532 now support multiple interfaces so we do not need to provide a custom image. We also added the swap file creation into our own charms since MAAS does not automatically do this.

Thank you for your help. You can now close this issue.

[Scott Moser (smoser)]

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[Scott Moser (smoser)]

A fix for this bug is being SRU to Ubuntu 16.04 and 17.10 under bug 1743618.