maas fails to deploy ibm power nodes

[rory schramm]

maas server is intel x86
works fine with x86 hardware. However, when I try do deploy our ibm power nodes via maas webui. The deployment is failing with a "Failed Deployment" status. ipmi sol console output is attached as well as the cloud init logs i pulled off the server.

power on the power8 nodes is configured as ipmi with the ip address and the password entered, username is left blank.

MAAS Version 1.9.0 (rc4+bzr4533)
Ubuntu 14.04.3

Nodes do come up and I can ssh in. However, os seems to be missing packages/components.

For example If i try to bring up eth6 via dhcp I get the following error:

## This file is generated by cloud-initramfs-dyn-netconf
auto lo
iface lo inet loopback
manual eth4
iface eth4 inet dhcp
        dns-nameservers 192.168.200.30
        dns-search maas

iface eth6 inet dhcp

root@pw8-07:~# ifup eth6
dhclient: error while loading shared libraries: libc.so.6: cannot stat shared object: Permission denied
Failed to bring up eth6.

maas log shows the following.

/var/log/maas/maas.log:

Jan 8 08:45:10 maas maas.node: [INFO] pw8-07: Status transition from READY to ALLOCATED
Jan 8 08:45:10 maas maas.node: [INFO] pw8-07: allocated to user root
Jan 8 08:45:11 maas maas.interface: [INFO] Allocated automatic static IP address 172.16.91.48 for eth4 on pw8-07.
Jan 8 08:45:11 maas maas.interface: [INFO] Allocated automatic static IP address 192.168.200.109 for eth6 on pw8-07.
Jan 8 08:45:11 maas maas.node: [INFO] pw8-07: Status transition from ALLOCATED to DEPLOYING
Jan 8 08:45:11 maas maas.node: [INFO] pw8-08: Status transition from READY to ALLOCATED
Jan 8 08:45:11 maas maas.node: [INFO] pw8-08: allocated to user root
Jan 8 08:45:11 maas maas.dns: [INFO] Generating new DNS zone file for maas
Jan 8 08:45:12 maas maas.dns: [INFO] Generating new DNS zone file for 91.16.172.in-addr.arpa
Jan 8 08:45:12 maas maas.interface: [INFO] Allocated automatic static IP address 172.16.91.49 for eth4 on pw8-08.
Jan 8 08:45:12 maas maas.dns: [INFO] Generating new DNS zone file for 200.168.192.in-addr.arpa
Jan 8 08:45:13 maas maas.power: [INFO] Changing power state (on) of node: pw8-07 (node-a0632f70-b5a6-11e5-b914-000af71c7982)
Jan 8 08:45:13 maas maas.drivers.power.ipmi: [WARNING] Failed to change the boot order to PXE 192.168.200.12: ERROR: Section post-commit `Chassis_Boot_Flags'
Jan 8 08:45:13 maas maas.interface: [INFO] Allocated automatic static IP address 192.168.200.110 for eth6 on pw8-08.
Jan 8 08:45:13 maas maas.node: [INFO] pw8-08: Status transition from ALLOCATED to DEPLOYING
Jan 8 08:45:13 maas maas.dns: [INFO] Generating new DNS zone file for maas
Jan 8 08:45:13 maas maas.dns: [INFO] Generating new DNS zone file for 91.16.172.in-addr.arpa
Jan 8 08:45:13 maas maas.dns: [INFO] Generating new DNS zone file for 200.168.192.in-addr.arpa
Jan 8 08:45:13 maas maas.power: [INFO] Changing power state (on) of node: pw8-08 (node-647c900a-b5a6-11e5-a738-000af71c7982)
Jan 8 08:45:13 maas maas.drivers.power.ipmi: [WARNING] Failed to change the boot order to PXE 192.168.200.13: ERROR: Section post-commit `Chassis_Boot_Flags'
Jan 8 08:45:14 maas maas.drivers.power.ipmi: [WARNING] Failed to change the boot order to PXE 192.168.200.12: ERROR: Section post-commit `Chassis_Boot_Flags'
Jan 8 08:45:14 maas maas.drivers.power.ipmi: [WARNING] Failed to change the boot order to PXE 192.168.200.13: ERROR: Section post-commit `Chassis_Boot_Flags'
Jan 8 08:45:16 maas maas.drivers.power.ipmi: [WARNING] Failed to change the boot order to PXE 192.168.200.12: ERROR: Section post-commit `Chassis_Boot_Flags'
Jan 8 08:45:16 maas maas.drivers.power.ipmi: [WARNING] Failed to change the boot order to PXE 192.168.200.13: ERROR: Section post-commit `Chassis_Boot_Flags'
Jan 8 08:45:18 maas maas.drivers.power.ipmi: [WARNING] Failed to change the boot order to PXE 192.168.200.12: ERROR: Section post-commit `Chassis_Boot_Flags'
Jan 8 08:45:19 maas maas.drivers.power.ipmi: [WARNING] Failed to change the boot order to PXE 192.168.200.13: ERROR: Section post-commit `Chassis_Boot_Flags'
Jan 8 08:45:23 maas maas.power: [INFO] Changed power state (on) of node: pw8-07 (node-a0632f70-b5a6-11e5-b914-000af71c7982)
Jan 8 08:45:24 maas maas.power: [INFO] Changed power state (on) of node: pw8-08 (node-647c900a-b5a6-11e5-a738-000af71c7982)
Jan 8 08:51:19 maas maas.node: [INFO] pw8-07: Status transition from DEPLOYING to FAILED_DEPLOYMENT
Jan 8 08:51:19 maas maas.node: [ERROR] pw8-07: Marking node failed: Installation failed (refer to the installation log for more information).
Jan 8 08:51:34 maas maas.node: [INFO] pw8-08: Status transition from DEPLOYING to FAILED_DEPLOYMENT
Jan 8 08:51:34 maas maas.node: [ERROR] pw8-08: Marking node failed: Installation failed (refer to the installation log for more information).
Jan 8 08:51:39 maas maas.node_query: [INFO] pw8-07: Power is on.
Jan 8 08:51:54 maas maas.node_query: [INFO] pw8-08: Power is on.
Jan 8 08:52:11 maas maas.import-images: [INFO] Writing boot image metadata and iSCSI targets.
Jan 8 08:52:11 maas maas.import-images: [INFO] Installing boot images snapshot /var/lib/maas/boot-resources/snapshot-20160108-164908
Jan 8 08:52:25 maas maas.import-images: [INFO] Updating boot image iSCSI targets.
Jan 8 08:52:26 maas maas.import-images: [INFO] Cleaning up old snapshots and cache.

[Douglas Sikora (doug-s1k0ra)]

Rory,

on maas server post output of : ldd `which ifup`

then ls -l each of the libraries returned (should be only 2)

Please post output here.

Thanks!

[rory schramm (roryschramm)]

the library errors were not happening on the maas sever. It was happening on the power node that fails deployment.

root@maas:/home/ubuntu# ldd `which ifup`
        linux-vdso.so.1 => (0x00007ffc349f4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9b142ac000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f9b14671000)

root@maas:/home/ubuntu# ls -ltrah /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Dec 21 12:00 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.19.so

root@maas:/home/ubuntu# ls -ltrah /lib64/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root 32 Dec 21 12:00 /lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.19.so

Also I tried deploying 15.10 to power. However, that failed with a kernel panic. Let me redeploy and get some logs for that.

I talked to our IBM rep about this and on power there is a special partition where the bootloader is supposed to get installed to called the "prep" partition. Grub is throwing an error and failing because its not deploying to this prep partition.

Installing for powerpc-ieee1275 platform.
grub-install: error: the chosen partition is not a PReP partition.
failed to install grub!
Unexpected error while running command.
Command: ['install-grub', '/tmp/tmp7CiDTI/target', '/dev/sda']
Exit code: 1
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
....

[Mark Brown (mstevenbrown)]

Possibly related to https://bugs.launchpad.net/maas/+bug/1523779 ?

[rory schramm (roryschramm)]

there are definitely some missing libraries/packages on power that's being caused by the grub error.

below is for ubuntu 14.04.3 attempting to deploy using vivid kernel via maas

root@pw8-07:/home/ubuntu# ldd `which ifup`
        linux-vdso64.so.1 => (0x00003fff9a3c0000)
        libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fff9a1e0000)
        /lib64/ld64.so.2 (0x00003fff9a3e0000)
root@pw8-07:/home/ubuntu# ls -ltrah libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6
ls: error while loading shared libraries: /lib/powerpc64le-linux-gnu/libc.so.6: file too short
root@pw8-07:/home/ubuntu# ls -ltrah /lib64/ld64.so.2
ls: error while loading shared libraries: /lib/powerpc64le-linux-gnu/libc.so.6: file too short
root@pw8-07:/home/ubuntu# ls -ltrah /lib/powerpc64le-linux-gnu/
ls: error while loading shared libraries: /lib/powerpc64le-linux-gnu/libc.so.6: file too short
root@pw8-07:/home/ubuntu# ls -ltrah /lib/
ls: error while loading shared libraries: /lib/powerpc64le-linux-gnu/libc.so.6: file too short
root@pw8-07:/home/ubuntu# ls -ltrah /
ls: error while loading shared libraries: /lib/powerpc64le-linux-gnu/libc.so.6: file too short
root@pw8-07:/home/ubuntu# ls -l
ls: error while loading shared libraries: /lib/powerpc64le-linux-gnu/libc.so.6: file too short
root@pw8-07:/home/ubuntu# dpkg --list
dpkg: error while loading shared libraries: /lib/powerpc64le-linux-gnu/libc.so.6: file too short

[Douglas Sikora (doug-s1k0ra)]

yes that was what kiko thought at first to, it may be related and i think
they triaged them together. if you have history in #maas you can see what
he and smoser said

Doug Sikora
Solutions Architect , Canonical Ltd.
[phone]

On Fri, Jan 8, 2016 at 3:01 PM, Mark Brown <email address hidden> wrote:

> Possibly related to https://bugs.launchpad.net/maas/+bug/1523779 ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1532271
>
> Title:
> maas fails to deploy ibm power nodes
>
> Status in MAAS:
> New
>
> Bug description:
>
> maas server is intel x86
> works fine with x86 hardware. However, when I try do deploy our ibm
> power nodes via maas webui. The deployment is failing with a "Failed
> Deployment" status. ipmi sol console output is attached as well as the
> cloud init logs i pulled off the server.
>
> power on the power8 nodes is configured as ipmi with the ip address
> and the password entered, username is left blank.
>
> MAAS Version 1.9.0 (rc4+bzr4533)
> Ubuntu 14.04.3
>
> Nodes do come up and I can ssh in. However, os seems to be missing
> packages/components.
>
> For example If i try to bring up eth6 via dhcp I get the following
> error:
>
>
> ## This file is generated by cloud-initramfs-dyn-netconf
> auto lo
> iface lo inet loopback
> manual eth4
> iface eth4 inet dhcp
> dns-nameservers 192.168.200.30
> dns-search maas
>
> iface eth6 inet dhcp
>
>
> root@pw8-07:~# ifup eth6
> dhclient: error while loading shared libraries: libc.so.6: cannot stat
> shared object: Permission denied
> Failed to bring up eth6.
>
> maas log shows the following.
>
> /var/log/maas/maas.log:
>
> Jan 8 08:45:10 maas maas.node: [INFO] pw8-07: Status transition from
> READY to ALLOCATED
> Jan 8 08:45:10 maas maas.node: [INFO] pw8-07: allocated to user root
> Jan 8 08:45:11 maas maas.interface: [INFO] Allocated automatic static
> IP address 172.16.91.48 for eth4 on pw8-07.
> Jan 8 08:45:11 maas maas.interface: [INFO] Allocated automatic static
> IP address 192.168.200.109 for eth6 on pw8-07.
> Jan 8 08:45:11 maas maas.node: [INFO] pw8-07: Status transition from
> ALLOCATED to DEPLOYING
> Jan 8 08:45:11 maas maas.node: [INFO] pw8-08: Status transition from
> READY to ALLOCATED
> Jan 8 08:45:11 maas maas.node: [INFO] pw8-08: allocated to user root
> Jan 8 08:45:11 maas maas.dns: [INFO] Generating new DNS zone file for
> maas
> Jan 8 08:45:12 maas maas.dns: [INFO] Generating new DNS zone file for
> 91.16.172.in-addr.arpa
> Jan 8 08:45:12 maas maas.interface: [INFO] Allocated automatic static
> IP address 172.16.91.49 for eth4 on pw8-08.
> Jan 8 08:45:12 maas maas.dns: [INFO] Generating new DNS zone file for
> 200.168.192.in-addr.arpa
> Jan 8 08:45:13 maas maas.power: [INFO] Changing power state (on) of
> node: pw8-07 (node-a0632f70-b5a6-11e5-b914-000af71c7982)
> Jan 8 08:45:13 maas maas.drivers.power.ipmi: [WARNING] Failed to change
> the boot order to PXE 192.168.200.12: ERROR: Section post-commit
> `Chassis_Boot_Flags'
> Jan 8 08:45:13 maas maas.interface: [INFO] Allocate...

[rory schramm (roryschramm)]

I'm not sure they're related - that issue seems to be specific to using a lvm partition layout as opposed to flat.

import log lines from that bug show cloud init/grub throwing a bunch of lvm pv errors as opposed to the prep partition errors I'm seeing in my logs.

[ 515.215462] cloud-init[2019]: Installing for powerpc-ieee1275 platform.
[ 519.989131] cloud-init[2019]: File descriptor 3 (pipe:[82785]) leaked on vgs invocation. Parent PID 38035: grub-install
[ 519.990023] cloud-init[2019]: File descriptor 5 (/dev/sde1) leaked on vgs invocation. Parent PID 38035: grub-install
[ 520.050768] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.075819] cloud-init[2019]: File descriptor 3 (pipe:[82785]) leaked on vgs invocation. Parent PID 38035: grub-install
[ 520.076906] cloud-init[2019]: File descriptor 5 (/dev/sde1) leaked on vgs invocation. Parent PID 38035: grub-install
[ 520.083843] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.210166] cloud-init[2019]: grub-install: error: Can't create file: No such file or directory.
[ 520.210863] cloud-init[2019]: Failed: grub-install --target=powerpc-ieee1275
[ 520.211620] cloud-init[2019]: WARNING: Bootloader is not properly installed, system may not be bootable
[ 520.621281] cloud-init[2019]: Generating grub configuration file ...
[ 520.654974] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.686596] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.706299] cloud-init[2019]: Found linux image: /boot/vmlinux-3.19.0-39-generic
[ 520.721718] cloud-init[2019]: Found initrd image: /boot/initrd.img-3.19.0-39-generic
[ 520.782321] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.810328] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.826707] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.852190] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.868691] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.913768] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.942332] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 520.966640] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 521.118092] cloud-init[2019]: Found linux image: /boot/vmlinux-3.19.0-37-generic
[ 521.123606] cloud-init[2019]: Found initrd image: /boot/initrd.img-3.19.0-37-generic
[ 521.159518] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 521.187518] cloud-init[2019]: Found duplicate PV yrEfpMbescROdKiPs5SBZXCwFIjtPMcN: using /dev/sde2 not /dev/sda2
[ 521.215355] cloud-init[2019]: done
[ 521.2...