PPC64 version of CirrOS

[Arx Cruz]

This is not exactly a bug report, but I didn't find any other place to report this:

Are you guys planning to have a CirrOS image build to PPC64? If so, when?

[Scott Moser (smoser)]

I've not looked at it at all, but if buildroot supports ppc64, then we can try it.

[Arx Cruz (arxcruz)]

I can confirm that Buildroot have support, at least the latest versions.

[Rafael Folco (rafaelfolco)]

Hello Scott.
We have built CirrOS on ppc64 but we're struggling with /dev/vda which is not being mount on /.

Basically the high level steps were:
1) Increase /run partition size to 2mb (800kb would be enough) on src/etc/fstab
2) Add virtio support to src/etc/modules: virtio_net, virtio_blk, and virtio_balloon
3) Small hacks to the bundle and prepare-grub scripts (rpm2cpio)
4) Download buildroot: http://buildroot.uclibc.org/downloads/buildroot-2013.11.tar.gz
5) Create a symbolic link: buildroot --> buildroot-2013.11
6) make menuconfig
target options -> target architecture -> powerpc
Toolchain -> C Library -> uClib
Toolchain -> uClib C library version -> a última versão que tiver
Toolchain -> Enable large files
Toolchain -> enable ipv6 support
Toolchain -> enable RPC support
Toolchain -> Enable WCHAR support
Toolchain -> Compile and install uClibc utilities
Toolchain -> Enable compiler tls support
System Configuration -> remount root filesystem read-write during boot
Target Packages -> Networking applications -> dropbear
Target Packages -> Networking applications -> dnsmasq
Target Packages -> Networking applications -> iputils
Target Packages -> Networking applications -> iptables
*do not enable openssh
Target Packages -> Shell and utilities -> sudo
Target Packages -> Shell and utilities -> screen
7) make ARCH=powerpc OUT_D=$PWD/output/powerpc
8) ./bin/bundle -v output/powerpc/rootfs.tar download/kernel-3.11.10-200.2.fc19.ppc64p7.rpm output/powerpc/images

x86:
$ df -h
Filesystem Size Used Available Use% Mounted on
/dev 998.2M 0 998.2M 0% /dev
/dev/vda 193.7M 9.9M 173.8M 5% /
tmpfs 1001.8M 0 1001.8M 0% /dev/shm
tmpfs 200.0K 96.0K 104.0K 48% /run

ppc64:
$ df -h
Filesystem Size Used Available Use% Mounted on
/dev 905.6M 0 905.6M 0% /dev
tmpfs 1007.8M 0 1007.8M 0% /dev/shm
tmpfs 2.0M 1.3M 704.0K 66% /run

I'll take a look at this issue.

Please let me know if you see any problems with the steps I described above.
Thanks.

[Rafael Folco (rafaelfolco)]

More differences to identify the root cause:

x86_64:
EXT3-fs (vda): mounted filesystem with ordered data mode
ppc64:
EXT4-fs (vda): mounting ext3 file system using the ext4 subsystem

x86_64:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 20G 0 disk /
sr0 11:0 1 410K 0 rom

ppc64:
N/A

[Scott Moser (smoser)]

believe that this is fix-committed in trunk.
we now have
  powerpc (32 bit userspace, 64bit kernel)
  ppc64: 64 bit big endian
  ppc64le: 64 bit little endian

I'd appreciate feedback on a daily biuld at http://download.cirros-cloud.net/daily/

[Rafael Folco (rafaelfolco)]

- powerpc: network works only if I change driver to ibmveth (spapr-vlan)
- ppc64le: kernel panic
Haven't tried ppc64 version yet.

separate bug report for each ?

[Rafael Folco (rafaelfolco)]

ppc64 also producing kernel panic, similar to LE:
http://download.cirros-cloud.net/daily/20150604/

# nova console-log test

SLOF[0m[?25l **********************************************************************
[1mQEMU Starting
[0m Build Date = Jul 4 2014 19:52:11
 FW Version = mockbuild@ release 20140630
 Press "s" to enter Open Firmware.

[0m[?25hC0000C0100C0120C0140C0200C0201C0220C0240C0260C0270C02E0C0300C0320C0340C0360C0370C0380C0371C0372C0373C0374C0390C03F0C0400C0480C04C0C04D0C0500Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/vty@30001000
Populating /vdevice/nvram@71000000
C0580C05A0Populating /pci@800000020000000

[root@devstack-fedora-1433346199 files]# nova console-log test

SLOF[0m[?25l **********************************************************************
[1mQEMU Starting
[0m Build Date = Jul 4 2014 19:52:11
 FW Version = mockbuild@ release 20140630
 Press "s" to enter Open Firmware.

[0m[?25hC0000C0100C0120C0140C0200C0201C0220C0240C0260C0270C02E0C0300C0320C0340C0360C0370C0380C0371C0372C0373C0374C0390C03F0C0400C0480C04C0C04D0C0500Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/vty@30001000
Populating /vdevice/nvram@71000000
C0580C05A0Populating /pci@800000020000000
 Adapters on 0800000020000000
                     00 0800 (D) : 1af4 1000 virtio [ net ]
                     00 1000 (D) : 106b 003f serial bus [ usb-ohci ]
                     00 1800 (D) : 1af4 1001 virtio [ block ]
                     00 2000 (D) : 1af4 1002 unknown-legacy-device*
C0600C0640C0690C06A0C06A8C06B0C06B8C06C0C06E0C0700C0800C0880No NVRAM common partition, re-initializing...
C0890C08A0C08A8C08B0Scanning USB
  OHCI: initializing
C08C0C08D0Using default console: /vdevice/vty@30000000
C08E0C08E8Detected RAM kernel at 400000 (1a62994 bytes) C08FF
  Welcome to Open Firmware

  Copyright (c) 2004, 2011 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php

Booting from memory...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.19.0-20-powerpc64-smp (buildd@sagari) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #20~14.04.1-Ubuntu SMP Sat May 30 00:30:20 UTC 2015 (Ubuntu 3.19.0-20.20~14.04.1-powerpc64-smp 3.19.8)
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 1024 (NR_CPUS = 1024)
Calling ibm,client-architecture-support... done
command line: root=/dev/vda console=tty0 console=ttyS0
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000001e67000
  alloc_top : 0000000010000000
  alloc_top_hi : 0000000040000000
  rmo_top : 0000000010000000
  ram_top : 0000000040000000
instantiating rtas at 0x000000000ffff000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000002768000 -> 0x00000000027687f5
Device tree struct 0x0000000002769000 -> 0x000000000276c000
Calling quiesce...
returning from prom_init
[ 0.000000] Using pSeries machine descr...

[Scott Moser (smoser)]

Hi, heres what I tested just now as functional for the power builds:

$ for arch in powerpc ppc64 ppc64le;
  do burl=http://download.cirros-cloud.net/daily/20150604/cirros-d150604-$arch;
 for f in kernel initramfs; do [ -f "${burl##*/}-$f" ] || wget $burl-$f; done; done

$ ls -l cirros-d150604-p*
-rw-rw-r-- 1 ubuntu ubuntu 4362299 Jun 4 04:53 cirros-d150604-powerpc-initramfs
-rw-rw-r-- 1 ubuntu ubuntu 25280072 Jun 4 04:53 cirros-d150604-powerpc-kernel
-rw-rw-r-- 1 ubuntu ubuntu 8702396 Jun 4 04:54 cirros-d150604-ppc64-initramfs
-rw-rw-r-- 1 ubuntu ubuntu 25280072 Jun 4 04:54 cirros-d150604-ppc64-kernel
-rw-rw-r-- 1 ubuntu ubuntu 8461981 Jun 4 04:54 cirros-d150604-ppc64le-initramfs
-rw-rw-r-- 1 ubuntu ubuntu 21761512 Jun 4 04:54 cirros-d150604-ppc64le-kernel

On the Ubuntu host, I did:
  sudo apt-get install -qy qemu-system-ppc
  sudo ppc64_cpu --smt=off
  sudo modprobe kvm_hv
  sudo modprobe kvm_pv
  sudo chmod 666 /dev/kvm

Then for each of the kernel/initramfs pairs, i ran 'go' (see below).

$ ./go cirros-d150604-ppc64le-kernel cirros-d150604-ppc64le-initramfs

For each kernel, it booted to finding the root device, the seed image, copied itself
over to the disk image and brought up networking via dhcp.

$ cat go
#!/bin/sh
kernel="$1"
initrd="$2"
shift 2
cmdline="$1"
disk="disk.img"
fail() { echo "$@" 1>&2; exit 1; }

rm -f "$disk"
qemu-img create -f raw "$disk" 1G || fail
out=$(mkfs.ext2 -F -L cirros-rootfs "$disk" >/dev/null 2>&1) ||
   fail "failed mkfs $disk: $out"

if [ ! -f seed.img ]; then
    echo '{"instance-id": "9068aef2-213e-4e43-830f-accdbadde897"}' > meta-data
    { echo '#!/bin/sh'; echo 'poweroff'; } > user-data
    cloud-localds seed.img user-data meta-data
fi

qemu-system-ppc64 -enable-kvm -machine pseries,usb=off -device spapr-vscsi \
   -device spapr-vlan,netdev=net00 -netdev type=user,id=net00 \
   -drive if=virtio,file=$disk -drive if=virtio,file=seed.img \
   -m 1G -echr 0x05 -nographic -vga none \
   -kernel "$kernel" -initrd "$initrd" -append "$cmdline"

[Scott Moser (smoser)]

It also seems that using
  -device virtio-net-pci,netdev=net00
for my network drivers at least seems to work.

[Rafael Folco (rafaelfolco)]

Ok, apparently my kernel has issues. I believe I am hitting a bug related to HTM. I've tested the images and confirm that ppc64, powerpc and ppc64le versions work well in 1st level for me. So the images seem to be good.

[Rafael Folco (rafaelfolco)]

Upgraded kernel in 1st level guest and resolved my HTM issues.

Results running in OpenStack as nested VMs:
- powerpc (32-bit): works as long as you change hw_vif_model from virtio to spapr-vlan (ibmveth)
- ppc64 (64-bit): works good
- ppc64le: works but produces a call trace while booting. See below.

ppc64le boot call trace:
[ 1.776496] irq 18: nobody cared (try booting with the "irqpoll" option)
[ 1.776501] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.19.0-20-generic #20~14.04.1-Ubuntu
[ 1.776501] Call Trace:
[ 1.776507] [c00000000fffbcb0] [c000000000a26030] dump_stack+0x90/0xbc (unreliable)
[ 1.776510] [c00000000fffbce0] [c00000000012a670] __report_bad_irq+0x60/0x150
[ 1.776512] [c00000000fffbd70] [c00000000012aeb0] note_interrupt+0x340/0x3b0
[ 1.776513] [c00000000fffbe10] [c000000000127124] handle_irq_event_percpu+0x124/0x2b0
[ 1.776514] [c00000000fffbed0] [c000000000127318] handle_irq_event+0x68/0xd0
[ 1.776516] [c00000000fffbf00] [c00000000012b7b4] handle_fasteoi_irq+0xe4/0x240
[ 1.776517] [c00000000fffbf30] [c000000000126288] generic_handle_irq+0x58/0x90
[ 1.776519] [c00000000fffbf60] [c000000000010f10] __do_irq+0x80/0x190
[ 1.776521] [c00000000fffbf90] [c00000000002476c] call_do_irq+0x14/0x24
[ 1.776522] [c00000002e642af0] [c0000000000110c0] do_IRQ+0xa0/0x120
[ 1.776524] [c00000002e642b50] [c0000000000099cc] restore_check_irq_replay+0x2c/0x70
[ 1.776526] --- interrupt: 501 at arch_local_irq_restore+0x74/0x90
[ 1.776526] LR = arch_local_irq_restore+0x74/0x90
[ 1.776527] [c00000002e642e40] [c00000002e630000] 0xc00000002e630000 (unreliable)
[ 1.776529] [c00000002e642e60] [c0000000000b4e8c] __do_softirq+0xfc/0x3b0
[ 1.776530] [c00000002e642f60] [c0000000000b54a8] irq_exit+0xc8/0x100
[ 1.776531] [c00000002e642f80] [c00000000001fa54] timer_interrupt+0xa4/0xe0
[ 1.776533] [c00000002e642fb0] [c0000000000099f4] restore_check_irq_replay+0x54/0x70
[ 1.776534] --- interrupt: 901 at arch_local_irq_restore+0x74/0x90
[ 1.776534] LR = arch_local_irq_restore+0x74/0x90
[ 1.776537] [c00000002e6432a0] [c0000000016027b8] logbuf_lock+0x0/0x8 (unreliable)
[ 1.776538] [c00000002e6432c0] [c000000000124a48] console_unlock+0x578/0x5c0
[ 1.776540] [c00000002e643390] [c000000000124d94] vprintk_emit+0x304/0x5c0
[ 1.776542] [c00000002e643410] [c00000000064556c] dev_vprintk_emit+0x10c/0x240
[ 1.776543] [c00000002e643530] [c0000000006456f0] dev_printk_emit+0x50/0x60
[ 1.776544] [c00000002e643570] [c00000000064577c] __dev_printk+0x7c/0xe0
[ 1.776545] [c00000002e6435f0] [c000000000645cdc] _dev_info+0x6c/0x90
[ 1.776547] [c00000002e643640] [c00000000076e0f8] usb_new_device+0x178/0x630
[ 1.776549] [c00000002e6436f0] [c000000000774c94] usb_add_hcd+0x804/0xc00
[ 1.776550] [c00000002e6437b0] [c00000000078dfcc] usb_hcd_pci_probe+0x1dc/0x5c0
[ 1.776553] [c00000002e643850] [c00000000057270c] local_pci_probe+0x6c/0x140
[ 1.776554] [c00000002e6438e0] [c000000000572938] pci_device_probe+0x158/0x1e0
[ 1.776556] [c00000002e643940] [c00000000064b3ec] driver_probe_device+0xec/0x470
[ 1.776557] [c00000002e6439d0] [c00000000064b92c] __driver_attach+0x11c/0x120
[ 1.7...

[Rafael Folco (rafaelfolco)]

smoser,
Any plans to add ppc64 to 0.3.4 download dir ?
It's working fine from what I tested.
Thanks.

[Rafael Folco (rafaelfolco)]

@smoser
We're using ppc64le cirros for months.
http://download.cirros-cloud.net/daily/20150923/cirros-d150923-ppc64le-uec.tar.gz

A recent log:
https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/54/277554/7/check/check-ibm-tempest-dsvm-full/928f767/

I confirmed that it works fine with virtio-blk, virtio-scsi, spapr-vscsi, virtio-net and spapr-vlan.

Would you consider including ppc64le cirros to the latest Cirros version dir?
http://download.cirros-cloud.net/0.3.4/

Thanks in advance.
--rfolco