Kernel BUG @of/base.c:251 during attach-device operation on Ubuntu 14.10 LE guest

[bugproxy]

-- Problem Description --
Trying to dynamically add a network device of type virtio to a Ubuntu 14.10 guest running on PowerKVM generates a kernel BUG message. Virsh reports the operation as successful.

Steps to recreate the problem :

1) Install 14.10 as a Power KVM guest.
2) Install ppc64-diag package

librtas1 1.3.10-1 ppc64el userspace RTAS library
librtasevent1 1.3.10-1 ppc64el RTAS events library
ppc64-diag 2.6.6-0ubuntu1 ppc64el Platform error log analysis tool and rtas_errd daemon
libservicelog-1.1-1:ppc64el 1.1.14-0ubuntu1 ppc64el Library to create and maintain a database for storing events
servicelog 1.1.12-0ubuntu1 ppc64el Platform Service/Error Message Database

3) Using a network device definition as mentioned below try a virsh attach-device operation

On the PowerKVM host :

# cat net
<interface type='bridge'>
   <mac address='52:54:00:A0:9D:9a'/>
   <source bridge='virbr0'/>
   <model type='virtio'/>
</interface>

# virsh attach-device ss_1410_1 net
Device attached successfully

#

Following kernel BUG is observed on the guest

[ 6380.112079] RTAS: event: 1, Type: Unknown, Severity: 1
[ 6380.209844] ------------[ cut here ]------------
[ 6380.209848] kernel BUG at /build/buildd/linux-3.15.0/drivers/of/base.c:251!
[ 6380.209922] Oops: Exception in kernel mode, sig: 5 [#1]
[ 6380.209963] SMP NR_CPUS=2048 NUMA pSeries
[ 6380.210006] Modules linked in: pseries_rng rtc_generic e1000
[ 6380.210084] CPU: 0 PID: 3642 Comm: drmgr Not tainted 3.15.0-6-generic #11-Ubuntu
[ 6380.210146] task: c0000000770968a0 ti: c000000079858000 task.ti: c000000079858000
[ 6380.210207] NIP: c000000000871c24 LR: c000000000872704 CTR: c000000000080ed0
[ 6380.210268] REGS: c00000007985b7f0 TRAP: 0700 Not tainted (3.15.0-6-generic)
[ 6380.210328] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 24002488 XER: 20000000
[ 6380.210473] CFAR: c000000000872700 SOFTE: 1
[ 6380.210473] GPR00: c000000000872704 c00000007985ba70 c000000001791188 c000000004192100
[ 6380.210473] GPR04: 0000000000000001 c00000007e039fc0 c00000000419ee80 c000000001aa1188
[ 6380.210473] GPR08: 0000000000000002 0000000000000001 0000000000000000 c00000000453025b
[ 6380.210473] GPR12: 0000000024002482 c00000000fe40000 0000000000000000 000000000000000c
[ 6380.210473] GPR16: 000001001e42eeda 0000000000000020 0000000000000000 0000000010016a00
[ 6380.210473] GPR20: 0000000010016950 00000000100167b0 0000000000000001 c00000007e039800
[ 6380.210473] GPR24: c000000000cc0310 c000000004530009 c000000077594f80 c00000007fff6d38
[ 6380.210473] GPR28: 0000000000000015 c000000004192100 c000000001aa01b0 c000000004192100
[ 6380.211355] NIP [c000000000871c24] .of_node_add+0x34/0x110
[ 6380.211397] LR [c000000000872704] .of_attach_node+0xd4/0xf0
[ 6380.211440] Call Trace:
[ 6380.211462] [c00000007985ba70] [0000000010016950] 0x10016950 (unreliable)
[ 6380.211525] [c00000007985baf0] [c000000000872704] .of_attach_node+0xd4/0xf0
[ 6380.211633] [c00000007985bb70] [c000000000076420] .ofdt_write+0x590/0x780
[ 6380.211753] [c00000007985bc50] [c00000000030727c] .proc_reg_write+0xac/0x110
[ 6380.211815] [c00000007985bcf0] [c000000000279370] .vfs_write+0xe0/0x260
[ 6380.211876] [c00000007985bd90] [c00000000027a034] .SyS_write+0x64/0x110
[ 6380.211939] [c00000007985be30] [c000000000009ffc] syscall_exit+0x0/0x7c
[ 6380.212001] Instruction dump:
[ 6380.212033] fbc1fff0 fbe1fff8 f8010010 f821ff81 7c7f1b78 60000000 60000000 2fbf0000
[ 6380.212142] 419e006c 813f0094 69290001 792907e0 <0b090000> 3fc2ffeb 3bdefe60 7fc3f378
[ 6380.212250] ---[ end trace 410b60b95ac50ef5 ]---
[ 6380.214151]
sachin@1410:/var/log$

sachin@1410:/var/log$ cat /etc/issue
Ubuntu Utopic Unicorn (development branch) \n \l

sachin@1410:/var/log$ uname -a
Linux 1410 3.15.0-6-generic #11-Ubuntu SMP Thu Jun 12 00:40:49 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux
sachin@1410:/var/log$

The BUG_ON() it is hitting is likely this one:

int of_node_add(struct device_node *np)
{
 int rc = 0;

 BUG_ON(!of_node_is_initialized(np));

This appears to be related to commit 75b57ecf9d1d1e17d099ab13b8f48e6e038676be.

This was part of the updates that Grant did to move the device tree from /proc/to sysfs. As part of this he removed the kref_init call we made prior to adding the device node.

Attached patch "powerpc/pseries: dynamically added OF nodes need to call of_node_init"

With the attach patch, the attach/detach device operation works correctly. No kernel BUG is observed.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1340789

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Tim Gardner (timg-tpi)]

commit 97a9a7179aad701ab676e6f29eb90766a1acfde2 (powerpc/pseries: dynamically added OF nodes need to call of_node_init) will be in 3.16-rc7

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-08-26 10:51 EDT-------
Works correctly with latest daily build ISO.

sachin@ubuntu:~$ uname -a
Linux ubuntu 3.16.0-10-generic #15-Ubuntu SMP Thu Aug 21 16:32:31 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux
sachin@ubuntu:~$

sachin@ubuntu:~$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 52:54:c0:99:80:76
inet addr:192.168.122.54 Bcast:192.168.122.255 Mask:255.255.255.0
inet6 addr: fe80::5054:c0ff:fe99:8076/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:975 errors:0 dropped:8 overruns:0 frame:0
TX packets:1618 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:602614 (602.6 KB) TX bytes:1536686 (1.5 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1184 (1.1 KB) TX bytes:1184 (1.1 KB)

sachin@ubuntu:~$

Run the following command on the host :

[root@host sachin]#
virsh attach-device ss_1410 net
Device attached successfully

[root@host sachin]#

sachin@ubuntu:~$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 52:54:c0:99:80:76
inet addr:192.168.122.54 Bcast:192.168.122.255 Mask:255.255.255.0
inet6 addr: fe80::5054:c0ff:fe99:8076/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:975 errors:0 dropped:8 overruns:0 frame:0
TX packets:1618 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:602614 (602.6 KB) TX bytes:1536686 (1.5 MB)

eth1 Link encap:Ethernet HWaddr 52:54:00:a0:9d:9a
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1184 (1.1 KB) TX bytes:1184 (1.1 KB)

sachin@ubuntu:~$ dmesg
[106570.803617] RTAS: event: 1, Type: Unknown, Severity: 1
[106570.906152] pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000
[106570.909758] pci 0000:00:05.0: reg 0x10: [io 0x100a0-0x100bf]
[106570.913095] pci 0000:00:05.0: reg 0x14: [mem 0x100a0001000-0x100a0001fff]
[106570.929847] pci 0000:00:05.0: reg 0x30: [mem 0x100a0040000-0x100a007ffff pref]
[106570.930298] pci 0000:00:05.0: BAR 6: assigned [mem 0x100a0000000-0x100a003ffff pref]
[106570.943212] pci 0000:00:05.0: BAR 1: assigned [mem 0x100a0040000-0x100a0040fff]
[106570.944066] pci 0000:00:05.0: BAR 0: assigned [io 0x10000-0x1001f]
sachin@ubuntu:~$