dpdk ip_pipeline app unable to dlopen target pmd lib .so

[bugproxy]

== Comment: #0 - Gowrishankar Muthukrishnan <email address hidden> - 2016-09-30 04:50:25 ==
---Problem Description---

Sample application 'ip_pipeline' crashes with rte_panic as below due to dlopen()
failing in rte_eal_init.

$ /usr/share/dpdk/examples/ip_pipeline/build/ip_pipeline -f /usr/share/dpdk/examples/ip_pipeline/config/l3fwd.cfg -p 0x3
[APP] Initializing CPU core map ...
[APP] CPU core mask = 0x0000000000000011
[APP] Initializing EAL ...
Aborted

More backtrace/verbose info with gdb (after also adding -l 2 after -p 0x3):

(gdb)
Starting program: /usr/share/dpdk/examples/ip_pipeline/build/ip_pipeline -f /usr/share/dpdk/examples/ip_pipeline/config/l3fwd.cfg -p 0x3 -l 2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
[APP] Initializing CPU core map ...
Socket 0:
[0] = [ 0 (0) 1 (0) 2 (0) 3 (0) ]
[1] = [ 4 (0) 5 (0) 6 (0) 7 (0) ]
[2] = [ 8 (8) 9 (8) 10 (8) 11 (8) ]
[3] = [ 12 (8) 13 (8) 14 (8) 15 (8) ]
Socket 1:
[0] = [ 16 (16) 17 (16) 18 (16) 19 (16) ]
[1] = [ 20 (16) 21 (16) 22 (16) 23 (16) ]
[2] = [ 24 (24) 25 (24) 26 (24) 27 (24) ]
[3] = [ 28 (24) 29 (24) 30 (24) 31 (24) ]
[APP] CPU core mask = 0x0000000000000011
[APP] Initializing EAL ...
[APP] EAL arguments: "-c11 -n4 --log-level=2 -d /usr/lib/powerpc64le-linux-gnu/librte_pmd_virtio.so --socket-mem=1024 -- "

Program received signal SIGABRT, Aborted.
0x00003fffb7a0e640 in __libc_signal_restore_set (set=0x3fffffffe8c0)
    at ../sysdeps/unix/sysv/linux/nptl-signals.h:79
79 ../sysdeps/unix/sysv/linux/nptl-signals.h: No such file or directory.
(gdb) bt
#0 0x00003fffb7a0e640 in __libc_signal_restore_set (set=0x3fffffffe8c0)
    at ../sysdeps/unix/sysv/linux/nptl-signals.h:79
#1 __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:55
#2 0x00003fffb7a10c70 in __GI_abort () at abort.c:89
#3 0x00003fffb7c557e8 in __rte_panic (funcname=<optimized out>,
    format=0x3fffb7c6cd00 "Cannot init plugins\n%.0s") at ./lib/librte_eal/linuxapp/eal/eal_debug.c:86
#4 0x00003fffb7c57218 in rte_eal_init (argc=<optimized out>, argv=<optimized out>)
    at ./lib/librte_eal/linuxapp/eal/eal.c:847
#5 0x0000000020022a10 in app_init_eal (app=0x204d2100 <app>) at /usr/share/dpdk/examples/ip_pipeline/init.c:305
#6 0x0000000020027314 in app_init (app=0x204d2100 <app>) at /usr/share/dpdk/examples/ip_pipeline/init.c:1750
#7 0x0000000020008818 in main (argc=7, argv=0x3ffffffff4c8) at /usr/share/dpdk/examples/ip_pipeline/main.c:55
(gdb)

Interestingly, most of other apps do not find dlopen failing, as they directly
call rte_eal_init in their main. ip_pipeline do one step extra of wrapping up
it due to its configuration file aid to run this app. Its app_init_eal creates
actual EAL params by parsing config file given.

I already posted a minor typo patch which is merged in deb_dpdk, so please ensure
deb you test carries it:
http://dpdk.org/dev/patchwork/patch/15995

---uname output---
Linux 9-114-219-126-ukvm1 4.4.0-9136-generic #55-Ubuntu SMP Fri Aug 26 05:56:24 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Machine Type = P8 LE

---Debugger---
A debugger is not configured

---Steps to Reproduce---
 (all as root)
1. install dpdk-doc
2. cd /usr/share/dpdk
   source dpdk-sdk-env.sh
3. cd examples/ip_pipeline
4. if init.c does not carry above patchwork fix, apply manually.
5. make
6. do below if igb_uio to be tested, else ensure your target pmd loaded.
   modprobe igb_uio
   dpdk-devbind --bind=igb_uio <PCI ID 1>
   dpdk-devbind --bind=igb_uio <PCI ID 2>
   check testpmd probes the above ports
7. Adjust config/l3fwd.cfg for only two pmd ports, as below:
 [EAL]
 log_level = 2
 socket_mem = 1024
 d = /usr/lib/powerpc64le-linux-gnu/librte_pmd_virtio.so

 [PIPELINE0]
 type = MASTER
 core = 0

 [PIPELINE1]
 type = ROUTING
 core = 1
 pktq_in = RXQ0.0 RXQ1.0
 pktq_out = TXQ0.0 TXQ1.0 SINK0
 encap = ethernet
 ;encap = ethernet_qinq
 ;encap = ethernet_mpls
 ip_hdr_offset = 270

8. build/ip_pipeline -f config/l3fwd.cfg -p 0x3
   ip_pipeline should not panic here.

Userspace tool common name: dpdk

The userspace tool has the following bit modes: both

Userspace rpm: dpdk-doc

Userspace tool obtained from project website: na

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-04 07:08 EDT-------
Modified and tested patch wrt dlopen is at:
http://dpdk.org/dev/patchwork/patch/16363/

With above, rte_eal_init is no more hitting issue on loading driver.

[Christian Ehrhardt  (paelzer)]

Thanks,
patch already included and uploaded along more fixes.
Given that it passes all tests it should be available soon.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpdk - 16.07-0ubuntu5

---------------
dpdk (16.07-0ubuntu5) yakkety; urgency=medium

  [ Christian Ehrhardt ]
  * Fix component mismatch by dropping the optional dpdk-pmdinfo tool
    (LP: #1630073).

  [ Gowrishankar Muthukrishnan ]
  * update d/p/dpdk-dev-examples-ip_pipeline-fix-pmd-driver-parameter.patch to
    fix dlopen issue (LP: #1630119)

 -- Christian Ehrhardt <email address hidden> Tue, 04 Oct 2016 09:27:54 +0200

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-19 14:06 EDT-------
Verified with dpdk 16.07-1~git3 in ppc64el. ip_pipeline is able to load PMD mentioned in config file.

Thanks.

------- Comment From <email address hidden> 2016-10-19 14:08 EDT-------
Fix also available in dpdk upstream:
http://dpdk.org/dev/patchwork/patch/16363/