dpdk-dev incorrect symlink under $RTE_SDK

[Patrick MacArthur]

$ ls -l /usr/share/dpdk/x86_64-default-linuxapp-gcc/lib/librte_eal.a
ls: cannot access '/usr/share/dpdk/x86_64-default-linuxapp-gcc/lib/librte_eal.a': No such file or directory
$ ls -l /usr/share/dpdk/x86_64-default-linuxapp-gcc/lib/x86_64-linux-gnu/librte_eal.a
-rw-r--r-- 1 root root 294718 Oct 4 03:27 /usr/share/dpdk/x86_64-default-linuxapp-gcc/lib/x86_64-linux-gnu/librte_eal.a

This prevents SPDK from building since it expects the libraries at that location.

I think the fix is just removing the symlink that DPDK installs and replacing it with the correct one.

[Patrick MacArthur (patrickmacarthur)]

(this was found on yakkety)

[Christian Ehrhardt  (paelzer)]

Hi Patrick,
thank you for your report to help making Ubuntu better.

After installing the dev packages via
$ sudo apt-get install dpdk libdpdk-dev dpdk-dev

I can confirm your report - not sure if it is a "bug" yet.
The file(s) are part of libdpdk-dev.
libdpdk-dev:amd64: /usr/lib/x86_64-linux-gnu/librte_eal.a

The symlink you refer to is just a convenience pointing from a dpdk specific dir to the libs following the usual file system layout.
/usr/share/dpdk/x86_64-default-linuxapp-gcc/lib -> ../../../lib/

The RTE paths are usually made available via /usr/share/dpdk/dpdk-sdk-env.sh.
In more recent releases there is also a packageconfig file.

So trying to confirm I understood your request - you mind the extra x86_64-linux-gnu in the path and would want it to change.
#1 Do you want the linkg to change:
/usr/share/dpdk/x86_64-default-linuxapp-gcc/lib -> ../../../lib/
to
/usr/share/dpdk/x86_64-default-linuxapp-gcc/lib -> ../../../lib/x86_64-linux-gnu

#2 or do you want the actual position to change:
File in:
${RTE_SDK}/${RTE_TARGET}/lib/x86_64-linux-gnu/
to be in
${RTE_SDK}/${RTE_TARGET}/lib/

So far we had no issues with it, also not building third party DPDK apps with that layout - e..g some of the dpdk examples.
I really wonder since the following works if we really should change that - can't SPDK work "the same way"?
. /usr/share/dpdk/dpdk-sdk-env.sh
make -C /usr/share/dpdk/examples/l2fwd "O=${ORIGDIR}/l2fwd"

[Patrick MacArthur (patrickmacarthur)]

I apologize that I did not make this more clear.

The library files themselves are in the correct location, according to the Ubuntu multiarch spec. The issue is that the symlink does *not* point to the correct location.

The reason this works for the DPDK examples themselves (including l2fwd) is because they link using e.g. -lrte_eal, with -L${RTE_SDK}/${RTE_TARGET}/lib *added* to the search path. The linker does not find the library there but *does* find it in the default search path, which includes the system library directory /usr/lib/x86_64-linux-gnu.

The SPDK [1] build system includes the following in spdk/lib/env_dpdk/env.mk:

```make
DPDK_LIB = $(DPDK_ABS_DIR)/lib/librte_eal.a $(DPDK_ABS_DIR)/lib/librte_mempool.a \
           $(DPDK_ABS_DIR)/lib/librte_ring.a
```

where DPDK_ABS_DIR is essentially defined to ${RTE_SDK}/${RTE_TARGET}. This means that SPDK does *not* find these libraries in the system search path, since it isn't using -l to look for them there but is hardcoding the specific absolute path where it expects the libraries to be (I presume they do this so that if there are multiple DPDK installations you are absolutely sure which one you are linking against).

Both before and after fixing the symlink, the l2fwd example program builds successfully:

```
$ ls -l /usr/share/dpdk/x86_64-default-linuxapp-gcc/
total 4
drwxr-xr-x 2 root root 4096 Dec 28 14:20 app
lrwxrwxrwx 1 root root 21 Oct 4 03:27 include -> ../../../include/dpdk
lrwxrwxrwx 1 root root 29 Jan 3 13:28 lib -> ../../../lib/x86_64-linux-gnu
$ make -C /usr/share/dpdk/examples/l2fwd O=${PWD}
make: Entering directory '/usr/share/dpdk/examples/l2fwd'
  CC main.o
  LD l2fwd
  INSTALL-APP l2fwd
  INSTALL-MAP l2fwd.map
make: Leaving directory '/usr/share/dpdk/examples/l2fwd'
```

[1]: https://github.com/spdk/spdk

[Christian Ehrhardt  (paelzer)]

Thank you for elaborating.

I was able to verify all that you presented so nicely - that said I'm good changing that in the packaging.

I checked where it comes from, and the TL;DR is that dpdk install-sdk target itself is creating this symlink via:
$(Q)$(call rte_symlink, $(DESTDIR)$(libdir), $(DESTDIR)$(targetdir)/lib)

But it does not know about multiarch spec.
That is only added later when moving the files into packages:
LIBDIR=usr/lib/$(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
[...]
mv debian/dpdk/usr/lib/lib*.a debian/libdpdk-dev/$(LIBDIR)/
(and similar)

The headers don't have an multiarch difference so that is only true for the lib dir.
Everything else works fine with it - juts the lib symlink as you identified is after this move pointing to the wrong subdir.

From package build log (to verify the intended change is actually happening)
Current:
./usr/share/dpdk/x86_64-default-linuxapp-gcc/lib -> ../../../lib
With change:
./usr/share/dpdk/x86_64-default-linuxapp-gcc/lib -> ../../../lib/x86_64-linux-gnu

Since we are working together with other partners on that in the deb_dpdk project I submitted a change as for review and integration there: https://gerrit.fd.io/r/#/c/4548
From there it will tickle down back to Ubuntu.

Since this is not the most common case and easily workaroundable by manually modifying the symlink I'm not sure yet if it would be worth an SRU into existing releases.

Thank you a lot for your report Patrick and your preanalysis what is the issue.
Please let me know if you consider the bug more urgent than I do.

[Christian Ehrhardt  (paelzer)]

FYI status update - fix accepted in deb_dpdk

[Christian Ehrhardt  (paelzer)]

FYI on status - the upload with the fix is complete, yet it is atm blocked to migrate to the archive by a component mismatch.
I have pinged the related MIR request and hope this gets resolved soon.
If not - given some time - I'll upload with the respective tool disabled, so it can migrate.

[Christian Ehrhardt  (paelzer)]

Hi,
I got the issues fixed that prevented it from migrating.
Now fixed in https://launchpad.net/ubuntu/+source/dpdk/16.11-1ubuntu3

The auto-close in the changelog got lost in the back and forth between sync and merge of the package, thereby closing manually now.
Thanks for your report