PCI passthrough: xapi / xcp-fe calls wrong toolstack

[Torsten Kästel]

I'm trying to pass a pci network Card to a domU on ubuntu 13.04 Server (i386) with xen-xapi installed.
The network card is hidden in dom0 and the pci param for the vm is set correctly to the value lspci shows.
The toolstack in /etc/default/xen is set to xapi.

output of lsb_release -rd:
Description: Ubuntu 13.04
Release: 13.04

Output of apt-cache policy xcp-fe:
xcp-fe:
  Installiert: 0.5.2-3build1
  Installationskandidat: 0.5.2-3build1
  Versionstabelle:
 *** 0.5.2-3build1 0
        500 http://de.archive.ubuntu.com/ubuntu/ raring/universe i386 Packages
        100 /var/lib/dpkg/status

After starting the domU the pci Card didn't show up in the vm, lspci in the vm shows no entries. In the syslog of dom0 you can find the following lines:

ay 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.941Z||1115|About to bind to /var/lib/xcp/forker/fd_31f21b3e-a6ad-d8da-bfac-a5996c624a6d
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.941Z||1115|bound, listening
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.942Z||2492|Child here!
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.948Z||2493|Grandchild here!
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.948Z||2493|Started: state.cmdargs = [/usr/lib/xcp/lib/pci-flr;flr-pre;0000:05:04.0]
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.948Z||2493|Started: state.env = [PATH=/sbin:/usr/sbin:/bin:/usr/bin]
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.948Z||2493|Selecting in handle_comms_no_fd_sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Done
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|fd sock
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Selecting in handle_comms_with_fd_sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Done
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|fd sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Received fd named: d29fa306-c067-1b9f-6667-379b89475bd6 - duping to 1 (from 6)
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Selecting in handle_comms_with_fd_sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Done
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|fd sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Received fd named: 5f5ac49f-8978-97b4-c913-285f3462b79d - duping to 2 (from 6)
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Selecting in handle_comms_with_fd_sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|Done
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.949Z||2493|comms sock
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.950Z||2493|Exec
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.950Z||2493|Finished...
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.950Z||2493|Args after replacement = [/usr/lib/xcp/lib/pci-flr;flr-pre;0000:05:04.0]
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.950Z||2493|I've received the following fds: [2;1]#012
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.953Z||2494|Caught unexpected exception: Unix.Unix_error(20, "execve", "/usr/lib/xcp/lib/pci-flr")
May 2 19:10:04 SuperNova xcp-fe: 2494 (/usr/lib/xcp/lib/pci-flr flr-pre 0000:05:04.0) exitted with code 1
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.960Z||1115|About to bind to /var/lib/xcp/forker/fd_9c2d13cf-dc88-0e82-034a-dc35f827d971
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.960Z||1115|bound, listening
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.963Z||2495|Child here!
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.965Z||2496|Grandchild here!
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.965Z||2496|Started: state.cmdargs = [/usr/lib/xcp/lib/pci-flr;flr-post;0000:05:04.0]
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.965Z||2496|Started: state.env = [PATH=/sbin:/usr/sbin:/bin:/usr/bin]
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.965Z||2496|Selecting in handle_comms_no_fd_sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.965Z||2496|Done
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|fd sock
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Selecting in handle_comms_with_fd_sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Done
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|fd sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Received fd named: d46d9d36-7ecf-4b20-cb22-f66f50de8400 - duping to 1 (from 6)
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Selecting in handle_comms_with_fd_sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Done
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|fd sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Received fd named: 22523c95-a5f7-7b4a-c379-a4a76774d87c - duping to 2 (from 6)
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Selecting in handle_comms_with_fd_sock2
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Done
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|comms sock
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Exec
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Finished...
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|Args after replacement = [/usr/lib/xcp/lib/pci-flr;flr-post;0000:05:04.0]
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.966Z||2496|I've received the following fds: [2;1]#012
May 2 19:10:04 SuperNova xcp-fe: 20130502T17:10:04.970Z||2497|Caught unexpected exception: Unix.Unix_error(20, "execve", "/usr/lib/xcp/lib/pci-flr")
May 2 19:10:04 SuperNova xcp-fe: 2497 (/usr/lib/xcp/lib/pci-flr flr-post 0000:05:04.0) exitted with code 1
May 2 19:10:04 SuperNova xapi: [20130502T17:10:04.975Z| info|SuperNova|113 INET 127.0.0.1:80|event.from D:c60730bcb055|audit] ('trackid=89b1b544e95a40d3fa56af5075fc82d4' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'event.from' ())
May 2 19:10:05 SuperNova xcp-fe: 2500 (/usr/sbin/xl pci-attach 1 0000:05:04.0) exitted with code 1

The last line is the one I'm wondering about. Calling xl with Toolstack set to XAPI returns an error, cause /usr/sbin/xl tests if the correct toolset is called. So something seems to be wrong with implementation of xcp-fe.

Here some more Information from apport:
ProblemType: Bug
ApportVersion: 2.9.2-0ubuntu8
Architecture: i386
Date: Thu May 2 21:40:07 2013
Dependencies:
 apt-utils 0.9.7.7ubuntu4
 debconf 1.5.49ubuntu1
 debconf-i18n 1.5.49ubuntu1
 dpkg 1.16.10ubuntu1
 gcc-4.7-base 4.7.3-1ubuntu1
 libapt-inst1.5 0.9.7.7ubuntu4
 libapt-pkg4.12 0.9.7.7ubuntu4
 libbz2-1.0 1.0.6-4
 libc6 2.17-0ubuntu5
 libdb5.1 5.1.29-5ubuntu8
 libgcc1 1:4.7.3-1ubuntu1
 liblocale-gettext-perl 1.05-7build2
 liblzma5 5.1.1alpha+20120614-2ubuntu1
 libselinux1 2.1.9-5ubuntu1
 libstdc++6 4.7.3-1ubuntu1
 libtext-charwidth-perl 0.04-7build2
 libtext-iconv-perl 1.7-5build1
 libtext-wrapi18n-perl 0.06-7
 multiarch-support 2.17-0ubuntu5
 perl-base 5.14.2-21
 tar 1.26+dfsg-5
 zlib1g 1:1.2.7.dfsg-13ubuntu2
DistroRelease: Ubuntu 13.04
InstallationDate: Installed on 2013-04-26 (6 days ago)
InstallationMedia: Ubuntu-Server 13.04 "Raring Ringtail" - Alpha i386 (20130407)
MarkForUpload: True
Package: xcp-fe 0.5.2-3build1
PackageArchitecture: i386
ProcEnviron:
 TERM=xterm
 SHELL=/bin/bash
 PATH=(custom, no user)
 LANG=de_DE.UTF-8
ProcVersionSignature: Ubuntu 3.8.0-19.29-generic 3.8.8
SourcePackage: xen-api-libs
Tags: raring
Uname: Linux 3.8.0-19-generic i686
UpgradeStatus: No upgrade log present (probably fresh install)

Kind regards
Torsten

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1175763/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Danny (danny.rohde)]

Same problem here. I also got the line calling xl:

Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.163Z||2899|About to bind to /var/lib/xcp/forker/fd_ed966256-6964-63a2-a392-0f16fe3911f0
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.164Z||2899|bound, listening
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.165Z||4533|Child here!
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.167Z||4534|Grandchild here!
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.167Z||4534|Started: state.cmdargs = [/usr/lib/xcp/lib/pci-flr;flr-post;0000:0b:00.0]
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.167Z||4534|Started: state.env = [PATH=/sbin:/usr/sbin:/bin:/usr/bin]
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.167Z||4534|Selecting in handle_comms_no_fd_sock2
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Done
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|fd sock
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Selecting in handle_comms_with_fd_sock2
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Done
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|fd sock2
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Received fd named: 4cb241ec-1752-f521-e71f-27884186977c - duping to 1 (from 6)
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Selecting in handle_comms_with_fd_sock2
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Done
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|fd sock2
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Received fd named: 0d1f9468-8479-b29f-e760-de95eae251ff - duping to 2 (from 6)
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Selecting in handle_comms_with_fd_sock2
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Done
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|comms sock
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Exec
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.168Z||4534|Finished...
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.169Z||4534|Args after replacement = [/usr/lib/xcp/lib/pci-flr;flr-post;0000:0b:00.0]
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.169Z||4534|I've received the following fds: [2;1]#012
Jun 15 14:01:29 xen xcp-fe: 20130615T13:01:29.172Z||4535|Caught unexpected exception: Unix.Unix_error(20, "execve", "/usr/lib/xcp/lib/pci-flr")
Jun 15 14:01:29 xen xcp-fe: 4535 (/usr/lib/xcp/lib/pci-flr flr-post 0000:0b:00.0) exitted with code 1
Jun 15 14:01:29 xen xcp-fe: 4538 (/usr/sbin/xl pci-attach 1 0000:0b:00.0) exitted with code 1

[Danny (danny.rohde)]

== ApportVersion =================================
2.0.1-0ubuntu17.3

== Architecture =================================
amd64

== Date =================================
Sat Jun 15 23:26:50 2013

== Dependencies =================================
coreutils 8.13-3ubuntu3.2
debconf 1.5.42ubuntu1
dpkg 1.16.1.2ubuntu7.1
gcc-4.6-base 4.6.3-1ubuntu5
libacl1 2.2.51-5ubuntu1
libattr1 1:2.4.46-5ubuntu1
libbz2-1.0 1.0.6-1
libc-bin 2.15-0ubuntu10.4
libc6 2.15-0ubuntu10.4
libgcc1 1:4.6.3-1ubuntu5
liblzma5 5.1.1alpha+20110809-3
libselinux1 2.1.0-4.1ubuntu1
multiarch-support 2.15-0ubuntu10.4
perl-base 5.14.2-6ubuntu2.3
tar 1.26-4ubuntu1
tzdata 2012e-0ubuntu0.12.04.1
xz-utils 5.1.1alpha+20110809-3
zlib1g 1:1.2.3.4.dfsg-3ubuntu4

== DistroRelease =================================
Ubuntu 12.04

== MarkForUpload =================================
True

== NonfreeKernelModules =================================
zfs zcommon znvpair zavl zunicode

== Package =================================
xcp-fe 0.5.2-3

== PackageArchitecture =================================
amd64

== ProblemType =================================
Bug

== ProcEnviron =================================
LANGUAGE=en_US:en
TERM=screen
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash

== ProcVersionSignature =================================
Ubuntu 3.2.0-48.74-generic 3.2.46

== SourcePackage =================================
xen-api-libs

== Tags =================================
 precise

== Uname =================================
Linux 3.2.0-48-generic x86_64

== UpgradeStatus =================================
No upgrade log present (probably fresh install)