Marvell 88SE9128 SATA controller problem with Xen hypervisor

[Heiko Sieger]

1. Linux Mint 13 Maya 64bit and xen hypervisor 4.1.2
3.2.0-27-generic kernel

Hardware:
Asus Sabertooth X79 motherboard, which includes an onboard Marvell 88SE9128
SATA controller
i7 3930K with VT-d enabled
32GB RAM

2. No issues booting into LM13 without Xen hypervisor! ~15-20sec boot time from SSD. Installed Xen hypervisor 4.1.2 from package manager and did some basic configuration.

3. When booting the xen-4.1.2 I get a long boot delay of around 1.5-2 min.
/var/log/xen/xend-debug.log reports errors (see below), also dmesg.

Booting the regular LM13 (not hypervisor) installation boots fine (15-20 sec.).

After disabling the Marvell 88SE9128 SATA controller in BIOS, the system boots normal. Currently there is no drive connected to the Marvell controller.

4. I expect to have support for the marvell SATA controller under Xen, same as under LM13 regular install. See just below under "Remark:" for hints on what may cause the Xen boot not to recognize the marvell controller !

Here the /var/log/xen/xend-debug.log output:

cat: /sys/bus/scsi/devices/host0/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host0/model: No such file or directory
cat: /sys/bus/scsi/devices/host0/type: No such file or directory
cat: /sys/bus/scsi/devices/host0/rev: No such file or directory
cat: /sys/bus/scsi/devices/host0/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host1/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host1/model: No such file or directory
cat: /sys/bus/scsi/devices/host1/type: No such file or directory
cat: /sys/bus/scsi/devices/host1/rev: No such file or directory
cat: /sys/bus/scsi/devices/host1/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host2/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host2/model: No such file or directory
cat: /sys/bus/scsi/devices/host2/type: No such file or directory
cat: /sys/bus/scsi/devices/host2/rev: No such file or directory
cat: /sys/bus/scsi/devices/host2/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host3/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host3/model: No such file or directory
cat: /sys/bus/scsi/devices/host3/type: No such file or directory
cat: /sys/bus/scsi/devices/host3/rev: No such file or directory
cat: /sys/bus/scsi/devices/host3/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host4/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host4/model: No such file or directory
cat: /sys/bus/scsi/devices/host4/type: No such file or directory
cat: /sys/bus/scsi/devices/host4/rev: No such file or directory
cat: /sys/bus/scsi/devices/host4/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host5/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host5/model: No such file or directory
cat: /sys/bus/scsi/devices/host5/type: No such file or directory
cat: /sys/bus/scsi/devices/host5/rev: No such file or directory
cat: /sys/bus/scsi/devices/host5/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host6/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host6/model: No such file or directory
cat: /sys/bus/scsi/devices/host6/type: No such file or directory
cat: /sys/bus/scsi/devices/host6/rev: No such file or directory
cat: /sys/bus/scsi/devices/host6/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host7/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host7/model: No such file or directory
cat: /sys/bus/scsi/devices/host7/type: No such file or directory
cat: /sys/bus/scsi/devices/host7/rev: No such file or directory
cat: /sys/bus/scsi/devices/host7/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host8/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host8/model: No such file or directory
cat: /sys/bus/scsi/devices/host8/type: No such file or directory
cat: /sys/bus/scsi/devices/host8/rev: No such file or directory
cat: /sys/bus/scsi/devices/host8/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host9/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host9/model: No such file or directory
cat: /sys/bus/scsi/devices/host9/type: No such file or directory
cat: /sys/bus/scsi/devices/host9/rev: No such file or directory
cat: /sys/bus/scsi/devices/host9/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host10/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host10/model: No such file or directory
cat: /sys/bus/scsi/devices/host10/type: No such file or directory
cat: /sys/bus/scsi/devices/host10/rev: No such file or directory
cat: /sys/bus/scsi/devices/host10/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host11/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host11/model: No such file or directory
cat: /sys/bus/scsi/devices/host11/type: No such file or directory
cat: /sys/bus/scsi/devices/host11/rev: No such file or directory
cat: /sys/bus/scsi/devices/host11/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host12/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host12/model: No such file or directory
cat: /sys/bus/scsi/devices/host12/type: No such file or directory
cat: /sys/bus/scsi/devices/host12/rev: No such file or directory
cat: /sys/bus/scsi/devices/host12/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host13/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host13/model: No such file or directory
cat: /sys/bus/scsi/devices/host13/type: No such file or directory
cat: /sys/bus/scsi/devices/host13/rev: No such file or directory
cat: /sys/bus/scsi/devices/host13/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host14/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host14/model: No such file or directory
cat: /sys/bus/scsi/devices/host14/type: No such file or directory
cat: /sys/bus/scsi/devices/host14/rev: No such file or directory
cat: /sys/bus/scsi/devices/host14/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/host15/vendor: No such file or directory
cat: /sys/bus/scsi/devices/host15/model: No such file or directory
cat: /sys/bus/scsi/devices/host15/type: No such file or directory
cat: /sys/bus/scsi/devices/host15/rev: No such file or directory
cat: /sys/bus/scsi/devices/host15/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/target0:0:0/vendor: No such file or directory
cat: /sys/bus/scsi/devices/target0:0:0/model: No such file or directory
cat: /sys/bus/scsi/devices/target0:0:0/type: No such file or directory
cat: /sys/bus/scsi/devices/target0:0:0/rev: No such file or directory
cat: /sys/bus/scsi/devices/target0:0:0/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/target1:0:0/vendor: No such file or directory
cat: /sys/bus/scsi/devices/target1:0:0/model: No such file or directory
cat: /sys/bus/scsi/devices/target1:0:0/type: No such file or directory
cat: /sys/bus/scsi/devices/target1:0:0/rev: No such file or directory
cat: /sys/bus/scsi/devices/target1:0:0/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/target2:0:0/vendor: No such file or directory
cat: /sys/bus/scsi/devices/target2:0:0/model: No such file or directory
cat: /sys/bus/scsi/devices/target2:0:0/type: No such file or directory
cat: /sys/bus/scsi/devices/target2:0:0/rev: No such file or directory
cat: /sys/bus/scsi/devices/target2:0:0/scsi_level: No such file or directory
cat: /sys/bus/scsi/devices/target5:0:0/vendor: No such file or directory
cat: /sys/bus/scsi/devices/target5:0:0/model: No such file or directory
cat: /sys/bus/scsi/devices/target5:0:0/type: No such file or directory
cat: /sys/bus/scsi/devices/target5:0:0/rev: No such file or directory
cat: /sys/bus/scsi/devices/target5:0:0/scsi_level: No such file or directory

Remark: The above file locations are wrong! The vendor, model, type, rev, and scsi_level files are found in /sys/bus/scsi/devices/host0/target0:0:0/0:0:0:0/
and /sys/bus/scsi/devices/target0:0:0/0:0:0:0/ respectively.

Here the relevant dmesg output:

[ 13.639427] ata16.00: qc timeout (cmd 0xa1)
[ 13.647472] ata16.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[ 13.975470] ata16: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 23.975548] ata16.00: qc timeout (cmd 0xa1)
[ 23.983481] ata16.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[ 23.983487] ata16: limiting SATA link speed to 1.5 Gbps
[ 24.311470] ata16: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[ 54.311440] ata16.00: qc timeout (cmd 0xa1)
[ 54.319426] ata16.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[ 54.647471] ata16: SATA link up 1.5 Gbps (SStatus 113 SControl 310)

Remark 2: After disabling the Marvell SATA controller, there are no more messages in /var/log/xen/xend-debug.log, and no problems appear in the dmesg output. So it clearly points to an issue with the Xen system not recognizing the controller (see my first remark above).

Here the lspci -vv output relating the the Marvell SATA controller:

0b:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6
Gb/s RAID controller with HyperDuo (rev 11) (prog-if 01 [AHCI 1.0])
    Subsystem: ASUSTeK Computer Inc. P8P67 Deluxe Motherboard
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 383
    Region 0: I/O ports at b040 [size=8]
    Region 1: I/O ports at b030 [size=4]
    Region 2: I/O ports at b020 [size=8]
    Region 3: I/O ports at b010 [size=4]
    Region 4: I/O ports at b000 [size=16]
    Region 5: Memory at fb110000 (32-bit, non-prefetchable) [size=2K]
    Expansion ROM at fb100000 [disabled] [size=64K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot+,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Address: fee00798 Data: 0000
    Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
        DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
        LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0
<512ns, L1 <64us
            ClockPM- Surprise- LLActRep- BwNot-
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-,
Selectable De-emphasis: -6dB
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance-
ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB
    Capabilities: [100 v1] Advanced Error Reporting
        UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+
MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
    Kernel driver in use: ahci

5. The problem is reproducible! Every time the Marvell controller is enabled, there is an endless boot time with above error messages of not recognizing the controller. Option ROM enabled/disabled under the Marvell config option in BIOS does not make a difference, too. I have to completely disable the Marvell controller.

Question: Can I change the location where the initialization script is looking for the vendor, type, etc. information? Where do I find that script?

[Andrew Cooks (acooks)]

This bug affects vanilla kernels up to and including 3.6.3.

The problem seems to be a design issue with the Marvell controller. With VT-d enabled, each device gets its own "view" of memory it can get access to. The Marvell chip only registers one device per SATA port, but actually uses more than one device. It's this second phantom device that is not allowed memory access when VT-d is enabled.

It may eventually be worked around in the kernel, but it's not an easy fix. AFAIK the current thinking is to register the phantom device automatically when the first one is found.

Also see https://bugzilla.kernel.org/show_bug.cgi?id=42679