Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

Fri Sep 2 12:55:18 UTC 2016

Hello,

Setup:

nvidia GTX960 in PCIe slot
intel i7-4790K CPU
FreeBSD 11-RC2 host
CentOS 7 guest with kernel 3.10.0-327.28.3.el7.x86_64
Using vm-bhyve port

I've hit two issues:

1. BAR allocation

Workaround (for me) is adding "pci=nocrs" to linux guest's kernel 
command line.
Without "pci=nocrs" (or with "pci=use_crs"), the GTX960 doesn't get its 
256MB block allocated.

Bhyve's ACPI table produces this error linux-side regardless of "pci=" 
setting:

[    0.163085] acpi PNP0A03:00: host bridge window 
[0xd000000000-0xd0100fffff] (ignored, not CPU addressable)

which then leads to this:

[    0.215369] pci 0000:00:06.0: can't claim BAR 1 [mem 
0xd000000000-0xd00fffffff 64bit pref]: no compatible bridge window

and then, with "pci=use_crs" (i.e. use ACPI host bridge windows):

[    0.164030] pci_bus 0000:00: root bus resource [bus 00]
[    0.164379] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7]
[    0.164799] pci_bus 0000:00: root bus resource [io 0x0d00-0x1fff]
[    0.165206] pci_bus 0000:00: root bus resource [io 0x2000-0x211f]
[    0.165623] pci_bus 0000:00: root bus resource [mem 
0xc0000000-0xc41fffff]
...
[    0.231762] pci 0000:00:06.0: BAR 1: no space for [mem size 
0x10000000 64bit pref]
[    0.232263] pci 0000:00:06.0: BAR 1: trying firmware assignment [mem 
size 0x10000000 64bit pref]
[    0.232855] pci 0000:00:06.0: BAR 1: [mem size 0x10000000 64bit pref] 
conflicts with PCI mem [mem 0x00000000-0x7fffffffff]
[    0.233579] pci 0000:00:06.0: BAR 1: failed to assign [mem size 
0x10000000 64bit pref]

but with "pci=nocrs" (i.e. ignore ACPI host bridge windows):

[    0.163967] pci_bus 0000:00: root bus resource [bus 00]
[    0.164323] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[    0.164745] pci_bus 0000:00: root bus resource [mem 
0x00000000-0x7fffffffff]
...
[    0.230203] pci 0000:00:06.0: BAR 1: assigned [mem 
0x140000000-0x14fffffff 64bit pref]

2. "can't derive routing for PCI INT"

Linux-side dmesg related output:

[    1.677168] nvidia 0000:00:06.0: can't derive routing for PCI INT A
[    1.677600] nvidia 0000:00:06.0: PCI INT A: no GSI

Host-side info (when GTX960 is NOT configured as a pass-thru dev):

vgapci0 at pci0:1:0:0:     class=0x030000 card=0x19623842 chip=0x140110de 
rev=0xa1 hdr=0x00
     vendor     = 'NVIDIA Corporation'
     device     = 'GM206 [GeForce GTX 960]'
     class      = display
     subclass   = VGA
     bar   [10] = type Memory, range 32, base 0xf6000000, size 16777216, 
enabled
     bar   [14] = type Prefetchable Memory, range 64, base 0xe0000000, 
size 268435456, enabled
     bar   [1c] = type Prefetchable Memory, range 64, base 0xf0000000, 
size 33554432, enabled
     bar   [24] = type I/O Port, range 32, base 0xe000, size 128, enabled
     cap 01[60] = powerspec 3  supports D0 D3  current D0
     cap 05[68] = MSI supports 1 message, 64 bit enabled with 1 message
     cap 10[78] = PCI-Express 2 legacy endpoint max data 128(256) RO
                  link x4(x16) speed 2.5(8.0)
     ecap 0002[100] = VC 1 max VC0
     ecap 001e[258] = unknown 1
     ecap 0004[128] = Power Budgeting 1
     ecap 0001[420] = AER 2 0 fatal 0 non-fatal 4 corrected
     ecap 000b[600] = Vendor 1 ID 1
     ecap 0019[900] = PCIe Sec 1 lane errors 0xf

Linux-side info (when GTX960 IS configured as a pass-thru dev, also with 
"pci=nocrs"):

00:06.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 
960] (rev a1) (prog-if 00 [VGA controller])
         Subsystem: eVga.com. Corp. Device 1962
         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping- SERR- FastB2B- DisINTx-
         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
         Latency: 0, Cache Line Size: 64 bytes
         Interrupt: pin A routed to IRQ 16
         Region 0: Memory at c1000000 (32-bit, non-prefetchable) [size=16M]
         Region 1: Memory at 140000000 (64-bit, prefetchable) [size=256M]
         Region 3: Memory at c2000000 (64-bit, prefetchable) [size=32M]
         Region 5: I/O ports at 2080 [size=128]
         Expansion ROM at f7000000 [disabled] [size=512K]
         Capabilities: [60] Power Management version 3
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
         Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
                 Address: 0000000000000000  Data: 0000
         Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s 
unlimited, L1 <64us
                         ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
Unsupported-
                         RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                         MaxPayload 128 bytes, MaxReadReq 512 bytes
                 DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ 
AuxPwr- TransPend-
                 LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not 
supported, Exit Latency L0s <1us, L1 <4us
                         ClockPM+ Surprise- LLActRep- BwNot-
                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                 LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ 
DLActive- BWMgmt- ABWMgmt-
                 DevCap2: Completion Timeout: Range AB, TimeoutDis+, 
LTR+, OBFF Via message
                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, 
LTR+, OBFF Disabled
                 LnkSta2: Current De-emphasis Level: -6dB, 
EqualizationComplete+, EqualizationPhase1+
                          EqualizationPhase2+, EqualizationPhase3+, 
LinkEqualizationRequest-
         Kernel driver in use: nvidia

Any ideas?

Thanks,
Dom