[Bug 207602] 10.3-BETA 2 zroot doesn't boot with VT-d enabled and vmm in loader.conf

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207602

            Bug ID: 207602
           Summary: 10.3-BETA 2 zroot doesn't boot with VT-d enabled and
                    vmm in loader.conf
           Product: Base System
           Version: 10.3-BETA2
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: misc
          Assignee: freebsd-bugs at FreeBSD.org
          Reporter: ehrmann at gmail.com
                CC: freebsd-amd64 at FreeBSD.org
                CC: freebsd-amd64 at FreeBSD.org

I installed FreeBSD 10.3-BETA2 on a system with a Z170 chipset and Skylake CPU
on a ZFS root device. When I have both VT-d enabled in the bios, vmm enabled
(for bhyve) and pptdevs="<any pci dev>" (for PCI passthrough) in loader.conf,
sometime during the boot, there's an unrecoverable error with my SATA hard
drive, it's reattched, but the boot was interrupted, and the boot loader can't
find the ZFS root.

If either VT-d is disabled, vmm isn't enabled in loader.conf, or there's no PCI
passthrough device, the system boots fine.

The cause seems to be an interrupt storm. It always happens on IRQ64, even when
a different device claims it. It's the lowest IRQ listed by vmstat -i.

Here's my /boot/loader.conf:

    kern.geom.label.gptid.enable="0"
    net.link.tap.up_on_open="1"
    net.inet.ip.forwarding="1"
    zfs_load="YES"
    pptdevs="0/31/6"
    vmm_load="YES"

This is the device I'm trying to do PCI passthrough on
    ppt0 at pci0:0:31:6:       class=0x020000 card=0x86721043 chip=0x15b88086
rev=0x31 hdr=0x00
        vendor     = 'Intel Corporation'
        device     = 'Ethernet Connection (2) I219-V'
        class      = network
        subclass   = ethernet

Because this is a modern motherboard/chipset, it lacks IO like RS-232, and I
couldn't capture the log directly. The best I could do is this video:
http://imgur.com/wiiU9Bf

Root mount waiting for: usbus1 usbus0
uhub1: 4 ports with 4 removable, self powered
uhub0: 26 ports with 26 removable, self powered
Root mount waiting for: usbus0
<another line?  it's not clear in the video>
xhci_interrupt: host controller halted
<repeated ~200 times>
interrupt storm detected on "irq264:"; throttling interrupt source
Trying to mount root from zfs:zroot/ROOT/default []...
uhub0: (ada0:ahcich0:0:0:0) READ_FPDMA_QUEUED. ACB: 60 00 00 00 00 40 00 00 00
01 00 00
(ada:ahcich0:0:0:0): CAM status: CCB request was invalid
at usbus0, port 1. addr 1 (disconnected)
xhci_interrupt: host controller halted
(ada0:ahcich0:0:0:0): Error 22. Unretryable error
xhci_interrupt: host controller halted
<missing lines?>
(aprobe0:ahcich0:0:0:0): CAM status: CCB request was invalid
(aprobe0:ahcich0:0:0:0): Error 22. Unretryable error.
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: ... detached
Solaris: NOTICE: Cannot find the pool label for 'zroot'
Mounting from zfs:zroot/ROOT/default failed with error 5.

-- 
You are receiving this mail because:
You are on the CC list for the bug.