PCIe SATA HBA for ZFS on -STABLE

[Matthew Dillon]

:I'm not on the -STABLE list so please reply to me.
:
:I'm using an Intel Core i3-530 on a Gigabyte H55M-D2H motherboard with 8 x
:2TB drives & 2 x 1TB drives.
:The plan is to have the 1 TB drives in a zmirror and the 8 in a raidz2.
:
:Now the Intel chipset has only 6 on board SATA II ports so ideally I'm
:looking for a non RAID SATA II HBA to give me 6 extra ports (4 min).
:Why 6 extra ?
:Well the case I'm using has 2 x eSATA ports so 6 would be ideal, 5 OK, and 4
:the minimum I need to do the job.
:
:So...
:
:What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
:using ZFS ?
:
:Not wanting to break the bank.
:Not interested in SATA III 6GB at this time... though it could be useful if
:I add an SSD for... (is it ZIL ?).
:Can this be added at any time ?
:
:The main issue is I need at least 10 ports total for all existing drives...
:ZIL would require 11 so ideally we are talking a 6 port HBA.

    The absolute cheapest solution is to buy a Sil-3132 PCIe card
    (providing 2 E-SATA ports), and then connect an external port multiplier
    to each port.  External port multiplier enclosures typically support
    5 drives each so that would give you your 10 drives.

    Even the 3132 is a piss-ant little card it does support FIS-Based
    switching so performance will be very good... it will just be limited
    to SATA-II speeds is all.

    Motherboard AHCI-based SATA ports typically do NOT have FIS-Based
    switching support (this would be the FBSS capability flag when the AHCI
    driver probes the chipset).  This means that while you can attach an
    external port multiplier enclosure to mobo SATA ports (see later
    on E-SATA vs SATA), read performance from multiple drives concurrently
    will be horrible.  Write performance will still be decent due to drive
    write caches despite being serialized.

    On E-SATA vs SATA.  Essentially there are only two differences between
    E-SATA and SATA.  One is the cable and connector format.  The other is
    hot swap detection.  Most mobo SATA ports can be strung out to E-SATA
    with an appropriate adapter.  High-end Intel ASUS mobos often come with
    such adapters (this is why they usually don't sport an actual E-Sata
    port on the backplane) and the BIOS has setup features to specify E-SATA
    on a port-by-port basis.

    --

    For SSDs you want to directly connect the SSD to a mobo SATA port and
    then either mount the SSD in the case or mount it in a hot-swap gadget
    that you can screw into a PCI slot (it doesn't actually use the PCI
    connector, just the slot).  A SATA-III port with a SATA-III SSD really
    shines here and 400-500 MBytes/sec random read performance from a single
    SSD is possible, but it isn't an absolute requirement.  A SATA-II port
    will still work fine as long as you don't mind maxing out the bandwidth
    at 250 MBytes/sec.

    --

    I can't help with any of the other questions.  Someone also suggested
    the MPS driver for FreeBSD, with caveats.

    I'll add a caveat on the port multiplier enclosures.  Nearly all such
    enclosures use another SIL chipset internally and it works pretty well
    EXCEPT that it isn't 100% dependable if you try to hot-swap drives in the
    enclosure while other drives in the enclosure are active.  So with that
    caveat, I recommend the port multiplier enclosure as the cheapest solution.

    To get robust hot-swap enclosures you either need to go with SAS or you
    need to go with discrete SATA ports (no port multiplication), and the
    ports have to support hot-swap.  The best hot-swap support for an AHCI
    port is if the AHCI chipset supports cold-presence-detect (CPD), and
    again Mobo AHCI chipsets usually don't.  Hot-swap is a bit hit or miss
    without CPD because power savings modes can effectively prevent hot-swap
    detect from working properly.  Drive disconnects will always be detected
    but drive connects might not be.

    And even with discrete SATA ports the AHCI firmware on mobos does not
    necessarily handle hot-swap properly.  For example my Intel-I7 ASUS mobo
    will generate spurious interrupts and status on a DIFFERENT discrete
    SATA port when I hot swap on some other discrete SATA port, in addition
    to generating the status interrupt on the correct port.  So then it comes
    down to the driver in the operating system properly handling the
    spurious status and properly stopping and restarting pending commands
    when necessary.   So, again, it is best for the machine to be idle before
    attempting a hot-swap.

    Lots of caveats.  Sorry... you can blame Intel for all the blasted issues
    with AHCI and SATA.  Intel didn't produce a very good chipset spec and
    vendors took all sorts of liberties.

					-Matt
					Matthew Dillon 
					<dillon at backplane.com>