NAS w/ multipath

[Kenneth D. Merry]

On Tue, Feb 07, 2006 at 12:22:33 -0800, Tom Samplonius wrote:
> 
> On Tue, 7 Feb 2006, Palle Girgensohn wrote:
> 
> >Hi!
> >
> >I'd like to connect a FreeBSD system to an IBM NAS system, using a 
> >FibreChannel connection. I can find two supported drivers in the system, 
> >mpt(4) and isp(4). It seems to me, none of them supports multipath, 
> >something we need. Is this a false assumption? Are there other drivers 
> >that do support multipath? Recommendations apprecieated.
> >
> >If not, an alternative would be iSCSI, but I have no experience and it 
> >seems it is hardly implemented at all for FreeBSD, is that true?
> >
> >Thanks
> >Palle
> >
> 
>   Well, typically HBA drivers don't need to support mulitpathing, and that 
>   is implemented int the SCSI layer of the OS.  Windows is a bit different, 
> since Windows tried to mount every device (no /etc/fstab file), so Windows 
> HBA drivers typically need multipath control to hide the fact that a single 
> logical disk is visible on multiple channels.
> 
>   With multipath, you typically just see the logical disk on two separate 
> channels (separate LUNs).  Depending on your disk controller, you might 
> need "trespass" support in the OS/driver, but that seems rare these days.
> 
>   So you can either do a cold multipath, by having the second path listed 
>   in your /etc/fstab, but leave it commented out, until you need to use the 
> second path.
> 
>   Now, in FreeBSD you could also multipath in the GEOM layer.  GEOM knows 
>   about devices going away, and knows how to handle that (ex. gmirror).  
> There is some support in GEOM for round-robin IO to two devices.  However, 
> phk has reported that the isp driver can hang forever on some timeouts, so 
> it might not be useful.  And I don't even know if GEOM round-robin is even 
> finished.
> 
>   It might an idea for a few people to get together some dollars for a 
> sponsership on FreeBSD SAN support, specifically:
> 
> - isp driver updates (timeout handling)

I think Matt's plan is to add a sysctl in the isp driver that will let the
user tweak the behavior there.

> - Multipath support support, either in CAM SCSI layer or in GEOM disk layer.

The right way to do it is in CAM, either at the transport layer level or
the peripheral driver level.  To get true multipathing support, there is a
reasonable amount of SCSI-specific stuff that you need to do to identify
which devices are actually multiple paths to the same device, and also
which path to that device should be used.  That stuff would be a lot easier
to do inside CAM, so we could just export the same basic disk API that we
do currently.

e.g. da0 would actually point to two or more paths to the same device, and
would use information from those devices to determine which paths to use.
(You can have an active/passive device, an active/active device where one
path is preferred, or an active/active device where it doesn't matter which
path you use...)

The quicker way to do it is in GEOM, and Poul-Henning has already been
talking about getting GEOM_FOX working to do basic multipathing.

Ken
-- 
Kenneth Merry
ken at FreeBSD.ORG