CAM Target Layer and dev/isp

[Trent Nelson]

On 7/17/12 12:46 PM, "Kenneth D. Merry" <ken at freebsd.org> wrote:

>On Mon, Jul 16, 2012 at 20:14:31 -0700, Trent Nelson wrote:
>> I'm not sure how to expose LUNs via my isp devices.  Here's what I've
>>got
>> so far:
>> 
>> [root at s24/ttypts/1(../misc/isp/bin)#] for i in {0..3}; do; ./isp_getrole
>> /dev/isp$i; done
>> /dev/isp0 chan 0: role target
>> /dev/isp1 chan 0: role target
>> /dev/isp2 chan 0: role target
>> /dev/isp3 chan 0: role target
>
>Looks good.  What model Qlogic board is it?  I know the driver works
>pretty
>well in target mode with 4Gb and 8Gb boards, but I don't know how well 2Gb
>and older boards work.

I'm still frolicking in 2Gb land, SAN-wise.  All my HBAs are Qlogic 2Gb
(single and dual port) models.  Sample dmesg from one of the dual port
ones:

sp0: Board Type 2312, Chip Revision 0x2, loaded F/W Revision 3.3.26
isp0: Attributes: TargetMode SCC-Lun Fabric 2K-Login
isp0: 876 max I/O command limit set


>
>> [root at s24/ttypts/1(../misc/isp/bin)#] ctladm devlist -v
>> LUN Backend       Size (Blocks)   BS Serial Number    Device ID
>>   0 ramdisk             2097152  512 MYSERIAL   0     MYDEVID   0
>>       lun_type=0
>
>That looks fine.  Note that it is a fake ramdisk that is only backed by
>1MB
>of memory no matter how large the reported size is.  So you can make it as
>large as you want.

Haha, yeah I was wondering about the "10485760000000000" you used in the
examples.  You should definitely make mention of that 1MB fact in the docs
somewhere if you're going to use 9536TB as the size in the example ;-)

(My interaction went like this: created the crazy huge one used in the
examples, was thoroughly confused when I realized it was 9536TB, deleted
it, then created a 1GB one instead.)

>All of the ports are online, so that looks good.
>
>> It's not obvious from the docs how to export CTL LUNs through my isp
>> devices.  I tried passing -D /dev/isp[n] to `ctladm create`, but that
>> returns: 'ctladm: cctl_create_lun: error issuing CTL_LUN_REQ ioctl:
>> Inappropriate ioctl for device'.
>
>The block backend only works on block devices or files.  e.g.:
>
>ctladm create -b block -o file=/path/to/my/file
>ctladm create -b block -o file=/dev/da5
>
>If you use a block device (like a zvol) as the backing store, you'll want
>to disable sending cache syncs to the disk, since that will trigger a GEOM
>assertion.
>
>ctladm realsync off
>
>(Do that before putting the ports online.)

Ah, ok, I played a bit more with block devices instead of ramdisk, and it
cleared up some stuff.

>
>> Sample output from one of the switches on my fabric (s24's /dev/isp1 is
>> plugged into port 8 of sf1):
>> 
>> sf1:admin> switchshow
>> switchName:	sf1
>> switchType:	17.2
>> switchState:	Online
>> switchMode:	Native
>> switchRole:	Principal
>> switchDomain:	1
>> switchId:	fffc01
>> switchWwn:	10:00:00:60:69:5a:1a:40
>> switchBeacon:	OFF
>> Zoning:      	ON (cfg_2012_07_16)
>> port  0: id N2 Online         F-Port 50:06:0b:00:00:13:18:72
>> port  1: id N2 No_Light
>> port  2: id N2 Online         E-Port 10:00:00:60:69:5a:09:c0 "sf2"
>> (downstream) 
>> port  3: id N2 No_Light
>> port  4: id N2 No_Light
>> port  5: id N2 Online         F-Port 21:01:00:e0:8b:ab:cf:be
>> port  6: id N2 Online         F-Port 21:01:00:e0:8b:a6:98:ca
>> port  7: id N2 Online         F-Port 21:01:00:e0:8b:30:2f:5b
>> port  8: id N1 Online         L-Port 1 public
>
>Looks like it is in loop mode.  Can your switch make a loop mode device
>visible on another port?

You know what, I have no idea what was going on there.  I can't replicate
that behavior (getting the port to present itself to the fabric as an
L-port) on another (almost identical) box.  And I managed to panic that
box whilst composing this e-mail, to the point where it doesn't even get
past the PCI BIOS routines.  (It happened when I unplugged the HBA, I'll
paste a backtrace and CC Matt in a separate e-mail.)

>  What are you using for an initiator?  Does it
>work if you connect the initiator directly to the FreeBSD target?

Ah, ok, those questions (and various other comments you made) have cleared
up a couple of things.  What I was expecting to see from the switch when I
did `portshow [n]` was a list of all the LUNs (via their WWNs) being made
available by camctl/isp/target-mode.  This is what I'm alluding to when I
say this:

> > Ideally I'd like to be able to use CTL to export multiple ZFS zvols as
> > separate targets (i.e. all with unique WWNN/WWPNs), such that, from the
> > fabric's point of view, the port would look like just another FC-AL
>JBOD,
> > like, say, port 11:
> > 
> > sf1:admin> portshow 11
[snip]
> > portWwn:    20:0b:00:60:69:5a:1a:40
> > portWwn of device(s) connected: 	21:00:00:14:c3:ca:23:ca
> > 	21:00:00:14:c3:c1:9c:90
> > 	21:00:00:14:c3:c4:41:9a
> > 	21:00:00:14:c3:c1:23:6b
> > 	21:00:00:14:c3:ca:23:df
> > 	21:00:00:14:c3:c4:47:97
> > 	21:00:00:14:c3:ca:20:0e
> > 	21:00:00:14:c3:c4:40:ca

However, it just occurred to me that in order for that to happen, the
block LUN I'm exporting would essentially have to mimic/implement an
FC-AL-ported disk (that is public-loop aware) -- I.e. it would have to
support fabric login and all the other fancy cruft my physical dual-ported
FC SCSI drives in the FC-AL JBODs implement.


I reckon it's safe to assume none of that is in place, no? ;-)

(Would that even be possible?  Seems like it would need a lot of
underlying driver support in dev/isp, at the very least.  As well as a
bucket-load of ctl support.  Perhaps better suited to a new backend type?
I.e. `ctladm create -b fc-al ...`.)

Now, with that being said, back to your questions...

> What are you using for an initiator?
> Does it work if you connect the initiator directly to the FreeBSD target?


Right, so, I haven't played around with any initiators yet as I wasn't
seeing the expected output from the SAN switch (I.e. the WWNs of "devices
connected to this port").  Now that I know this isn't going to happen,
I'll try see if I can forcibly connect initiators.  I was happy to see
that when I created a block device via ctladm, it was assigned WWPN/WWNNs
automatically.

I'm not sure if this means I'll still be able to use a fabric (with zoning
enabled) or not.  If I have no luck with that approach, I'll try plug the
initiator's HBA port directly into the target HBA and see if that helps.

>If your switch has NP-IV support, you can also try creating multiple
>virtual ports with the isp(4) driver if you set the hint.isp.0.vports=N
>loader tunable, where N is the number of virtual ports.  I haven't tried
>that in several years, though, and Matt Jacob has indicated it needs more
>testing.

Yeah unfortunately neither my switches nor my HBAs have support for VSAN
stuff.

>As for using zvols, the code that is in FreeBSD right now will lead to
>very
>slow performance with zvols.  Justin Gibbs and Will Andrews gave a talk at
>BSDCan that explained their work to eliminate COW (Copy On Write) faults
>for files used as block devices and zvols.  Their slides are here:
>
>http://www.bsdcan.org/2012/schedule/events/316.en.html
>
>And the talk itself is here:
>
>http://www.youtube.com/watch?v=LtY3vpX-cdM
>
>It's fine to use zvols now, but you may want to wait until their code does
>into FreeBSD/head at least to use zvols for anything that requires
>reasonable performance.
>
>Until their code goes in the tree, you'll probably get somewhat faster
>performance by using files on top of ZFS, or on top of UFS.  (UFS will be
>much faster at the moment, but you don't get the software RAID
>functionality of ZFS.)
>
>The first write pass through on a zvol or a ZFS file will go very quickly,
>but subsequent writes will be pretty slow, especially if they are not on
>exact ZFS record boundaries.

Gotcha'.  Super info, thanks.  The "perfect" solution down the track would
be zvol, but there's an enormous amount of other stuff I'd need to flush
about before that.

(What I'm aiming to do is present zvols to either virtual or physical
boxes for the OS install disk.  That would allow me to snapshot/clone
entire OS instances (I.e. AIX/HP-UX/IRIX/Tru64-UNIX et al) and export them
as a new disk that could be picked up by another box, which would be
incredibly cool for provisioning, tinkering, dev stuff, etc.)

>Also, Matt Jacob and I are chasing down a possible data corruption bug.
>We don't know exactly where it is, and it might not be in the code that
>is in FreeBSD right now.  The point is, run some data integrity tests
>before using this in production.

Heh, roger.  (FWIW, this is all pie-in-the-sky level at the moment --
it'll be a while before I can put it into production.)

>
>> Also, the HA stuff sounds bad-ass.  What's the best way to stay up to
>>date
>> with CTL development?  Watch commits to sys/cam/ctl?  (There's no
>>separate
>> list or anything for this stuff, right?)
>
>That's pretty much the best way to keep up with it, there's not a separate
>mailing list.
>
>I don't think I'm going to have time to do anything with the HA hooks in
>the near future.  Hopefully other folks will be interested and do some
>development in that area.  It would be nice to have a fully HA block
>stack,
>but that will take a lot of effort.

Nod.  It's certainly not a deal breaker for me, and I can get around it by
other (clunky) means, but it certainly would be awesome.  I'd essentially
be able to build a ZFS-backed storage controller indistinguishable from
the (proprietary) HA RAID controllers that I was originally intended to
use with my disk arrays instead of just straight JBOD.)

Thanks for such thorough information.  Very useful.  I've got a few things
to play around with now.

Regards,

	Trent.