scsi-target and the buffer cache

[Eric Anderson]

Scott Long wrote:

> Eric Anderson wrote:
>
>> Nate Lawson wrote:
>>
>>> Scott Long wrote:
>>>
>>>> Eric Anderson wrote:
>>>>
>>>>> Nate Lawson wrote:
>>>>>
>>>>>> Eric Anderson wrote:
>>>>>>
>>>>>>> I'm curious about whether a target mode device would use the 
>>>>>>> buffer cache or not.  Here's a scenario:
>>>>>>>
>>>>>>> Host A: has fibre channel host adapter, in target mode, large 
>>>>>>> memory pool, and another fiber channel host adapter connecting 
>>>>>>> to fibre channel block device.
>>>>>>> Host B: Fibre channel host adapter, connecting to Host A.  
>>>>>>> 'sees' the target mode block device created by Host A.
>>>>>>>
>>>>>>> Will Host A use the buffer cache to cache blocks between the 
>>>>>>> real block device, and the shared target mode device?
>>>>>>> What about if Host A put a filesystem on the block device, 
>>>>>>> created a single file the size of the filesystem, and shared 
>>>>>>> that filesystem via a target mode device to Host B?
>>>>>>> What I'm wanting is a box (FreeBSD?) that can be placed between 
>>>>>>> a fibre channel block device (like a RAID array), and a fibre 
>>>>>>> channel host using that block device, and act as a block cache 
>>>>>>> for that device, using the FreeBSD's memory.  If it had a 
>>>>>>> significant amount of memory, this could be very useful.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> If you use the example scsi_target usermode 
>>>>>> (usr/share/examples/scsi_target), then the buffer cache will be 
>>>>>> used since its reads/writes are from usermode like normal.  If 
>>>>>> you don't want that behavior, you can set O_DIRECT in the open() 
>>>>>> call of the backing store file.
>>>>>>
>>>>>> If you chose to modify the kernel side, you'd have to make sure 
>>>>>> your accesses were through the VOP layer and then it would be 
>>>>>> cached.
>>>>>>
>>>>>> You should check to be sure the target mode performance meets 
>>>>>> your expectations also.
>>>>>>
>>>>>
>>>>> I guess I would be using the user mode tool, unless there's 
>>>>> another way?  Your comment on performance also makes me a little 
>>>>> worried about that now - do you think I would see a large 
>>>>> performance hit?
>>>>> Thanks!
>>>>> Eric
>>>>>
>>>>>
>>>>
>>>> The way the target mode stack works in FreeBSD is that the kernel 
>>>> provides some of the basic services, but the actual target emulator
>>>> is meant to live in userland.  The userland program responds to
>>>> events from the kernel via the select interface.  This generally
>>>> works pretty well.  However, it does mean that control has to
>>>> cross the kernel-userland boundary at least once for every event.
>>>>
>>>> What I'd suggest doing is prototyping your target emulator in userland
>>>> and evaluating the performance there, and then moving it to the kernel
>>>> if you _really_ need more performance.
>>>
>>>
>>>
>>>
>>> Agree 100%.  While having it in usermode means there are boundary 
>>> crossings that increase per-transaction latency, the actual bulk 
>>> data transfer is via zero-copy IO and you should be able to exceed 
>>> the data transfer rates of several 10K RPM drives on decent hardware.
>>>
>>
>>
>> Ok, great.. Now, will scsi_target work ok with raw devices, or only 
>> files?  (although I'm not sure theres all that much difference really).
>>
>> Thanks!!
>> Eric
>>
>>
>
> You can write your userland code to use whatever files or devices you
> want.  Are you talking about the scs_target.c code in
> /usr/share/examples?  That's just a skeletal example that you can use
> as a starting point for your own work.


Alright.. I was indeed talking about the example code, but I suppose it 
wouldn't be too hard to make it work with raw devices.

Thanks for the help!

Eric




-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
Anything that works is better than anything that doesn't.
------------------------------------------------------------------------