scsi-target and the buffer cache

[Scott Long]

Eric Anderson wrote:
> Nate Lawson wrote:
> 
>> Scott Long wrote:
>>
>>> Eric Anderson wrote:
>>>
>>>> Nate Lawson wrote:
>>>>
>>>>> Eric Anderson wrote:
>>>>>
>>>>>> I'm curious about whether a target mode device would use the 
>>>>>> buffer cache or not.  Here's a scenario:
>>>>>>
>>>>>> Host A: has fibre channel host adapter, in target mode, large 
>>>>>> memory pool, and another fiber channel host adapter connecting to 
>>>>>> fibre channel block device.
>>>>>> Host B: Fibre channel host adapter, connecting to Host A.  'sees' 
>>>>>> the target mode block device created by Host A.
>>>>>>
>>>>>> Will Host A use the buffer cache to cache blocks between the real 
>>>>>> block device, and the shared target mode device?
>>>>>> What about if Host A put a filesystem on the block device, created 
>>>>>> a single file the size of the filesystem, and shared that 
>>>>>> filesystem via a target mode device to Host B?
>>>>>> What I'm wanting is a box (FreeBSD?) that can be placed between a 
>>>>>> fibre channel block device (like a RAID array), and a fibre 
>>>>>> channel host using that block device, and act as a block cache for 
>>>>>> that device, using the FreeBSD's memory.  If it had a significant 
>>>>>> amount of memory, this could be very useful.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> If you use the example scsi_target usermode 
>>>>> (usr/share/examples/scsi_target), then the buffer cache will be 
>>>>> used since its reads/writes are from usermode like normal.  If you 
>>>>> don't want that behavior, you can set O_DIRECT in the open() call 
>>>>> of the backing store file.
>>>>>
>>>>> If you chose to modify the kernel side, you'd have to make sure 
>>>>> your accesses were through the VOP layer and then it would be cached.
>>>>>
>>>>> You should check to be sure the target mode performance meets your 
>>>>> expectations also.
>>>>>
>>>>
>>>> I guess I would be using the user mode tool, unless there's another 
>>>> way?  Your comment on performance also makes me a little worried 
>>>> about that now - do you think I would see a large performance hit?
>>>> Thanks!
>>>> Eric
>>>>
>>>>
>>>
>>> The way the target mode stack works in FreeBSD is that the kernel 
>>> provides some of the basic services, but the actual target emulator
>>> is meant to live in userland.  The userland program responds to
>>> events from the kernel via the select interface.  This generally
>>> works pretty well.  However, it does mean that control has to
>>> cross the kernel-userland boundary at least once for every event.
>>>
>>> What I'd suggest doing is prototyping your target emulator in userland
>>> and evaluating the performance there, and then moving it to the kernel
>>> if you _really_ need more performance.
>>
>>
>>
>> Agree 100%.  While having it in usermode means there are boundary 
>> crossings that increase per-transaction latency, the actual bulk data 
>> transfer is via zero-copy IO and you should be able to exceed the data 
>> transfer rates of several 10K RPM drives on decent hardware.
>>
> 
> 
> Ok, great.. Now, will scsi_target work ok with raw devices, or only 
> files?  (although I'm not sure theres all that much difference really).
> 
> Thanks!!
> Eric
> 
> 

You can write your userland code to use whatever files or devices you
want.  Are you talking about the scs_target.c code in
/usr/share/examples?  That's just a skeletal example that you can use
as a starting point for your own work.

Scott

Scott