scsi-target and the buffer cache

[Nate Lawson]

Scott Long wrote:
> Eric Anderson wrote:
> 
>> Nate Lawson wrote:
>>
>>> Eric Anderson wrote:
>>>
>>>> I'm curious about whether a target mode device would use the buffer 
>>>> cache or not.  Here's a scenario:
>>>>
>>>> Host A: has fibre channel host adapter, in target mode, large memory 
>>>> pool, and another fiber channel host adapter connecting to fibre 
>>>> channel block device.
>>>> Host B: Fibre channel host adapter, connecting to Host A.  'sees' 
>>>> the target mode block device created by Host A.
>>>>
>>>> Will Host A use the buffer cache to cache blocks between the real 
>>>> block device, and the shared target mode device?
>>>> What about if Host A put a filesystem on the block device, created a 
>>>> single file the size of the filesystem, and shared that filesystem 
>>>> via a target mode device to Host B?
>>>> What I'm wanting is a box (FreeBSD?) that can be placed between a 
>>>> fibre channel block device (like a RAID array), and a fibre channel 
>>>> host using that block device, and act as a block cache for that 
>>>> device, using the FreeBSD's memory.  If it had a significant amount 
>>>> of memory, this could be very useful.
>>>
>>>
>>>
>>>
>>> If you use the example scsi_target usermode 
>>> (usr/share/examples/scsi_target), then the buffer cache will be used 
>>> since its reads/writes are from usermode like normal.  If you don't 
>>> want that behavior, you can set O_DIRECT in the open() call of the 
>>> backing store file.
>>>
>>> If you chose to modify the kernel side, you'd have to make sure your 
>>> accesses were through the VOP layer and then it would be cached.
>>>
>>> You should check to be sure the target mode performance meets your 
>>> expectations also.
>>>
>>
>> I guess I would be using the user mode tool, unless there's another 
>> way?  Your comment on performance also makes me a little worried about 
>> that now - do you think I would see a large performance hit?
>> Thanks!
>> Eric
>>
>>
> 
> The way the target mode stack works in FreeBSD is that the kernel 
> provides some of the basic services, but the actual target emulator
> is meant to live in userland.  The userland program responds to
> events from the kernel via the select interface.  This generally
> works pretty well.  However, it does mean that control has to
> cross the kernel-userland boundary at least once for every event.
> 
> What I'd suggest doing is prototyping your target emulator in userland
> and evaluating the performance there, and then moving it to the kernel
> if you _really_ need more performance.

Agree 100%.  While having it in usermode means there are boundary 
crossings that increase per-transaction latency, the actual bulk data 
transfer is via zero-copy IO and you should be able to exceed the data 
transfer rates of several 10K RPM drives on decent hardware.

-- 
Nate