scsi-target and the buffer cache

[Scott Long]

Eric Anderson wrote:
> Nate Lawson wrote:
> 
>> Eric Anderson wrote:
>>
>>> I'm curious about whether a target mode device would use the buffer 
>>> cache or not.  Here's a scenario:
>>>
>>> Host A: has fibre channel host adapter, in target mode, large memory 
>>> pool, and another fiber channel host adapter connecting to fibre 
>>> channel block device.
>>> Host B: Fibre channel host adapter, connecting to Host A.  'sees' the 
>>> target mode block device created by Host A.
>>>
>>> Will Host A use the buffer cache to cache blocks between the real 
>>> block device, and the shared target mode device?
>>> What about if Host A put a filesystem on the block device, created a 
>>> single file the size of the filesystem, and shared that filesystem 
>>> via a target mode device to Host B?
>>> What I'm wanting is a box (FreeBSD?) that can be placed between a 
>>> fibre channel block device (like a RAID array), and a fibre channel 
>>> host using that block device, and act as a block cache for that 
>>> device, using the FreeBSD's memory.  If it had a significant amount 
>>> of memory, this could be very useful.
>>
>>
>>
>> If you use the example scsi_target usermode 
>> (usr/share/examples/scsi_target), then the buffer cache will be used 
>> since its reads/writes are from usermode like normal.  If you don't 
>> want that behavior, you can set O_DIRECT in the open() call of the 
>> backing store file.
>>
>> If you chose to modify the kernel side, you'd have to make sure your 
>> accesses were through the VOP layer and then it would be cached.
>>
>> You should check to be sure the target mode performance meets your 
>> expectations also.
>>
> 
> I guess I would be using the user mode tool, unless there's another 
> way?  Your comment on performance also makes me a little worried about 
> that now - do you think I would see a large performance hit?
> Thanks!
> Eric
> 
> 

The way the target mode stack works in FreeBSD is that the kernel 
provides some of the basic services, but the actual target emulator
is meant to live in userland.  The userland program responds to
events from the kernel via the select interface.  This generally
works pretty well.  However, it does mean that control has to
cross the kernel-userland boundary at least once for every event.

What I'd suggest doing is prototyping your target emulator in userland
and evaluating the performance there, and then moving it to the kernel
if you _really_ need more performance.

Scott