scsi-target and the buffer cache

[Eric Anderson]

Nate Lawson wrote:

> Scott Long wrote:
>
>> Eric Anderson wrote:
>>
>>> Nate Lawson wrote:
>>>
>>>> Eric Anderson wrote:
>>>>
>>>>> I'm curious about whether a target mode device would use the 
>>>>> buffer cache or not.  Here's a scenario:
>>>>>
>>>>> Host A: has fibre channel host adapter, in target mode, large 
>>>>> memory pool, and another fiber channel host adapter connecting to 
>>>>> fibre channel block device.
>>>>> Host B: Fibre channel host adapter, connecting to Host A.  'sees' 
>>>>> the target mode block device created by Host A.
>>>>>
>>>>> Will Host A use the buffer cache to cache blocks between the real 
>>>>> block device, and the shared target mode device?
>>>>> What about if Host A put a filesystem on the block device, created 
>>>>> a single file the size of the filesystem, and shared that 
>>>>> filesystem via a target mode device to Host B?
>>>>> What I'm wanting is a box (FreeBSD?) that can be placed between a 
>>>>> fibre channel block device (like a RAID array), and a fibre 
>>>>> channel host using that block device, and act as a block cache for 
>>>>> that device, using the FreeBSD's memory.  If it had a significant 
>>>>> amount of memory, this could be very useful.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> If you use the example scsi_target usermode 
>>>> (usr/share/examples/scsi_target), then the buffer cache will be 
>>>> used since its reads/writes are from usermode like normal.  If you 
>>>> don't want that behavior, you can set O_DIRECT in the open() call 
>>>> of the backing store file.
>>>>
>>>> If you chose to modify the kernel side, you'd have to make sure 
>>>> your accesses were through the VOP layer and then it would be cached.
>>>>
>>>> You should check to be sure the target mode performance meets your 
>>>> expectations also.
>>>>
>>>
>>> I guess I would be using the user mode tool, unless there's another 
>>> way?  Your comment on performance also makes me a little worried 
>>> about that now - do you think I would see a large performance hit?
>>> Thanks!
>>> Eric
>>>
>>>
>>
>> The way the target mode stack works in FreeBSD is that the kernel 
>> provides some of the basic services, but the actual target emulator
>> is meant to live in userland.  The userland program responds to
>> events from the kernel via the select interface.  This generally
>> works pretty well.  However, it does mean that control has to
>> cross the kernel-userland boundary at least once for every event.
>>
>> What I'd suggest doing is prototyping your target emulator in userland
>> and evaluating the performance there, and then moving it to the kernel
>> if you _really_ need more performance.
>
>
> Agree 100%.  While having it in usermode means there are boundary 
> crossings that increase per-transaction latency, the actual bulk data 
> transfer is via zero-copy IO and you should be able to exceed the data 
> transfer rates of several 10K RPM drives on decent hardware.
>


Ok, great.. Now, will scsi_target work ok with raw devices, or only 
files?  (although I'm not sure theres all that much difference really).

Thanks!!
Eric


-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
Anything that works is better than anything that doesn't.
------------------------------------------------------------------------