Pipes, cat buffer size

[Ivan Voras]

Dan Nelson wrote:
> In the last episode (Oct 18), Ivan Voras said:
>> I'm working on a program that's intended to be used as a "filter", as
>> in "something | myprogram > file". I'm trying it with cat and I'm
>> seeing my read()s return small blocks, 64 kB in size. I suppose this
>> is because cat writes in 64 kB blocks. So:
>>
>> a) Is there a way to programatically, per-process, set the pipe buffer
>> size? The program in question is a compressor and it's particularly
>> inefficient when given small blocks and I'm wondering if the system can
>> buffer enough data for it.
> 
> Why not keep reading until you reach your desired compression block
> size?  Bzip2's default blocksize is 900k, for example.

Of course. But that's not the point :) From what I see (didn't look at
the code), Linux for example does some kind of internal buffering that
decouples how the reader and the writer interact. I think that with
FreeBSD's current behaviour the writer could write 1-byte buffers and
the reader will be forced to read each byte individually. I don't know
if there's some ulterior reason for this.

>> b) Is there any objection to the following patch to cat:
> 
> It might be simpler to just use "dd if=myfile obs=1m" instead of
> patching cat.

I believe patching cat to bring its block size into the century of the
fruitbat has its own benefits.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 258 bytes
Desc: OpenPGP digital signature
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20081018/e23ea5db/signature.pgp