sendfile(2) SF_NOPUSH flag proposal

[Igor Sysoev]

On Tue, 27 May 2003, Terry Lambert wrote:

> Igor Sysoev wrote:
> > > I don't see this as being terrifically useful; small files
> > > should probably just be mapped and written; the copy expense
> > > is still there for the headers and trailers, no matter what,
> > > and the file size itself is very small overhead, relatively
> > > speaking, for files small enough for this to be an issue.
> > 
> > FreeBSD 4.x has no zero_copy(9) so mmap()ed files would be copied.
> > sendfile() allows to avoid this copy.
> 
> It's actually "one copy", from an external mbuf reference
> to a VM buffer to the card, vs. two copies, from the mapped
> file in the process address space to mbufs, to the card.

I know what "zero copy" means.

> My point, though was that fr files small enough to fit in a
> packet, you are probably going to spend more on the headers
> and trailers copyin and the mbuf list assembly and the system
> call overhead, than you'll end up spending one the extra data
> copy for the very small file contents.
> 
> I would be really surprised if you were able to demonstrate a
> measuarble performance difference which was above the noise.

I hope I will demonstrate at least CPU usage in near future.

> > By the way what do you call by small files ? 4K, 30K or 100K ?
> 
> You were talking about the file and the header living in the
> same packet.

I mean that if you have 230 bytes header then sendfile() will send it
in separate packet nevertheless the size of header and of the file.
Something like this - 230, 1460, 1460, ...

> > > I also think your headers and trailers are very small, if
> > > they are fitting with the file contents in a single packet.
> > > I think this is atypical.
> > 
> > If I ask Google:
> > ----
> > HEAD /images/hp0.gif HTTP/1.0
> > Host: www.google.com
> > 
> > ----
> > it will return me 230 bytes:
> 
> The "HEAD" is atypical, compared to the "GET"; the full Google
> front page is larger than that, and consists of multiple files;
> assuming you support HTTP/1.1 and pipelining, it's going to be
> a back-to-back transfer involving multiple sendfile() calls.

I use HEAD to show you the size of the HTTP header.
The HEAD is atypical but such small HTTP header is typical.

> > > BTW: if you go ahead with this, you should verify that it
> > > also works for the trailers, etc., and you should probably
> > > skip it if you headers > transmit queue depth, or file size
> > > > transmit queue depth, or trailers > transmit queue depth.
> > 
> > Currently sendfile() can send the file in not full packets even
> > file is bigger then the transmit queue depth. It can send
> > it in 2 x 1460 + 1176 or 5 x 1460 + 892 packets.
> 
> 3 packets vs. 6.  And using HTTP/1.0, there's also the three
> handshake packets, SYN/SYN-ACK/ACK, and the tear-down three
> teardown packets, FIN/FIN-ACK/ACK (or 4), plus the ACK's for
> the packets you sent (should be one ACK, since that's below
> the TCP window size).

Actually 6 vs. 6 for this 8K file. But I said about another thing.
Let's see 48K file and 250 bytes header. sendfile() usually sends
it as 4K or 8K hunks so there are 48/8 * 6 + 1 (header) = 37 packets.
But (48K + 250) / 1460 = 33 * 1460 + 1270 i.e. 34 packets.
It's 8% decrease of data packets. Add here the possible retransmitions.

> Really: it's in the noise.  Unless you are paying by packet
> count, you probably shouldn't care.

So do you consider that IP fragmentation is the good thing ?


Igor Sysoev
http://sysoev.ru/en/