FreeBSD 4.x and OS-X tcp performance

[Charles Sprickman]

Hello all,

As promised, I have 4 tcpdump traces (saved with the "-w" option 
per request of another poster).  Since these are a little too big to be 
broadcast to everyone on the list, I'm posting them here:

http://home.manymonkeys.com/tcpdump/

For all tests I used os-x's command line ftp program and uploaded a 1.7MB 
file to each system.

tcpdump.osx-fbsd - tcpdump ran on os-x box while transferring to FBSD 
tcpdump.fbsd-osx - tcpdump ran on FBSD box while transferring from os-x

tcpdump.osx-obsd - tcpdump ran on os-x box while transferring to OBSD
tcpdump.obsd-osx - tcpdump ran on OBSD box while transferring from os-x

So there's a view of a single file transfer on each run of the test.

Observing this showed that the os-x to fbsd transfer went at about 200KB/s 
and the os-x to obsd transfer went at about 2.6MB/s.

Again I'd like to note that the OpenBSD box is at 3.3, which may prove 
important since it's quite dated.

If there's anything else I can provide, let me know.

Thanks,

Charles

On Mon, 7 Mar 2005, Daniel Hartmeier wrote:

> On Mon, Mar 07, 2005 at 02:04:01PM -0500, Charles Sprickman wrote:
>
>> Very interesting, thank you for that read of the tcpdump output.  If you
>> have the time, could you post back a few lines of the tcpdump with
>> comments so that I might learn a little about what's going on?  I don't
>> have the best understanding of the intricacies of tcp...
>
> Data is flowing only in one direction (from client to server), so we
> see the client sending payload and the server acknowledging it.
>
> As long as no packets are lost and the server keeps acknowledging, the
> client could send one continuous stream of maximum sized payload packets.
> However, if the server stops acknowledging, the client must stop sending
> more data once it has filled up the window advertised by the server and
> wait for the server to either acknowledge further data or increase its
> window.
>
> Nothing exciting happens until timestamp 16:41:29.008909 in your log.
>
> 16:41:29.008672 client 321457:322905(1448)
> 16:41:29.008751 server ack 322905 win 7240
> 16:41:29.008909 client 324353:325801(1448)
>
> Since the log represents the server's view, the lack of segment
> 324353:322905 in the log means that this packet from the client was lost
> in transit. Now the server keeps acknowledging 322905, and we expect the
> client to notice and retransmit the lost segment.
>
> 16:41:29.009064 server ack 322905 win 7400
> 16:41:29.009146 client 325801:327249(1448)
> 16:41:29.009233 server ack 322905 win 8424
> 16:41:29.009409 client 328697:330145(1448)
> 16:41:29.009521 server ack 322905 win 9448
>
> The client hasn't noticed yet and is still sending further segments past
> the gap. The server is advertising increasing window sizes, probably
> because it's still draining the data up to the gap.
>
> Notice that there's a new gap created as 327249:328697(1448) was lost.
>
> 16:41:29.011250 client 335937:337385(1448)
> 16:41:29.011331 server ack 322905 win 14568
>
> Now the client has filled up the window. If it doesn't retransmit now,
> it has to stall. It can't send anymore higher segments because the
> window is full.
>
> 16:41:29.011758 client 322905:324353(1448)
>
> Finally, the retransmit.
>
> 16:41:29.011919 server ack 327249 win 10224
>
> Now the server acknowledges up the the second gap.
>
> 16:41:29.021620 server ack 327249 win 13296
> ...
> 16:41:29.024911 server ack 327249 win 56304
>
> And keeps acknowledging the same, while the window grows again. But
> nothing from the client.
>
> 16:41:29.146218 192.168.0.40.58347 > home.manymonkeys.com.ftp: . ack 376
> win 65535 <nop,nop,timestamp 38 05665564 543875729> (DF) [tos 0x10]
>
> Odd, an acknowledgment on the FTP control connection just at this time.
> I don't know what that means, but it's not a coincidence, I'd bet.
>
> 16:41:30.462822 client 327249:328697(1448)
>
> Now the client finally retransmitted the segment from the second gap.
> But look at the timestamp, about 1.4 seconds have passed, which is much
> too long.
>
> After that, things go back to normal.
>
> The same thing repeats nine more times throughout the connection. You
> can spot the places by comparing subsequent timestamps. You'll find the
> spots where there are >1s delays in between two packets.
>
>> The tcpdump was run on the server (FBSD).  Later today I will gladly do
>> this again with a dump from each side.
>
> Before we blame the client, let's look at the dump from the client.
> Maybe the client is trying to retransmit earlier, but those packets get
> lost. It's just odd that this would happen for more than a full second.
>
> Daniel
>