(TCP/IP) Server side sends RST after 3-way handshake.Syn flood defense or queue overflow?

[John Baldwin]

On Wednesday, July 13, 2011 8:37:24 am Vladimir Budnev wrote:
> statistics i have:
> (cutted from netstat -s -p tcp)
>     89514 syncache entries added
>         0 retransmitted
>         0 dupsyn
>         0 dropped
>         50220 completed
>         0 bucket overflow
>         0 cache overflow
>         0 reset
>         0 stale
>         39294 aborted
>         0 badack
>         0 unreach
>         0 zone failures
>     89514 cookies sent
> According to your hint I see nonzero aborted field, but dont know how to get
> the reason/interpret this.

Yep, the aborted entries are likely your resets.

> > However, if your app just sends data and calls close() without doing any
> > reads, it might close() succesfully while the data is in flight before the
> > client machine sees the RST, so the client app will not see any errors.  If
> > the RST arrives before you finish calling write() and close() then you will
> > get ECONNRESET errors from write() and close().
> >
> 
> Hm..hope i get your idea, the difference is that client application recieve
> ECONNRESET as result to _connect_ call not as result from write/read/close,
> at that point i cant ignore it. I can spin/wait and try again, but that way
> every N iterations when problem occures there will be this "problem". As i
> mentioned(and in code example) client on every iteration makes
> connect/close, and those come very fast.
> Mb i misunderstood smth.

Ah, when I was tracking down a similar issue I was only seeing errors on write
(or sometimes not at all) not on connect().   A bit odd you are getting errors
on connect().  Maybe the RST is arriving so fast the kernel notices it and
marks the socket error before the process asleep in connect() gets a chance to
run after it is awakened by the SYN/ACK processing.

> You can try turning off the syncache and syncookies as a test.  This will
> > probably trigger more ECONNRESET errors in connect() (which your app will
> > need
> > to retry on).
> 
> Spin idea...yeah it sounds friendly but i dont think sy/syn-ack/ack/rst
> flood would be great:(

Yes, fixing it for real would be better.

> >  However, the better fix is to track down what is causing your
> > connections to be dropped in the first place, e.g. if you are hitting the
> > limit on inpcbs (look for failures in vmstat -z output) and fix that.
> >
> # vmstat -z
> ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS
> FAILURES
> <...>
> inpcb:                    288,    25610,        6,     5155,
> 142849,        0
> tcpcb:                    728,    25600,        6,      454,
> 142849,        0
> <...>
> Those ones looks ok?

Hmm, perhaps compare the 'kern.ipc.numopensockets' and 'kern.ipc.maxsockets' tunables under load on the server?

-- 
John Baldwin