[Bug 227760] Race condition in syncache_lookup and syncache_insert in TCP Handshake
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Wed Apr 25 10:32:29 UTC 2018
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227760
Bug ID: 227760
Summary: Race condition in syncache_lookup and syncache_insert
in TCP Handshake
Product: Base System
Version: CURRENT
Hardware: amd64
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: bugs at FreeBSD.org
Reporter: harsh at chelsio.com
Created attachment 192796
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=192796&action=edit
server_100.sh
Some of TCP connection Resets when we try to establish multiple connections
with Chelsio TOE Driver.
Test Setup Details:
2 Machines connected Back 2 Back with Chelsio T6 Adapter.
OS : FreeBSD-current.
Application Used: openssl s_time/s_server. Each s_server can handle single
client(s_time) connection.
Steps to Re-produce the issue.
Server:
1) kldload if_cxgbe
2) kldload t4_tom
3) ifconfig cc0 1.0.0.9/24 toe up
4) sh server_100.sh (file attached). It will start ~200 s_server connection.
Client:
1) kldload if_cxgbe
2) kldload t4_tom
3) ifconfig cc0 1.0.0.35 toe up
4) ./bash_client_100.sh(file attached). It will submit http GET request for
File transfer and repeats the same for given time interval.
Problem:
After some time some of connections will reset.
On debugging We observed that syncache_lookup() in syncache_expand() fails
while handling ack(3rd hadshake message) from client. On failure following code
snippet in syncache_expand sends reset
if (!V_tcp_syncookiesonly &&
sch->sch_last_overflow < time_uptime - SYNCOOKIE_LIFETIME)
{
SCH_UNLOCK(sch);
if ((s = tcp_log_addrs(inc, th, NULL, NULL)))
log(LOG_DEBUG, "%s; %s: Spurious ACK, "
"segment rejected (no syncache entry)\n",
s, __func__);
goto failed;
}
Detailed steps:
Client sends SYNC to server ==> 1 server in syncache_add calls
syncache_respond.
2 syncache_respond will send SYNC+ACK to client
<== SYNC+ACK
3 thread handling SYNC will context switch
because of that syncache_insert()will not get
a chance to add entry.
ACK ==> another thread in server will start handling
ack message. During syncache_expand lookup will
fail and RESET will be sent to client.
Following workarounds tried avoid this
1) Setting sysctl net.inet.tcp.syncookies_only=1 avoids this issue.
2) Following hack to make sure insert happens before sending SYNC+ACK also
works.
@@ -1558,17 +1575,23 @@ syncache_add(struct in_conninfo *inc, struct tcpopt
*to,
struct tcphdr *th,
--More--(byte 11894) /*
* Do a standard 3-way handshake.
*/
+ if (V_tcp_syncookies && V_tcp_syncookiesonly && sc != &scs) {
+ // compilation error
+ rv = rv +1;
+ rv = rv - 1;
+ }else if (sc != &scs)
+ syncache_insert(sc, sch); /* locks and unlocks sch */
+
if (syncache_respond(sc, sch, 0, m) == 0) {
if (V_tcp_syncookies && V_tcp_syncookiesonly && sc != &scs)
syncache_free(sc);
- else if (sc != &scs)
- syncache_insert(sc, sch); /* locks and unlocks sch */
- TCPSTAT_INC(tcps_sndacks);
- TCPSTAT_INC(tcps_sndtotal);
} else {
- if (sc != &scs)
- syncache_free(sc);
- TCPSTAT_INC(tcps_sc_dropped);
+ if (sc != &scs) {
+ SCH_LOCK(sch);
+ syncache_drop(sc, sch); /* locks and unlocks sch */
+ SCH_UNLOCK(sch);
+ TCPSTAT_INC(tcps_sc_dropped);
+ }
}
netstat -sp tcp at the time of issue
42824 syncache entries added
12 retransmitted
0 dupsyn
0 dropped
42820 completed
0 bucket overflow
0 cache overflow
0 reset
4 stale
0 aborted
0 badack
0 unreach
0 zone failures
dmesg log with "sysctl net.inet.tcp.log_debug=1"
TCP: [1.0.0.35]:46690 to [1.0.0.9]:1021 tcpflags 0x10<ACK>; syncache_expand:
Spurious ACK, segment rejected (no syncache ent ry)
TCP: [1.0.0.35]:46690 to [1.0.0.9]:1021; syncache_timer: Response timeout,
retransmitting (1) SYN|ACK
TCP: [1.0.0.35]:46690 to [1.0.0.9]:1021; syncache_timer: Response timeout,
retransmitting (2) SYN|ACK
TCP: [1.0.0.35]:46690 to [1.0.0.9]:1021; syncache_timer: Response timeout,
retransmitting (3) SYN|ACK
--
You are receiving this mail because:
You are the assignee for the bug.
More information about the freebsd-bugs
mailing list