tcp keepalive after fin+ack from client and server
Felix J. Ogris
fjo-lists at ogris.de
Mon Dec 14 00:08:08 UTC 2009
Hi,
I am experiencing some strange problem where FreeBSD sometimes starts
sending tcp keepalives after client and server have sent and ack'ed FINs.
The server runs 7.1-RELEASE/amd64 with open-vm-tools-nox11-148847 in a
VMware ESXi 4.0. The client runs a CentOS Linux 2.6.18-164.6.1.el5PAE SMP on
a bare metal machine. FreeBSD houses a Apache installation with sendfile and
mmap enabled. The Linux machine runs a homemade monitoring system and starts
a Perl script every 5 minutes to check if Apache is still alive. I have put
a tcpdump output on http://ogris.de/keepalive.txt for readability and can
provide the raw tcpdump file, if needed. Client and server keep sending
those keepalives for about 2 hours (yielding 300kB/s constantly) if not
stopped manually by an ipfw rule. lsof shows that no user process has open
the corresponding sockets anymore, whereas netstat shows established
connections.
FreeBSD has loaded ipfw with some keep-state rules, the Linux box has
iptables disabled.
kldstat:
Id Refs Address Size Name
1 7 0xffffffff80100000 b4be40 kernel
2 1 0xffffffff80c4c000 7d0 accf_data.ko
3 1 0xffffffff80c4d000 14d8 accf_http.ko
4 1 0xffffffffae531000 175a vmmemctl.ko
5 1 0xffffffffae543000 1e2e vmxnet.ko
6 1 0xffffffffae548000 9dd2 ipfw.ko
netstat -s -p tcp:
tcp:
952726388 packets sent
9941686 data packets (12846102403 bytes)
27667 data packets (37621159 bytes) retransmitted
844 data packets unnecessarily retransmitted
10 resends initiated by MTU discovery
942517629 ack-only packets (125912 delayed)
0 URG only packets
632 window probe packets
61151 window update packets
177623 control packets
949354853 packets received
6393724 acks (for 12784848708 bytes)
217997 duplicate acks
941774559 acks for unsent data
677499 packets (448005826 bytes) received in-sequence
179120 completely duplicate packets (1862276 bytes)
178 old duplicate packets
12 packets with some dup. data (6813 bytes duped)
4934 out-of-order packets (6499234 bytes)
0 packets (0 bytes) of data after window
0 window probes
92981 window update packets
1848 packets received after close
19 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
81 discarded due to memory problems
24262 connection requests
152765 connection accepts
0 bad connection attempts
0 listen queue overflows
21854 ignored RSTs in the windows
176860 connections established (including accepts)
179292 connections closed (including 23036 drops)
111998 connections updated cached RTT on close
112122 connections updated cached RTT variance on close
43123 connections updated cached ssthresh on close
58 embryonic connections dropped
5513567 segments updated rtt (of 3368553 attempts)
17054 retransmit timeouts
751 connections dropped by rexmit timeout
803 persist timeouts
3 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
2008 keepalive timeouts
1786 keepalive probes sent
222 connections dropped by keepalive
1234887 correct ACK header predictions
434353 correct data packet header predictions
152809 syncache entries added
738 retransmitted
506 dupsyn
0 dropped
152765 completed
0 bucket overflow
0 cache overflow
144 reset
165 stale
0 aborted
0 badack
0 unreach
0 zone failures
152809 cookies sent
265 cookies received
6416 SACK recovery episodes
12522 segment rexmits in SACK recovery episodes
17837785 byte rexmits in SACK recovery episodes
62528 SACK options (SACK blocks) received
3807 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
sysctl net:
net.local.stream.recvspace: 8192
net.local.stream.sendspace: 8192
net.local.dgram.recvspace: 4096
net.local.dgram.maxdgram: 2048
net.local.recycled: 0
net.local.taskcount: 0
net.local.inflight: 0
net.inet.ip.portrange.randomtime: 45
net.inet.ip.portrange.randomcps: 10
net.inet.ip.portrange.randomized: 1
net.inet.ip.portrange.reservedlow: 0
net.inet.ip.portrange.reservedhigh: 1023
net.inet.ip.portrange.hilast: 65535
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.forwarding: 0
net.inet.ip.redirect: 1
net.inet.ip.ttl: 64
net.inet.ip.rtexpire: 3600
net.inet.ip.rtminexpire: 10
net.inet.ip.rtmaxcache: 128
net.inet.ip.sourceroute: 0
net.inet.ip.intr_queue_maxlen: 50
net.inet.ip.intr_queue_drops: 0
net.inet.ip.accept_sourceroute: 0
net.inet.ip.keepfaith: 0
net.inet.ip.gifttl: 30
net.inet.ip.same_prefix_carp_only: 0
net.inet.ip.subnets_are_local: 0
net.inet.ip.fastforwarding: 0
net.inet.ip.maxfragpackets: 1024
net.inet.ip.maxfragsperpacket: 16
net.inet.ip.fragpackets: 0
net.inet.ip.check_interface: 0
net.inet.ip.random_id: 0
net.inet.ip.sendsourcequench: 0
net.inet.ip.process_options: 1
net.inet.ip.fw.dyn_keepalive: 1
net.inet.ip.fw.dyn_short_lifetime: 5
net.inet.ip.fw.dyn_udp_lifetime: 10
net.inet.ip.fw.dyn_rst_lifetime: 1
net.inet.ip.fw.dyn_fin_lifetime: 1
net.inet.ip.fw.dyn_syn_lifetime: 20
net.inet.ip.fw.dyn_ack_lifetime: 300
net.inet.ip.fw.static_count: 22
net.inet.ip.fw.dyn_max: 4096
net.inet.ip.fw.dyn_count: 1038
net.inet.ip.fw.curr_dyn_buckets: 1024
net.inet.ip.fw.dyn_buckets: 1024
net.inet.ip.fw.default_rule: 65535
net.inet.ip.fw.verbose_limit: 0
net.inet.ip.fw.verbose: 0
net.inet.ip.fw.debug: 1
net.inet.ip.fw.one_pass: 1
net.inet.ip.fw.autoinc_step: 100
net.inet.ip.fw.enable: 1
net.inet.icmp.maskrepl: 0
net.inet.icmp.icmplim: 200
net.inet.icmp.bmcastecho: 0
net.inet.icmp.quotelen: 8
net.inet.icmp.reply_from_interface: 0
net.inet.icmp.reply_src:
net.inet.icmp.icmplim_output: 1
net.inet.icmp.log_redirect: 0
net.inet.icmp.drop_redirect: 0
net.inet.icmp.maskfake: 0
net.inet.tcp.rfc1323: 1
net.inet.tcp.mssdflt: 512
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 65536
net.inet.tcp.keepinit: 75000
net.inet.tcp.delacktime: 100
net.inet.tcp.v6mssdflt: 1024
net.inet.tcp.hostcache.purge: 0
net.inet.tcp.hostcache.prune: 300
net.inet.tcp.hostcache.expire: 3600
net.inet.tcp.hostcache.count: 116
net.inet.tcp.hostcache.bucketlimit: 30
net.inet.tcp.hostcache.hashsize: 512
net.inet.tcp.hostcache.cachelimit: 15360
net.inet.tcp.recvbuf_max: 262144
net.inet.tcp.recvbuf_inc: 16384
net.inet.tcp.recvbuf_auto: 1
net.inet.tcp.insecure_rst: 0
net.inet.tcp.rfc3390: 1
net.inet.tcp.rfc3042: 1
net.inet.tcp.drop_synfin: 0
net.inet.tcp.delayed_ack: 1
net.inet.tcp.blackhole: 0
net.inet.tcp.log_in_vain: 0
net.inet.tcp.sendbuf_max: 262144
net.inet.tcp.sendbuf_inc: 8192
net.inet.tcp.sendbuf_auto: 1
net.inet.tcp.tso: 1
net.inet.tcp.newreno: 1
net.inet.tcp.local_slowstart_flightsize: 4
net.inet.tcp.slowstart_flightsize: 1
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.reass.overflows: 81
net.inet.tcp.reass.maxqlen: 48
net.inet.tcp.reass.cursegments: 0
net.inet.tcp.reass.maxsegments: 2048
net.inet.tcp.sack.globalholes: 0
net.inet.tcp.sack.globalmaxholes: 65536
net.inet.tcp.sack.maxholes: 128
net.inet.tcp.sack.enable: 1
net.inet.tcp.inflight.stab: 20
net.inet.tcp.inflight.max: 1073725440
net.inet.tcp.inflight.min: 6144
net.inet.tcp.inflight.rttthresh: 10
net.inet.tcp.inflight.debug: 0
net.inet.tcp.inflight.enable: 1
net.inet.tcp.isn_reseed_interval: 0
net.inet.tcp.icmp_may_rst: 1
net.inet.tcp.pcbcount: 168
net.inet.tcp.do_tcpdrain: 1
net.inet.tcp.tcbhashsize: 512
net.inet.tcp.log_debug: 0
net.inet.tcp.minmss: 216
net.inet.tcp.syncache.rst_on_sock_fail: 1
net.inet.tcp.syncache.rexmtlimit: 3
net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.count: 0
net.inet.tcp.syncache.cachelimit: 15360
net.inet.tcp.syncache.bucketlimit: 30
net.inet.tcp.syncookies_only: 0
net.inet.tcp.syncookies: 1
net.inet.tcp.timer_race: 0
net.inet.tcp.finwait2_timeout: 60000
net.inet.tcp.fast_finwait2_recycle: 0
net.inet.tcp.always_keepalive: 1
net.inet.tcp.rexmit_slop: 200
net.inet.tcp.rexmit_min: 30
net.inet.tcp.msl: 30000
net.inet.tcp.nolocaltimewait: 0
net.inet.tcp.maxtcptw: 6553
net.inet.udp.checksum: 1
net.inet.udp.maxdgram: 9216
net.inet.udp.recvspace: 42080
net.inet.udp.soreceive_dgram_enabled: 0
net.inet.udp.blackhole: 0
net.inet.udp.log_in_vain: 0
net.inet.sctp.enable_sack_immediately: 0
net.inet.sctp.udp_tunneling_port: 0
net.inet.sctp.udp_tunneling_for_client_enable: 0
net.inet.sctp.mobility_fasthandoff: 0
net.inet.sctp.mobility_base: 0
net.inet.sctp.default_frag_interleave: 1
net.inet.sctp.default_cc_module: 0
net.inet.sctp.log_level: 0
net.inet.sctp.max_retran_chunk: 30
net.inet.sctp.min_residual: 1452
net.inet.sctp.strict_data_order: 0
net.inet.sctp.abort_at_limit: 0
net.inet.sctp.hb_max_burst: 4
net.inet.sctp.do_sctp_drain: 1
net.inet.sctp.max_chained_mbufs: 5
net.inet.sctp.abc_l_var: 1
net.inet.sctp.nat_friendly: 1
net.inet.sctp.auth_disable: 0
net.inet.sctp.asconf_auth_nochk: 0
net.inet.sctp.early_fast_retran_msec: 250
net.inet.sctp.early_fast_retran: 0
net.inet.sctp.cwnd_maxburst: 1
net.inet.sctp.cmt_pf: 0
net.inet.sctp.cmt_use_dac: 0
net.inet.sctp.cmt_on_off: 0
net.inet.sctp.outgoing_streams: 10
net.inet.sctp.add_more_on_output: 1452
net.inet.sctp.path_rtx_max: 5
net.inet.sctp.assoc_rtx_max: 10
net.inet.sctp.init_rtx_max: 8
net.inet.sctp.valid_cookie_life: 60000
net.inet.sctp.init_rto_max: 60000
net.inet.sctp.rto_initial: 3000
net.inet.sctp.rto_min: 1000
net.inet.sctp.rto_max: 60000
net.inet.sctp.secret_lifetime: 3600
net.inet.sctp.shutdown_guard_time: 180
net.inet.sctp.pmtu_raise_time: 600
net.inet.sctp.heartbeat_interval: 30000
net.inet.sctp.asoc_resource: 10
net.inet.sctp.sys_resource: 1000
net.inet.sctp.sack_freq: 2
net.inet.sctp.delayed_sack_time: 200
net.inet.sctp.chunkscale: 10
net.inet.sctp.min_split_point: 2904
net.inet.sctp.pcbhashsize: 256
net.inet.sctp.tcbhashsize: 1024
net.inet.sctp.maxchunks: 4096
net.inet.sctp.maxburst: 4
net.inet.sctp.peer_chkoh: 256
net.inet.sctp.strict_init: 1
net.inet.sctp.loopback_nocsum: 1
net.inet.sctp.strict_sacks: 0
net.inet.sctp.ecn_nonce: 0
net.inet.sctp.ecn_enable: 1
net.inet.sctp.auto_asconf: 1
net.inet.sctp.recvspace: 233016
net.inet.sctp.sendspace: 233016
net.inet.raw.recvspace: 9216
net.inet.raw.maxdgram: 9216
net.inet.accf.unloadable: 0
net.inet.accf.http.parsehttpversion: 1
net.link.generic.system.ifcount: 3
net.link.ether.inet.log_arp_permanent_modify: 1
net.link.ether.inet.log_arp_movements: 1
net.link.ether.inet.log_arp_wrong_iface: 1
net.link.ether.inet.proxyall: 0
net.link.ether.inet.useloopback: 1
net.link.ether.inet.maxtries: 5
net.link.ether.inet.max_age: 1200
net.link.ether.ipfw: 0
net.link.gif.parallel_tunnels: 0
net.link.gif.max_nesting: 1
net.link.log_link_state_change: 1
net.link.tun.devfs_cloning: 1
net.inet6.ip6.forwarding: 0
net.inet6.ip6.redirect: 1
net.inet6.ip6.hlim: 64
net.inet6.ip6.maxfragpackets: 8192
net.inet6.ip6.accept_rtadv: 0
net.inet6.ip6.keepfaith: 0
net.inet6.ip6.log_interval: 5
net.inet6.ip6.hdrnestlimit: 15
net.inet6.ip6.dad_count: 1
net.inet6.ip6.auto_flowlabel: 1
net.inet6.ip6.defmcasthlim: 1
net.inet6.ip6.gifhlim: 30
net.inet6.ip6.kame_version: FreeBSD
net.inet6.ip6.use_deprecated: 1
net.inet6.ip6.rr_prune: 5
net.inet6.ip6.v6only: 1
net.inet6.ip6.rtexpire: 3600
net.inet6.ip6.rtminexpire: 10
net.inet6.ip6.rtmaxcache: 128
net.inet6.ip6.use_tempaddr: 0
net.inet6.ip6.temppltime: 86400
net.inet6.ip6.tempvltime: 604800
net.inet6.ip6.auto_linklocal: 0
net.inet6.ip6.prefer_tempaddr: 0
net.inet6.ip6.use_defaultzone: 0
net.inet6.ip6.maxfrags: 8192
net.inet6.ip6.mcast_pmtu: 0
net.inet6.ip6.fw.enable: 1
net.inet6.ip6.fw.deny_unknown_exthdrs: 1
net.inet6.icmp6.rediraccept: 1
net.inet6.icmp6.redirtimeout: 600
net.inet6.icmp6.nd6_prune: 1
net.inet6.icmp6.nd6_delay: 5
net.inet6.icmp6.nd6_umaxtries: 3
net.inet6.icmp6.nd6_mmaxtries: 3
net.inet6.icmp6.nd6_useloopback: 1
net.inet6.icmp6.nodeinfo: 3
net.inet6.icmp6.errppslimit: 100
net.inet6.icmp6.nd6_maxnudhint: 0
net.inet6.icmp6.nd6_debug: 0
net.inet6.icmp6.nd6_maxqueuelen: 1
net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0
net.bpf.maxinsns: 512
net.bpf.maxbufsize: 524288
net.bpf.bufsize: 4096
net.isr.swi_count: 751963064
net.isr.drop: 0
net.isr.queued: 952769795
net.isr.deferred: 0
net.isr.directed: 952250693
net.isr.count: 952250693
net.isr.direct: 1
net.raw.recvspace: 8192
net.raw.sendspace: 8192
net.my_fibnum: 0
net.add_addr_allfibs: 1
net.fibs: 1
net.route.netisr_maxqlen: 256
net.wlan.recv_bar: 1
net.wlan.debug: 0
sysctl kern.ipc:
kern.ipc.maxsockbuf: 262144
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.somaxconn: 4096
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 60
kern.ipc.max_hdr: 76
kern.ipc.max_datalen: 100
kern.ipc.nmbjumbo16: 3200
kern.ipc.nmbjumbo9: 6400
kern.ipc.nmbjumbop: 12800
kern.ipc.nmbclusters: 32768
kern.ipc.piperesizeallowed: 1
kern.ipc.piperesizefail: 0
kern.ipc.pipeallocfail: 0
kern.ipc.pipefragretry: 0
kern.ipc.pipekva: 147456
kern.ipc.maxpipekva: 20971520
kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.semaem: 16384
kern.ipc.semvmx: 32767
kern.ipc.semusz: 104
kern.ipc.semume: 10
kern.ipc.semopm: 100
kern.ipc.semmsl: 60
kern.ipc.semmnu: 512
kern.ipc.semmns: 1024
kern.ipc.semmni: 512
kern.ipc.semmap: 256
kern.ipc.shm_allow_removed: 0
kern.ipc.shm_use_phys: 0
kern.ipc.shmall: 32768
kern.ipc.shmseg: 128
kern.ipc.shmmni: 192
kern.ipc.shmmin: 1
kern.ipc.shmmax: 134217728
kern.ipc.maxsockets: 32768
kern.ipc.numopensockets: 188
kern.ipc.nsfbufsused: 0
kern.ipc.nsfbufspeak: 0
kern.ipc.nsfbufs: 0
TIA,
Felix
More information about the freebsd-net
mailing list