tcp keepalive after fin+ack from client and server
Julian Elischer
julian at elischer.org
Mon Dec 14 04:32:39 UTC 2009
Felix J. Ogris wrote:
> Hi,
>
> I am experiencing some strange problem where FreeBSD sometimes starts
> sending tcp keepalives after client and server have sent and ack'ed FINs.
> The server runs 7.1-RELEASE/amd64 with open-vm-tools-nox11-148847 in a
> VMware ESXi 4.0. The client runs a CentOS Linux 2.6.18-164.6.1.el5PAE SMP on
> a bare metal machine. FreeBSD houses a Apache installation with sendfile and
> mmap enabled. The Linux machine runs a homemade monitoring system and starts
> a Perl script every 5 minutes to check if Apache is still alive. I have put
> a tcpdump output on http://ogris.de/keepalive.txt for readability and can
> provide the raw tcpdump file, if needed. Client and server keep sending
> those keepalives for about 2 hours (yielding 300kB/s constantly) if not
> stopped manually by an ipfw rule. lsof shows that no user process has open
> the corresponding sockets anymore, whereas netstat shows established
> connections.
> FreeBSD has loaded ipfw with some keep-state rules, the Linux box has
> iptables disabled.
are you sure it isn't the firewall (ipfw) sending keepalives? it is
one of the options with kept state to inject keepalives.
if it didint' see all the FINs for some reason, it may think the
session is still alive.
>
> kldstat:
> Id Refs Address Size Name
> 1 7 0xffffffff80100000 b4be40 kernel
> 2 1 0xffffffff80c4c000 7d0 accf_data.ko
> 3 1 0xffffffff80c4d000 14d8 accf_http.ko
> 4 1 0xffffffffae531000 175a vmmemctl.ko
> 5 1 0xffffffffae543000 1e2e vmxnet.ko
> 6 1 0xffffffffae548000 9dd2 ipfw.ko
>
> netstat -s -p tcp:
> tcp:
> 952726388 packets sent
> 9941686 data packets (12846102403 bytes)
> 27667 data packets (37621159 bytes) retransmitted
> 844 data packets unnecessarily retransmitted
> 10 resends initiated by MTU discovery
> 942517629 ack-only packets (125912 delayed)
> 0 URG only packets
> 632 window probe packets
> 61151 window update packets
> 177623 control packets
> 949354853 packets received
> 6393724 acks (for 12784848708 bytes)
> 217997 duplicate acks
> 941774559 acks for unsent data
> 677499 packets (448005826 bytes) received in-sequence
> 179120 completely duplicate packets (1862276 bytes)
> 178 old duplicate packets
> 12 packets with some dup. data (6813 bytes duped)
> 4934 out-of-order packets (6499234 bytes)
> 0 packets (0 bytes) of data after window
> 0 window probes
> 92981 window update packets
> 1848 packets received after close
> 19 discarded for bad checksums
> 0 discarded for bad header offset fields
> 0 discarded because packet too short
> 81 discarded due to memory problems
> 24262 connection requests
> 152765 connection accepts
> 0 bad connection attempts
> 0 listen queue overflows
> 21854 ignored RSTs in the windows
> 176860 connections established (including accepts)
> 179292 connections closed (including 23036 drops)
> 111998 connections updated cached RTT on close
> 112122 connections updated cached RTT variance on close
> 43123 connections updated cached ssthresh on close
> 58 embryonic connections dropped
> 5513567 segments updated rtt (of 3368553 attempts)
> 17054 retransmit timeouts
> 751 connections dropped by rexmit timeout
> 803 persist timeouts
> 3 connections dropped by persist timeout
> 0 Connections (fin_wait_2) dropped because of timeout
> 2008 keepalive timeouts
> 1786 keepalive probes sent
> 222 connections dropped by keepalive
> 1234887 correct ACK header predictions
> 434353 correct data packet header predictions
> 152809 syncache entries added
> 738 retransmitted
> 506 dupsyn
> 0 dropped
> 152765 completed
> 0 bucket overflow
> 0 cache overflow
> 144 reset
> 165 stale
> 0 aborted
> 0 badack
> 0 unreach
> 0 zone failures
> 152809 cookies sent
> 265 cookies received
> 6416 SACK recovery episodes
> 12522 segment rexmits in SACK recovery episodes
> 17837785 byte rexmits in SACK recovery episodes
> 62528 SACK options (SACK blocks) received
> 3807 SACK options (SACK blocks) sent
> 0 SACK scoreboard overflow
>
> sysctl net:
> net.local.stream.recvspace: 8192
> net.local.stream.sendspace: 8192
> net.local.dgram.recvspace: 4096
> net.local.dgram.maxdgram: 2048
> net.local.recycled: 0
> net.local.taskcount: 0
> net.local.inflight: 0
> net.inet.ip.portrange.randomtime: 45
> net.inet.ip.portrange.randomcps: 10
> net.inet.ip.portrange.randomized: 1
> net.inet.ip.portrange.reservedlow: 0
> net.inet.ip.portrange.reservedhigh: 1023
> net.inet.ip.portrange.hilast: 65535
> net.inet.ip.portrange.hifirst: 49152
> net.inet.ip.portrange.last: 65535
> net.inet.ip.portrange.first: 49152
> net.inet.ip.portrange.lowlast: 600
> net.inet.ip.portrange.lowfirst: 1023
> net.inet.ip.forwarding: 0
> net.inet.ip.redirect: 1
> net.inet.ip.ttl: 64
> net.inet.ip.rtexpire: 3600
> net.inet.ip.rtminexpire: 10
> net.inet.ip.rtmaxcache: 128
> net.inet.ip.sourceroute: 0
> net.inet.ip.intr_queue_maxlen: 50
> net.inet.ip.intr_queue_drops: 0
> net.inet.ip.accept_sourceroute: 0
> net.inet.ip.keepfaith: 0
> net.inet.ip.gifttl: 30
> net.inet.ip.same_prefix_carp_only: 0
> net.inet.ip.subnets_are_local: 0
> net.inet.ip.fastforwarding: 0
> net.inet.ip.maxfragpackets: 1024
> net.inet.ip.maxfragsperpacket: 16
> net.inet.ip.fragpackets: 0
> net.inet.ip.check_interface: 0
> net.inet.ip.random_id: 0
> net.inet.ip.sendsourcequench: 0
> net.inet.ip.process_options: 1
> net.inet.ip.fw.dyn_keepalive: 1
> net.inet.ip.fw.dyn_short_lifetime: 5
> net.inet.ip.fw.dyn_udp_lifetime: 10
> net.inet.ip.fw.dyn_rst_lifetime: 1
> net.inet.ip.fw.dyn_fin_lifetime: 1
> net.inet.ip.fw.dyn_syn_lifetime: 20
> net.inet.ip.fw.dyn_ack_lifetime: 300
> net.inet.ip.fw.static_count: 22
> net.inet.ip.fw.dyn_max: 4096
> net.inet.ip.fw.dyn_count: 1038
> net.inet.ip.fw.curr_dyn_buckets: 1024
> net.inet.ip.fw.dyn_buckets: 1024
> net.inet.ip.fw.default_rule: 65535
> net.inet.ip.fw.verbose_limit: 0
> net.inet.ip.fw.verbose: 0
> net.inet.ip.fw.debug: 1
> net.inet.ip.fw.one_pass: 1
> net.inet.ip.fw.autoinc_step: 100
> net.inet.ip.fw.enable: 1
> net.inet.icmp.maskrepl: 0
> net.inet.icmp.icmplim: 200
> net.inet.icmp.bmcastecho: 0
> net.inet.icmp.quotelen: 8
> net.inet.icmp.reply_from_interface: 0
> net.inet.icmp.reply_src:
> net.inet.icmp.icmplim_output: 1
> net.inet.icmp.log_redirect: 0
> net.inet.icmp.drop_redirect: 0
> net.inet.icmp.maskfake: 0
> net.inet.tcp.rfc1323: 1
> net.inet.tcp.mssdflt: 512
> net.inet.tcp.keepidle: 7200000
> net.inet.tcp.keepintvl: 75000
> net.inet.tcp.sendspace: 65536
> net.inet.tcp.recvspace: 65536
> net.inet.tcp.keepinit: 75000
> net.inet.tcp.delacktime: 100
> net.inet.tcp.v6mssdflt: 1024
> net.inet.tcp.hostcache.purge: 0
> net.inet.tcp.hostcache.prune: 300
> net.inet.tcp.hostcache.expire: 3600
> net.inet.tcp.hostcache.count: 116
> net.inet.tcp.hostcache.bucketlimit: 30
> net.inet.tcp.hostcache.hashsize: 512
> net.inet.tcp.hostcache.cachelimit: 15360
> net.inet.tcp.recvbuf_max: 262144
> net.inet.tcp.recvbuf_inc: 16384
> net.inet.tcp.recvbuf_auto: 1
> net.inet.tcp.insecure_rst: 0
> net.inet.tcp.rfc3390: 1
> net.inet.tcp.rfc3042: 1
> net.inet.tcp.drop_synfin: 0
> net.inet.tcp.delayed_ack: 1
> net.inet.tcp.blackhole: 0
> net.inet.tcp.log_in_vain: 0
> net.inet.tcp.sendbuf_max: 262144
> net.inet.tcp.sendbuf_inc: 8192
> net.inet.tcp.sendbuf_auto: 1
> net.inet.tcp.tso: 1
> net.inet.tcp.newreno: 1
> net.inet.tcp.local_slowstart_flightsize: 4
> net.inet.tcp.slowstart_flightsize: 1
> net.inet.tcp.path_mtu_discovery: 1
> net.inet.tcp.reass.overflows: 81
> net.inet.tcp.reass.maxqlen: 48
> net.inet.tcp.reass.cursegments: 0
> net.inet.tcp.reass.maxsegments: 2048
> net.inet.tcp.sack.globalholes: 0
> net.inet.tcp.sack.globalmaxholes: 65536
> net.inet.tcp.sack.maxholes: 128
> net.inet.tcp.sack.enable: 1
> net.inet.tcp.inflight.stab: 20
> net.inet.tcp.inflight.max: 1073725440
> net.inet.tcp.inflight.min: 6144
> net.inet.tcp.inflight.rttthresh: 10
> net.inet.tcp.inflight.debug: 0
> net.inet.tcp.inflight.enable: 1
> net.inet.tcp.isn_reseed_interval: 0
> net.inet.tcp.icmp_may_rst: 1
> net.inet.tcp.pcbcount: 168
> net.inet.tcp.do_tcpdrain: 1
> net.inet.tcp.tcbhashsize: 512
> net.inet.tcp.log_debug: 0
> net.inet.tcp.minmss: 216
> net.inet.tcp.syncache.rst_on_sock_fail: 1
> net.inet.tcp.syncache.rexmtlimit: 3
> net.inet.tcp.syncache.hashsize: 512
> net.inet.tcp.syncache.count: 0
> net.inet.tcp.syncache.cachelimit: 15360
> net.inet.tcp.syncache.bucketlimit: 30
> net.inet.tcp.syncookies_only: 0
> net.inet.tcp.syncookies: 1
> net.inet.tcp.timer_race: 0
> net.inet.tcp.finwait2_timeout: 60000
> net.inet.tcp.fast_finwait2_recycle: 0
> net.inet.tcp.always_keepalive: 1
> net.inet.tcp.rexmit_slop: 200
> net.inet.tcp.rexmit_min: 30
> net.inet.tcp.msl: 30000
> net.inet.tcp.nolocaltimewait: 0
> net.inet.tcp.maxtcptw: 6553
> net.inet.udp.checksum: 1
> net.inet.udp.maxdgram: 9216
> net.inet.udp.recvspace: 42080
> net.inet.udp.soreceive_dgram_enabled: 0
> net.inet.udp.blackhole: 0
> net.inet.udp.log_in_vain: 0
> net.inet.sctp.enable_sack_immediately: 0
> net.inet.sctp.udp_tunneling_port: 0
> net.inet.sctp.udp_tunneling_for_client_enable: 0
> net.inet.sctp.mobility_fasthandoff: 0
> net.inet.sctp.mobility_base: 0
> net.inet.sctp.default_frag_interleave: 1
> net.inet.sctp.default_cc_module: 0
> net.inet.sctp.log_level: 0
> net.inet.sctp.max_retran_chunk: 30
> net.inet.sctp.min_residual: 1452
> net.inet.sctp.strict_data_order: 0
> net.inet.sctp.abort_at_limit: 0
> net.inet.sctp.hb_max_burst: 4
> net.inet.sctp.do_sctp_drain: 1
> net.inet.sctp.max_chained_mbufs: 5
> net.inet.sctp.abc_l_var: 1
> net.inet.sctp.nat_friendly: 1
> net.inet.sctp.auth_disable: 0
> net.inet.sctp.asconf_auth_nochk: 0
> net.inet.sctp.early_fast_retran_msec: 250
> net.inet.sctp.early_fast_retran: 0
> net.inet.sctp.cwnd_maxburst: 1
> net.inet.sctp.cmt_pf: 0
> net.inet.sctp.cmt_use_dac: 0
> net.inet.sctp.cmt_on_off: 0
> net.inet.sctp.outgoing_streams: 10
> net.inet.sctp.add_more_on_output: 1452
> net.inet.sctp.path_rtx_max: 5
> net.inet.sctp.assoc_rtx_max: 10
> net.inet.sctp.init_rtx_max: 8
> net.inet.sctp.valid_cookie_life: 60000
> net.inet.sctp.init_rto_max: 60000
> net.inet.sctp.rto_initial: 3000
> net.inet.sctp.rto_min: 1000
> net.inet.sctp.rto_max: 60000
> net.inet.sctp.secret_lifetime: 3600
> net.inet.sctp.shutdown_guard_time: 180
> net.inet.sctp.pmtu_raise_time: 600
> net.inet.sctp.heartbeat_interval: 30000
> net.inet.sctp.asoc_resource: 10
> net.inet.sctp.sys_resource: 1000
> net.inet.sctp.sack_freq: 2
> net.inet.sctp.delayed_sack_time: 200
> net.inet.sctp.chunkscale: 10
> net.inet.sctp.min_split_point: 2904
> net.inet.sctp.pcbhashsize: 256
> net.inet.sctp.tcbhashsize: 1024
> net.inet.sctp.maxchunks: 4096
> net.inet.sctp.maxburst: 4
> net.inet.sctp.peer_chkoh: 256
> net.inet.sctp.strict_init: 1
> net.inet.sctp.loopback_nocsum: 1
> net.inet.sctp.strict_sacks: 0
> net.inet.sctp.ecn_nonce: 0
> net.inet.sctp.ecn_enable: 1
> net.inet.sctp.auto_asconf: 1
> net.inet.sctp.recvspace: 233016
> net.inet.sctp.sendspace: 233016
> net.inet.raw.recvspace: 9216
> net.inet.raw.maxdgram: 9216
> net.inet.accf.unloadable: 0
> net.inet.accf.http.parsehttpversion: 1
> net.link.generic.system.ifcount: 3
> net.link.ether.inet.log_arp_permanent_modify: 1
> net.link.ether.inet.log_arp_movements: 1
> net.link.ether.inet.log_arp_wrong_iface: 1
> net.link.ether.inet.proxyall: 0
> net.link.ether.inet.useloopback: 1
> net.link.ether.inet.maxtries: 5
> net.link.ether.inet.max_age: 1200
> net.link.ether.ipfw: 0
> net.link.gif.parallel_tunnels: 0
> net.link.gif.max_nesting: 1
> net.link.log_link_state_change: 1
> net.link.tun.devfs_cloning: 1
> net.inet6.ip6.forwarding: 0
> net.inet6.ip6.redirect: 1
> net.inet6.ip6.hlim: 64
> net.inet6.ip6.maxfragpackets: 8192
> net.inet6.ip6.accept_rtadv: 0
> net.inet6.ip6.keepfaith: 0
> net.inet6.ip6.log_interval: 5
> net.inet6.ip6.hdrnestlimit: 15
> net.inet6.ip6.dad_count: 1
> net.inet6.ip6.auto_flowlabel: 1
> net.inet6.ip6.defmcasthlim: 1
> net.inet6.ip6.gifhlim: 30
> net.inet6.ip6.kame_version: FreeBSD
> net.inet6.ip6.use_deprecated: 1
> net.inet6.ip6.rr_prune: 5
> net.inet6.ip6.v6only: 1
> net.inet6.ip6.rtexpire: 3600
> net.inet6.ip6.rtminexpire: 10
> net.inet6.ip6.rtmaxcache: 128
> net.inet6.ip6.use_tempaddr: 0
> net.inet6.ip6.temppltime: 86400
> net.inet6.ip6.tempvltime: 604800
> net.inet6.ip6.auto_linklocal: 0
> net.inet6.ip6.prefer_tempaddr: 0
> net.inet6.ip6.use_defaultzone: 0
> net.inet6.ip6.maxfrags: 8192
> net.inet6.ip6.mcast_pmtu: 0
> net.inet6.ip6.fw.enable: 1
> net.inet6.ip6.fw.deny_unknown_exthdrs: 1
> net.inet6.icmp6.rediraccept: 1
> net.inet6.icmp6.redirtimeout: 600
> net.inet6.icmp6.nd6_prune: 1
> net.inet6.icmp6.nd6_delay: 5
> net.inet6.icmp6.nd6_umaxtries: 3
> net.inet6.icmp6.nd6_mmaxtries: 3
> net.inet6.icmp6.nd6_useloopback: 1
> net.inet6.icmp6.nodeinfo: 3
> net.inet6.icmp6.errppslimit: 100
> net.inet6.icmp6.nd6_maxnudhint: 0
> net.inet6.icmp6.nd6_debug: 0
> net.inet6.icmp6.nd6_maxqueuelen: 1
> net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0
> net.bpf.maxinsns: 512
> net.bpf.maxbufsize: 524288
> net.bpf.bufsize: 4096
> net.isr.swi_count: 751963064
> net.isr.drop: 0
> net.isr.queued: 952769795
> net.isr.deferred: 0
> net.isr.directed: 952250693
> net.isr.count: 952250693
> net.isr.direct: 1
> net.raw.recvspace: 8192
> net.raw.sendspace: 8192
> net.my_fibnum: 0
> net.add_addr_allfibs: 1
> net.fibs: 1
> net.route.netisr_maxqlen: 256
> net.wlan.recv_bar: 1
> net.wlan.debug: 0
>
> sysctl kern.ipc:
> kern.ipc.maxsockbuf: 262144
> kern.ipc.sockbuf_waste_factor: 8
> kern.ipc.somaxconn: 4096
> kern.ipc.max_linkhdr: 16
> kern.ipc.max_protohdr: 60
> kern.ipc.max_hdr: 76
> kern.ipc.max_datalen: 100
> kern.ipc.nmbjumbo16: 3200
> kern.ipc.nmbjumbo9: 6400
> kern.ipc.nmbjumbop: 12800
> kern.ipc.nmbclusters: 32768
> kern.ipc.piperesizeallowed: 1
> kern.ipc.piperesizefail: 0
> kern.ipc.pipeallocfail: 0
> kern.ipc.pipefragretry: 0
> kern.ipc.pipekva: 147456
> kern.ipc.maxpipekva: 20971520
> kern.ipc.msgseg: 2048
> kern.ipc.msgssz: 8
> kern.ipc.msgtql: 40
> kern.ipc.msgmnb: 2048
> kern.ipc.msgmni: 40
> kern.ipc.msgmax: 16384
> kern.ipc.semaem: 16384
> kern.ipc.semvmx: 32767
> kern.ipc.semusz: 104
> kern.ipc.semume: 10
> kern.ipc.semopm: 100
> kern.ipc.semmsl: 60
> kern.ipc.semmnu: 512
> kern.ipc.semmns: 1024
> kern.ipc.semmni: 512
> kern.ipc.semmap: 256
> kern.ipc.shm_allow_removed: 0
> kern.ipc.shm_use_phys: 0
> kern.ipc.shmall: 32768
> kern.ipc.shmseg: 128
> kern.ipc.shmmni: 192
> kern.ipc.shmmin: 1
> kern.ipc.shmmax: 134217728
> kern.ipc.maxsockets: 32768
> kern.ipc.numopensockets: 188
> kern.ipc.nsfbufsused: 0
> kern.ipc.nsfbufspeak: 0
> kern.ipc.nsfbufs: 0
>
>
> TIA,
> Felix
>
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
More information about the freebsd-net
mailing list