svn commit: r221346 - head/sys/netinet

Sat May 14 17:07:25 UTC 2011

On Sat, 14 May 2011 10:37:51 -0400 John Baldwin wrote:

 JB> Can you capture a tcpdump (probably easiest to do from the other host)?

I replaced the asserts with log statements to make the host not panic and the
captured dump survive.

Below are two examples. You can find the raw dump for the first case at
http://people.freebsd.org/~trociny/rcv_nxt.dump.1 (65Mb). I have the second
one too but it is rather large (290Mb) so I have not uploaded it.

May 14 18:37:54 kopusha hastd[4258]: Connection from tcp4://192.168.1.103:54038 to tcp4://192.168.1.103:7772.
May 14 18:37:54 kopusha hastd[4258]: Connection from tcp4://192.168.1.103:37712 to tcp4://192.168.1.103:7772.
May 14 18:37:54 kopusha hastd[4254]: [test] (primary) Connected to tcp4://kopusha:7772.
May 14 18:37:54 kopusha hastd[4254]: [test] (primary) Successfully reconnected to tcp4://kopusha:7772.
May 14 18:37:54 kopusha hastd[4254]: [test] (primary) Synchronization started. 512MB to go.
* May 14 18:38:15 kopusha kernel: tcp_input negative window: tp 0xc8dfd2b4 rcv_nxt 70085243 rcv_adv 70085242
May 14 18:38:19 kopusha hastd[4254]: [test] (primary) Unable to receive reply header: Operation timed out.
May 14 18:38:19 kopusha hastd[4254]: [test] (primary) Disconnected from tcp4://kopusha:7772.
May 14 18:38:19 kopusha hastd[4254]: [test] (primary) Unable to write synchronization data: Invalid argument.
May 14 18:38:19 kopusha hastd[4254]: [test] (primary) Synchronization interrupted after 25s. 187MB synchronized so far.
* May 14 18:38:20 kopusha kernel: tcp_input negative window: tp 0xc8dfd2b4 rcv_nxt 70085244 rcv_adv 70085242
May 14 18:38:24 kopusha hastd[4258]: Connection from tcp4://192.168.1.103:11242 to tcp4://192.168.1.103:7772.

18:37:59.645347 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [.], seq 53762563:53778895, ack 838, win 8859, options [nop,nop,TS val 23012 ecr 1886728700], length 16332
18:37:59.645355 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53762563, win 6891, options [nop,nop,TS val 1886728700 ecr 23012], length 0
18:37:59.645360 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [P.], seq 53778895:53778999, ack 838, win 8859, options [nop,nop,TS val 23012 ecr 1886728700], length 104
18:37:59.645379 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53778999, win 4837, options [nop,nop,TS val 1886728700 ecr 23012], length 0
18:37:59.645398 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [.], seq 53778999:53795331, ack 838, win 8859, options [nop,nop,TS val 23012 ecr 1886728700], length 16332
18:37:59.645418 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [P.], seq 53795331:53805402, ack 838, win 8859, options [nop,nop,TS val 23012 ecr 1886728700], length 10071
18:37:59.645441 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53805402, win 1536, options [nop,nop,TS val 1886728700 ecr 23012], length 0
18:37:59.645444 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [P.], seq 53805402:53811767, ack 838, win 8859, options [nop,nop,TS val 23012 ecr 1886728700], length 6365
18:37:59.739969 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53811767, win 741, options [nop,nop,TS val 1886728710 ecr 23012], length 0
18:38:04.739929 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [.], seq 53811767:53817695, ack 838, win 8859, options [nop,nop,TS val 23522 ecr 1886728710], length 5928
18:38:04.839913 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53817695, win 0, options [nop,nop,TS val 1886729220 ecr 23522], length 0
18:38:09.839927 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [.], seq 53817695:53817696, ack 838, win 8859, options [nop,nop,TS val 24032 ecr 1886729220], length 1
18:38:09.839968 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53817696, win 0, options [nop,nop,TS val 1886729720 ecr 24032], length 0
18:38:14.839923 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [.], seq 53817696:53817697, ack 838, win 8859, options [nop,nop,TS val 24532 ecr 1886729720], length 1
18:38:14.839977 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53817697, win 0, options [nop,nop,TS val 1886730220 ecr 24532], length 0
18:38:19.839933 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [.], seq 53817697:53817698, ack 838, win 8859, options [nop,nop,TS val 25032 ecr 1886730220], length 1
18:38:19.840010 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53817698, win 0, options [nop,nop,TS val 1886730720 ecr 25032], length 0
18:38:24.455025 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [.], ack 53817698, win 8960, options [nop,nop,TS val 1886731181 ecr 25032], length 0
18:38:24.455061 IP 192.168.1.103.54038 > 192.168.1.103.7772: Flags [FP.], seq 53817698:53820133, ack 838, win 8859, options [nop,nop,TS val 25493 ecr 1886731181], length 2435
18:38:24.455276 IP 192.168.1.103.7772 > 192.168.1.103.54038: Flags [R], seq 1895170621, win 0, length 0

May 14 19:14:17 kopusha hastd[6879]: Connection from tcp4://192.168.1.103:19814 to tcp4://192.168.1.103:7772.
May 14 19:14:17 kopusha hastd[6879]: Connection from tcp4://192.168.1.103:27111 to tcp4://192.168.1.103:7772.
May 14 19:14:17 kopusha hastd[6875]: [test] (primary) Connected to tcp4://kopusha:7772.
May 14 19:14:17 kopusha hastd[6875]: [test] (primary) Successfully reconnected to tcp4://kopusha:7772.
May 14 19:14:17 kopusha hastd[6875]: [test] (primary) Synchronization started. 512MB to go.
May 14 19:14:41 kopusha hastd[6875]: [test] (primary) Synchronization complete. 512MB synchronized in 24s (20MB/sec).
May 14 19:15:24 kopusha hastd[6875]: [test] (primary) Unable to receive reply header: Operation timed out.
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369184 rcv_adv 325369183
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369185 rcv_adv 325369183
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369186 rcv_adv 325369183
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369187 rcv_adv 325369183
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369188 rcv_adv 325369183
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369189 rcv_adv 325369183
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369190 rcv_adv 325369183
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369191 rcv_adv 325369183
* May 14 19:16:30 kopusha kernel: tcp_input negative window: tp 0xc9975ad0 rcv_nxt 325369192 rcv_adv 325369183
May 14 19:16:30 kopusha hastd[6875]: [test] (primary) Unable to send request (Broken pipe): WRITE(81870848, 16384).
May 14 19:16:30 kopusha hastd[6875]: [test] (primary) Disconnected from tcp4://kopusha:7772.
May 14 19:16:30 kopusha hastd[6879]: Connection from tcp4://192.168.1.103:26315 to tcp4://192.168.1.103:7772.

19:15:03.970985 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 282942005:282958337, ack 838, win 8896, options [nop,nop,TS val 245445 ecr 3738933168], length 16332
19:15:04.069929 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 282958337, win 5031, options [nop,nop,TS val 3738933178 ecr 245445], length 0
19:15:04.069981 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 282958337:282974669, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 16332
19:15:04.070002 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 282974669:282991001, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 16332
19:15:04.070033 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 282991001, win 948, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:04.070106 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 282991001, win 5443, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:04.070129 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 282991001:283007333, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 16332
19:15:04.070156 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283007333:283023665, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 16332
19:15:04.070208 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283023665, win 1360, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:04.072417 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283023665, win 5448, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:04.072456 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283023665:283039997, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 16332
19:15:04.072478 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [P.], seq 283039997:283045338, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 5341
19:15:04.072498 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283045338, win 2739, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:04.072949 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283045338, win 8097, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:04.073225 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [P.], seq 283045338:283045559, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 221
19:15:04.073282 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283045559:283061891, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 16332
19:15:04.073300 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283061891, win 6028, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:04.073322 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283061891:283078223, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 16332
19:15:04.073340 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [P.], seq 283078223:283078327, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 104
19:15:04.073369 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283078327, win 3973, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:04.075631 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283078327:283094659, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 16332
19:15:04.075708 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283094659:283110111, ack 838, win 8896, options [nop,nop,TS val 245455 ecr 3738933178], length 15452
19:15:04.075736 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110111, win 0, options [nop,nop,TS val 3738933178 ecr 245455], length 0
19:15:09.069925 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110111:283110112, ack 838, win 8896, options [nop,nop,TS val 245955 ecr 3738933178], length 1
19:15:09.069966 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110112, win 0, options [nop,nop,TS val 3738933678 ecr 245955], length 0
19:15:14.069931 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110112:283110113, ack 838, win 8896, options [nop,nop,TS val 246455 ecr 3738933678], length 1
19:15:14.069971 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110113, win 0, options [nop,nop,TS val 3738934178 ecr 246455], length 0
19:15:19.069929 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110113:283110114, ack 838, win 8896, options [nop,nop,TS val 246955 ecr 3738934178], length 1
19:15:19.069970 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110114, win 0, options [nop,nop,TS val 3738934678 ecr 246955], length 0
19:15:24.069943 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110114:283110115, ack 838, win 8896, options [nop,nop,TS val 247455 ecr 3738934678], length 1
19:15:24.069994 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110115, win 0, options [nop,nop,TS val 3738935178 ecr 247455], length 0
19:15:29.069929 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110115:283110116, ack 838, win 8896, options [nop,nop,TS val 247955 ecr 3738935178], length 1
19:15:29.069987 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110116, win 0, options [nop,nop,TS val 3738935678 ecr 247955], length 0
19:15:34.069927 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110116:283110117, ack 838, win 8896, options [nop,nop,TS val 248455 ecr 3738935678], length 1
19:15:34.069971 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110117, win 0, options [nop,nop,TS val 3738936178 ecr 248455], length 0
19:15:39.069931 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110117:283110118, ack 838, win 8896, options [nop,nop,TS val 248955 ecr 3738936178], length 1
19:15:39.069978 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110118, win 0, options [nop,nop,TS val 3738936678 ecr 248955], length 0
19:15:44.069921 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110118:283110119, ack 838, win 8896, options [nop,nop,TS val 249455 ecr 3738936678], length 1
19:15:44.069960 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110119, win 0, options [nop,nop,TS val 3738937178 ecr 249455], length 0
19:15:49.069920 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110119:283110120, ack 838, win 8896, options [nop,nop,TS val 249955 ecr 3738937178], length 1
19:15:49.069974 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110120, win 0, options [nop,nop,TS val 3738937678 ecr 249955], length 0
19:15:54.069928 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110120:283110121, ack 838, win 8896, options [nop,nop,TS val 250455 ecr 3738937678], length 1
19:15:54.069989 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110121, win 0, options [nop,nop,TS val 3738938178 ecr 250455], length 0
19:15:59.069927 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110121:283110122, ack 838, win 8896, options [nop,nop,TS val 250955 ecr 3738938178], length 1
19:15:59.069981 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110122, win 0, options [nop,nop,TS val 3738938678 ecr 250955], length 0
19:16:04.069926 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110122:283110123, ack 838, win 8896, options [nop,nop,TS val 251455 ecr 3738938678], length 1
19:16:04.069992 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110123, win 0, options [nop,nop,TS val 3738939178 ecr 251455], length 0
19:16:09.069921 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110123:283110124, ack 838, win 8896, options [nop,nop,TS val 251955 ecr 3738939178], length 1
19:16:09.069980 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110124, win 0, options [nop,nop,TS val 3738939678 ecr 251955], length 0
19:16:14.069924 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110124:283110125, ack 838, win 8896, options [nop,nop,TS val 252455 ecr 3738939678], length 1
19:16:14.069997 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110125, win 0, options [nop,nop,TS val 3738940178 ecr 252455], length 0
19:16:19.069925 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110125:283110126, ack 838, win 8896, options [nop,nop,TS val 252955 ecr 3738940178], length 1
19:16:19.069996 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110126, win 0, options [nop,nop,TS val 3738940678 ecr 252955], length 0
19:16:24.069924 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110126:283110127, ack 838, win 8896, options [nop,nop,TS val 253455 ecr 3738940678], length 1
19:16:24.069994 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110127, win 0, options [nop,nop,TS val 3738941178 ecr 253455], length 0
19:16:29.069924 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110127:283110128, ack 838, win 8896, options [nop,nop,TS val 253955 ecr 3738941178], length 1
19:16:29.069995 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110128, win 0, options [nop,nop,TS val 3738941678 ecr 253955], length 0
19:16:30.751499 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [.], ack 283110128, win 8960, options [nop,nop,TS val 3738941846 ecr 253955], length 0
19:16:30.751571 IP 192.168.1.103.19814 > 192.168.1.103.7772: Flags [.], seq 283110128:283126460, ack 838, win 8896, options [nop,nop,TS val 254123 ecr 3738941846], length 16332
19:16:30.751649 IP 192.168.1.103.7772 > 192.168.1.103.19814: Flags [R], seq 1092154945, win 0, length 0

-- 
Mikolaj Golub