How to go about diagnosing cause of packet loss

[D G Teed]

Hi,

How quick I sometimes find answers after posting these types
of questions.  I've discovered that if ipaudit is shutdown
(the promiscuous data gathering filter), the packet loss
is very low.  Once it starts up again at the half hour,
the packet loss returns.

Is there any strategy for dealing with this.  I've heard of
putting ipaudit on a second machine on a hub outside the firewall.
But this is for a medium to large institution, so it would
have to be something that can survive significant bandwidth.

--Donald

On 11/28/06, D G Teed <donald.teed at gmail.com> wrote:
>
> Howdy,
>
> Lately we have been seeing increased packet loss
> on our gateway/firewall.  Running a ping plotter outside of
> the firewall shows the hops are running clean.
> From on or behind the firewall, we have 20 to 50%
> packet loss to each hop, reaching several popular test
> destinations.
>
> e.g.:
> $ mtr -c 100 -r www.cnn.com
> HOST:                             Loss%   Snt   Last   Avg  Best  Wrst
> StDev
>   1. vlan-136.acadiau.ca            0.0%   100    0.4   6.1   0.4 179.9
> 26.5
>   2. silverhorde.acadiau.ca        4.0%   100    0.6   0.9   0.3   7.8
> 1.0
>   3. wfvlnsauh05-fe-0-0.aliant.ne 17.0%   100    3.4   6.3   2.6   55.0
> 8.8
>   4. hlfxns01h29-ge-4-0.aliant.ne 27.0%   100    3.6   3.8   2.5  12.4
> 1.4
>   5. rtp629049rts                 15.0%   100    4.2   4.0   2.6   9.1
> 1.2
>   6. core1-halifax_POS5-0.net.bel 22.0%   100    6.2   3.7   2.6   6.2
> 0.9
>   7. core3-montrealak_pos1-1.net.  4.0%   100   24.2  26.8  20.3 126.2
> 19.2
>   8. core1-newyork83_pos_5_0_0.ne 19.0%   100   26.1  26.9  26.0  34.1
> 1.2
>   9. bx4-newyork83_pos_2_0_0.net. 31.0%   100   27.7  28.1  27.1  30.1
> 0.8
>  10. pop1-nye-P8-1.atdn.net        9.0%   100   26.2  45.2  26.2 227.4
> 48.0
>  11. bb2-nye-P0-0.atdn.net        16.0%   100   29.0  31.1  26.3 178.2
> 19.4
>  12. bb2-vie-P12-0.atdn.net       14.0%   100   33.0  46.3  32.3 206.4
> 37.6
>  13. bb2-atm-P3-0.atdn.net        18.0%   100   42.9  44.9  42.5 106.6
> 9.7
>  14. ???                          100.0   100    0.0   0.0   0.0   0.0
> 0.0
>
> We have tested ipfw to allow ip from any as rule 01
> to see if logging and filtering were the issue, but it stayed
> the same.  It is beginning to look like the gateway server
> might be saturated.
>
> A reboot initially cleared up the problem, but 10 minutes
> later we saw the packet loss again.
>
> Does anyone have suggestions no how to troubleshoot/resolve
> this problem?  The things I'd like to measure in a short
> time snap are numbers of concurrent packets,  and
> bandwidth.  Suggestions on measuring and tweaking this
> in FreeBSD (4.11) welcomed.
>
> --Donald
>
>