How to go about diagnosing cause of packet loss

[D G Teed]

Howdy,

Lately we have been seeing increased packet loss
on our gateway/firewall.  Running a ping plotter outside of
the firewall shows the hops are running clean.
>From on or behind the firewall, we have 20 to 50%
packet loss to each hop, reaching several popular test
destinations.

e.g.:
$ mtr -c 100 -r www.cnn.com
HOST:                             Loss%   Snt   Last   Avg  Best  Wrst StDev

  1. vlan-136.acadiau.ca           0.0%   100    0.4   6.1   0.4 179.9  26.5
  2. silverhorde.acadiau.ca        4.0%   100    0.6   0.9   0.3   7.8   1.0
  3. wfvlnsauh05-fe-0-0.aliant.ne 17.0%   100    3.4   6.3   2.6   55.0
8.8
  4. hlfxns01h29-ge-4-0.aliant.ne 27.0%   100    3.6   3.8   2.5  12.4   1.4
  5. rtp629049rts                 15.0%   100    4.2   4.0   2.6   9.1   1.2
  6. core1-halifax_POS5-0.net.bel 22.0%   100    6.2   3.7   2.6   6.2   0.9
  7. core3-montrealak_pos1-1.net.  4.0%   100   24.2  26.8  20.3 126.2  19.2
  8. core1-newyork83_pos_5_0_0.ne 19.0%   100   26.1  26.9  26.0  34.1   1.2
  9. bx4-newyork83_pos_2_0_0.net. 31.0%   100   27.7  28.1  27.1  30.1   0.8
 10. pop1-nye-P8-1.atdn.net        9.0%   100   26.2  45.2  26.2 227.4  48.0
 11. bb2-nye-P0-0.atdn.net        16.0%   100   29.0  31.1  26.3 178.2
19.4
 12. bb2-vie-P12-0.atdn.net       14.0%   100   33.0  46.3  32.3 206.4
37.6
 13. bb2-atm-P3-0.atdn.net        18.0%   100   42.9  44.9  42.5 106.6
9.7
 14. ???                          100.0   100    0.0   0.0   0.0   0.0   0.0

We have tested ipfw to allow ip from any as rule 01
to see if logging and filtering were the issue, but it stayed
the same.  It is beginning to look like the gateway server
might be saturated.

A reboot initially cleared up the problem, but 10 minutes
later we saw the packet loss again.

Does anyone have suggestions no how to troubleshoot/resolve
this problem?  The things I'd like to measure in a short
time snap are numbers of concurrent packets,  and
bandwidth.  Suggestions on measuring and tweaking this
in FreeBSD (4.11) welcomed.

--Donald