re0 take 100% CPU

Eugen Konkov kes-kes at yandex.ru
Fri May 25 18:15:48 UTC 2012


How to look what re0 is doing?? And how to debug that?

last pid: 74164;  load averages: 17.97, 13.81, 14.40                           up 47+09:31:54  21:50:58
110 processes: 9 running, 87 sleeping, 14 waiting
CPU:  0.5% user,  0.0% nice,  1.6% system, 97.9% interrupt,  0.0% idle
Mem: 194M Active, 449M Inact, 204M Wired, 20K Cache, 112M Buf, 1143M Free
Swap: 512M Total, 512M Free

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   12 root     -76    -     0K   112K WAIT    65.3H 100.00% {irq256: re0}
65795 root      39    0 13420K  5804K select  19:58  0.63% snmpd
 2664 root      36    0 14608K  6812K RUN     75:49  0.54% bgpd
    0 root     -76    0     0K    80K -       31.6H  0.44% {dummynet}
 7758 bind      36    0 75268K 62880K RUN    178:21  0.24% {named}
   12 root     -44    -     0K   112K WAIT    29.4H  0.10% {swi4: clock}

# systat -v
    2 users    Load 13.61 12.24 13.96                  May 24 21:49

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
        Tot   Share      Tot    Share    Free           in   out     in   out
Act  256100    6368   626872     8116 1171172  count
All  353960    9500  2792812    26356          pages
Proc:                                                            Interrupts
  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt      3 cow    4404 total
 12          32      2207  106 4216 4404 3983   48     28 zfod      1 ata0 14
                                                          ozfod  4119 hpet0 20
 4.1%Sys  93.4%Intr  2.6%User  0.0%Nice  0.0%Idle        %ozfod       uhci0 ehci
|    |    |    |    |    |    |    |    |    |    |       daefr   284 re0 256
==+++++++++++++++++++++++++++++++++++++++++++++++>      7 prcfr
                                        40 dtbuf       44 totfr
Namei     Name-cache   Dir-cache    111725 desvn          react
   Calls    hits   %    hits   %     64097 numvn          pdwak
    3177    3177 100                 27926 frevn          pdpgs
                                                          intrn
Disks   ad1   da0 pass0                            209000 wire
KB/t  14.85  0.00  0.00                            198616 act
tps       1     0     0                            459200 inact
MB/s   0.01  0.00  0.00                                20 cache
%busy     2     0     0                           1169956 free

# vmstat -i
interrupt                          total       rate
irq14: ata0                      5946124          1
irq20: hpet0                  3979660990        971
irq23: uhci0 ehci0                    85          0
irq256: re0                   3847013663        939
Total                         7832620862       1912
# uptime
 9:44PM  up 47 days,  9:25, 2 users, load averages: 9.05, 14.17, 15.61

#bwm-ng -I re0
  bwm-ng v0.6 (probing every 0.500s), press 'h' for help
  input: getifaddrs type: rate
  -         iface                   Rx                   Tx                Total
  ==============================================================================
              re0:          13.32 Mb/s           12.42 Mb/s           25.74 Mb/s
  ------------------------------------------------------------------------------
            total:          14.95 Mb/s           13.94 Mb/s           28.89 Mb/s
# vmstat -z
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               128,      0,      92,      28,      92,   0,   0
UMA Zones:              224,      0,      92,      10,      92,   0,   0
UMA Slabs:              284,      0,    1158,       4,   29780,   0,   0
UMA RCntSlabs:          544,      0,     919,       5,     919,   0,   0
UMA Hash:               128,      0,       2,      28,       3,   0,   0
16 Bucket:               76,      0,      36,      14,      36,   0,   0
32 Bucket:              140,      0,      43,      13,      44,   0,   0
64 Bucket:              268,      0,      59,      11,      67,  90,   0
128 Bucket:             524,      0,     895,       1,   27120, 585,   0
VM OBJECT:              136,      0,   37976,     130,40520105,   0,   0
MAP:                    140,      0,       7,      49,       7,   0,   0
KMAP ENTRY:              72,  57505,      32,     127,   79899,   0,   0
MAP ENTRY:               72,      0,    2094,    1192,100480394,   0,   0
DP fakepg:               72,      0,       0,       0,       0,   0,   0
SG fakepg:               72,      0,       0,       0,       0,   0,   0
mt_zone:               2060,      0,     268,      11,     268,   0,   0
16:                      16,      0,    5106,     375,251057215,   0,   0
32:                      32,      0,   26924,     196,1931894354,   0,   0
64:                      64,      0,    6372,     413,172309547,   0,   0
128:                    128,      0,   23931,   41019,195195229,   0,   0
256:                    256,      0,    2786,     289,2302309652,   0,   0
512:                    512,      0,     638,      58, 4189195,   0,   0
1024:                  1024,      0,     533,      15, 1586826,   0,   0
2048:                  2048,      0,     251,      59,   17903,   0,   0
4096:                  4096,      0,     154,      29, 2738963,   0,   0
Files:                   56,      0,     180,     289,224690270,   0,   0
TURNSTILE:               72,      0,     156,      24,     156,   0,   0
umtx pi:                 52,      0,       0,       0,       0,   0,   0
MAC labels:              20,      0,       0,       0,       0,   0,   0
PROC:                   688,      0,      57,      23, 1272718,   0,   0
THREAD:                 728,      0,     124,      31,  206859,   0,   0
SLEEPQUEUE:              44,      0,     156,      80,     156,   0,   0
VMSPACE:                232,      0,      38,      30, 1272743,   0,   0
cpuset:                  40,      0,       2,     182,       2,   0,   0
audit_record:           816,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,     256,    1536,5400971809,   0,   0
mbuf:                   256,      0,       6,    1412,9019029846,   0,   0
mbuf_cluster:          2048,  25600,    1792,       6,    1792,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,      20,  828696,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
NetGraph items:          36,   4130,       2,     175,1535407748,   0,   0
NetGraph data items:     36,    531,       0,     531,3322859915,26160,   0
g_bio:                  140,      0,       0,    4144,23787066,   0,   0
ttyinq:                 152,      0,     300,      12,     675,   0,   0
ttyoutq:                256,      0,     157,       8,     353,   0,   0
ata_request:            204,      0,       0,    1045, 5946685,   0,   0
ata_composite:          180,      0,       0,       0,       0,   0,   0
VNODE:                  272,      0,   64112,     134, 7873012,   0,   0
VNODEPOLL:               60,      0,       0,       0,       0,   0,   0
S VFS Cache:             72,      0,   66367,     466, 8051562,   0,   0
L VFS Cache:            292,      0,     595,     198,  109305,   0,   0
NAMEI:                 1024,      0,       0,      12,171046456,   0,   0
NFSMOUNT:               532,      0,       0,       0,       0,   0,   0
NFSNODE:                460,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,     485,      11,     497,   0,   0
pipe:                   392,      0,      10,      30,  784482,   0,   0
Mountpoints:            648,      0,       6,       6,       6,   0,   0
ksiginfo:                80,      0,      66,     990,  210783,   0,   0
itimer:                 220,      0,       0,       0,       0,   0,   0
KNOTE:                   72,      0,       8,     310,92381338,   0,   0
socket:                 416,  25605,      73,     242,131092823,   0,   0
unpcb:                  172,  25622,      26,      43, 3331350,   0,   0
ipq:                     32,    904,       0,     226,    8756,   0,   0
udp_inpcb:              220,  25614,       7,     245,127020097,   0,   0
udpcb:                    8,  25781,       7,     399,127020097,   0,   0
tcp_inpcb:              220,  25614,      33,      57,  644727,   0,   0
tcpcb:                  692,  25600,      26,      39,  644727,   0,   0
tcptw:                   52,   5184,       7,     137,   92058,   0,   0
syncache:               112,  15365,       0,      70,  360404,   0,   0
hostcache:               76,  15400,      30,      70,    3515,   0,   0
tcpreass:                20,   1690,       0,     169,    1927,   0,   0
sackhole:                20,      0,       0,     169,     620,   0,   0
sctp_ep:                880,  25600,       0,       0,       0,   0,   0
sctp_asoc:             1524,  40000,       0,       0,       0,   0,   0
sctp_laddr:              24,  80040,       0,     145,      45,   0,   0
sctp_raddr:             464,  80000,       0,       0,       0,   0,   0
sctp_chunk:              96, 400000,       0,       0,       0,   0,   0
sctp_readq:              76, 400000,       0,       0,       0,   0,   0
sctp_stream_msg_out:     72, 400044,       0,       0,       0,   0,   0
sctp_asconf:             24, 400055,       0,       0,       0,   0,   0
sctp_asconf_ack:         24, 400055,       0,       0,       0,   0,   0
ripcb:                  220,  25614,       6,      48,   78019,   0,   0
rtentry:                108,      0,   24066,      18,  288480,   0,   0
IPFW dynamic rule:      108,      0,       0,       0,       0,   0,   0
divcb:                  220,  25614,       0,       0,       0,   0,   0
selfd:                   28,      0,     158,     223,1786944834,   0,   0
ip4flow:                 40,  25668,     680,   24160,36376221,   0,   0
ip6flow:                 64,  25636,       0,     174,      20,   0,   0
SWAPMETA:               276, 121576,       0,       0,       0,   0,   0
FFS inode:              116,      0,   64073,      46, 7872685,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,   64073,      22, 7872685,   0,   0

# uname -a
FreeBSD bor 9.0-CURRENT FreeBSD 9.0-CURRENT #1: Wed Feb 23 09:39:22 UTC 2011     @:/usr/obj/usr/src/sys/KES_KERN_v9  i386

#tail /var/log/messages
May 24 21:58:57 bor kernel: Limiting icmp unreach response from 273 to 200 packets/sec
May 24 21:58:58 bor kernel: Limiting icmp unreach response from 223 to 200 packets/sec
May 24 21:58:59 bor kernel: Limiting icmp unreach response from 235 to 200 packets/sec
May 24 21:59:00 bor kernel: Limiting icmp unreach response from 278 to 200 packets/sec
May 24 21:59:01 bor kernel: Limiting icmp unreach response from 283 to 200 packets/sec
May 24 21:59:02 bor kernel: Limiting icmp unreach response from 255 to 200 packets/sec
May 24 21:59:03 bor kernel: Limiting icmp unreach response from 269 to 200 packets/sec
May 24 21:59:04 bor kernel: Limiting icmp unreach response from 286 to 200 packets/sec
May 24 21:59:05 bor kernel: Limiting icmp unreach response from 225 to 200 packets/sec
May 24 21:59:06 bor kernel: Limiting icmp unreach response from 328 to 200 packets/sec
May 24 21:59:07 bor kernel: Limiting icmp unreach response from 285 to 200 packets/sec
May 24 21:59:08 bor kernel: Limiting icmp unreach response from 286 to 200 packets/sec
May 24 21:59:09 bor kernel: Limiting icmp unreach response from 304 to 200 packets/sec
May 24 21:59:10 bor kernel: Limiting icmp unreach response from 304 to 200 packets/sec
May 24 21:59:11 bor kernel: Limiting icmp unreach response from 311 to 200 packets/sec

Thank you.




More information about the freebsd-questions mailing list