Some MSI are not routed correctly

[Maxim Sobolev]

Oh, bingo! Just checked packets count and found the following:

$ sysctl -a | grep dev.igb.1.queue | grep packets
dev.igb.1.queue3.rx_packets: 2997
dev.igb.1.queue3.tx_packets: 21045801676
dev.igb.1.queue2.rx_packets: 3084
dev.igb.1.queue2.tx_packets: 21265692009
dev.igb.1.queue1.rx_packets: 3016
dev.igb.1.queue1.tx_packets: 21496134503
dev.igb.1.queue0.rx_packets: 48868
dev.igb.1.queue0.tx_packets: 21729900371
$ sysctl -a | grep dev.igb.0.queue | grep packets
dev.igb.0.queue3.rx_packets: 40760861870
dev.igb.0.queue3.tx_packets: 21068449957
dev.igb.0.queue2.rx_packets: 40724698310
dev.igb.0.queue2.tx_packets: 21288469372
dev.igb.0.queue1.rx_packets: 40739376158
dev.igb.0.queue1.tx_packets: 21519768656
dev.igb.0.queue0.rx_packets: 40602824141
dev.igb.0.queue0.tx_packets: 21754065014

Apparently all incoming packets are going through igb0, while outbound get
distributed. This means the upstream switch is not doing proper load
balancing between two ports. We'll take it to the DC to fix.

Thanks John, for helping to drill that down!

On Wed, Oct 21, 2015 at 11:31 AM, John Baldwin <jhb at freebsd.org> wrote:

> On Wednesday, October 21, 2015 11:29:17 AM Maxim Sobolev wrote:
> > Yes, I do. However, please note that for some reason they are not using
> > nearly as much CPU time as the other 4 for some reason.
> >
> >    11 root          -92    -     0K  1104K WAIT    3  95.3H  28.96%
> > intr{irq267: igb0:que}
> >    11 root          -92    -     0K  1104K WAIT    1  95.5H  24.41%
> > intr{irq265: igb0:que}
> >    11 root          -92    -     0K  1104K CPU2    2  95.2H  23.73%
> > intr{irq266: igb0:que}
> >    11 root          -92    -     0K  1104K WAIT    0  95.2H  23.05%
> > intr{irq264: igb0:que}
> >    11 root          -92    -     0K  1104K WAIT    6 286:37   1.12%
> > intr{irq271: igb1:que}
> >    11 root          -92    -     0K  1104K WAIT    7 278:05   1.12%
> > intr{irq272: igb1:que}
> >    11 root          -92    -     0K  1104K WAIT    5 284:26   1.07%
> > intr{irq270: igb1:que}
> >    11 root          -92    -     0K  1104K WAIT    4 290:41   0.98%
> > intr{irq269: igb1:que}
> >
> > CPU 0:   0.0% user,  0.0% nice,  0.9% system, 24.9% interrupt, 74.2% idle
> > CPU 1:   0.5% user,  0.0% nice,  0.0% system, 26.3% interrupt, 73.2% idle
> > CPU 2:   0.0% user,  0.0% nice,  1.4% system, 25.4% interrupt, 73.2% idle
> > CPU 3:   0.0% user,  0.0% nice,  0.5% system, 23.9% interrupt, 75.6% idle
> > CPU 4:   0.9% user,  0.0% nice,  2.3% system,  2.3% interrupt, 94.4% idle
> > CPU 5:   1.4% user,  0.0% nice,  4.2% system,  4.2% interrupt, 90.1% idle
> > CPU 6:   1.4% user,  0.0% nice,  3.8% system,  1.4% interrupt, 93.4% idle
> > CPU 7:   2.8% user,  0.0% nice,  0.0% system,  3.8% interrupt, 93.4% idle
> >
> > 34263 igb0:que 0
> > 32308 igb0:que 1
> > 35022 igb0:que 2
> > 34593 igb0:que 3
> > 14931 igb1:que 0
> > 13059 igb1:que 1
> > 12971 igb1:que 2
> > 13032 igb1:que 3
> >
> > So I guess interrupts are routed correctly after all, but for some reason
> > driver takes some 5 times less time to process it on cpus 4-7
> > (per-interrupt). Weird.
>
> Are the pps rates the same?  It seems like the interrupt rates on igb0
> are double those of igb1?
>
> --
> John Baldwin
>