lagg/lacp poor traffic distribution

[Shtorm]

On Sun, 2010-12-19 at 00:35 +0600, Eugene Grosbein wrote:
> Hi!
> 
> I've loaded router using two lagg interfaces in LACP mode.
> lagg0 has IP address and two ports (em0 and em1) and carry untagged frames.
> lagg1 has no IP address and has two ports (igb0 and igb1) and carry
> about 1000 dot-q vlans with lots of hosts in each vlan.
> 
> For lagg1, lagg distributes outgoing traffic over two ports just fine.
> For lagg0 (untagged ethernet segment with only 2 MAC addresses)
> less than 0.07% (54Mbit/s max) of traffic goes to em0
> and over 99.92% goes to em1, that's bad.
> 
> That's general traffic of several thousands of customers surfing the web,
> using torrents etc.  I've glanced over lagg/lacp sources if src/sys/net/
> and found nothing suspicious, it should extract and use srcIP/dstIP for hash.
> 
> How do I debug this problem?
> 
> Eugene Grosbein
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"

I had this problem with igb driver, and I found, that lagg selects
outgoing interface based on packet header flowid field if M_FLOWID field
is set. And in the igb driver code flowid is set as 

#if __FreeBSD_version >= 800000
<------><------><------>rxr->fmp->m_pkthdr.flowid = que->msix;
<------><------><------>rxr->fmp->m_flags |= M_FLOWID;
#endif

The same thing in em driver with MULTIQUEUE 

That does not give enough number of flows to balance traffic well, so I
commented check in if_lagg.c

lagg_lb_start(struct lagg_softc *sc, struct mbuf *m)
{
<------>struct lagg_lb *lb = (struct lagg_lb *)sc->sc_psc;
<------>struct lagg_port *lp = NULL;
<------>uint32_t p = 0;

//<---->if (m->m_flags & M_FLOWID)
//<----><------>p = m->m_pkthdr.flowid;
//<---->else

and with this change I have much better load distribution across interfaces.

Hope it helps.