lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance

Tue Jul 10 08:35:19 UTC 2012

On 7/10/12 9:10 AM, Jason Hellenthal wrote:
> 
> 
> On Mon, Jul 09, 2012 at 05:38:24PM -0700, Adarsh Joshi wrote:
>> Hi,
>>
>> I am trying to configure lacp lagg interfaces with 2 systems connected back to back as follows:
>>
>> Ifconfig lagg0 create
>> Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netmask 255.255.255.0
>>
>> Sometimes, the lag interface comes up correctly but sometimes the laggport flags do not show properly. Instead of 1c<ACTIVE,COLLECTING,DISTRIBUTING>, it shows values of 18. I have seen similar issues reported on various forums with no solution.
>> Looking at the lagg driver code and reading the standard, I thought the laggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in file ieee8023ad_lacp.h. But the following ifconfig -v output does not make any sense to me.
>>
>> My concern is that when all the interfaces show flags as 1c, the traffic is distributed across both the interfaces uniformly and I get aggregated throughput. If not, the traffic flows only on 1 interface.
>>
>> Is this a bug? How do I solve this? Or am I doing something wrong?
>>
>> I am using Free-BSD 9.0 release.
>>
>> System 1:
>> # ifconfig -v lagg0
>> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
>>         ether 00:0e:1e:08:05:20
>>         inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
>>         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>         media: Ethernet autoselect
>>         status: active
>>         groups: lagg
>>         laggproto lacp
>>         lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000),
>>                  (8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
>>         laggport: ql1 flags=18<COLLECTING,DISTRIBUTING> state=7D
>>                 [(8000,00-0E-1E-08-05-20,0213,8000,000F),
>>                  (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
>>         laggport: ql0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D
>>                 [(8000,00-0E-1E-08-05-20,0213,8000,000E),
>>                  (8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
>>
>> System 2:
>>
>> # ifconfig -v lagg0
>> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
>>         ether 00:0e:1e:04:2c:f0
>>         inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255
>>         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>         media: Ethernet autoselect
>>         status: active
>>         groups: lagg
>>         laggproto lacp
>>         lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),
>>                  (FFFF,00-00-00-00-00-00,0000,0000,0000)]
>>         laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D
>>                [(8000,00-0E-1E-04-2C-F0,0213,8000,000F),
>>                  (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
>>         laggport: ql0 flags=18<COLLECTING,DISTRIBUTING> state=3D
>>                 [(8000,00-0E-1E-04-2C-F0,0213,8000,000E),
>>                  (8000,00-0E-1E-08-05-20,0213,8000,000E)]
>>
>>
> 
> Just for reference ... (stable/8 @ r238264)
> 
> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=80048<VLAN_MTU,POLLING,LINKSTATE>
>         ether 00:0c:41:21:1d:b5
>         inet 192.168.XX.X netmask 0xffffff00 broadcast 192.168.XX.XXX
> 	media: Ethernet autoselect
>         status: active
>         groups: lagg 
>         laggproto lacp lagghash l2,l3,l4
>         lag id: [(8000,00-0C-41-21-1D-B5,00E6,0000,0000),
>                  (FFFF,00-00-00-00-00-00,0000,0000,0000)]
>         laggport: dc1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D
>                 [(8000,00-0C-41-21-1D-B5,00E6,8000,0002),
>                  (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
>         laggport: dc0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D
>                 [(8000,00-0C-41-21-1D-B5,00E6,8000,0001),
>                  (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> 
> 
> They have had flags = 1c for quite some time and state = 7D
> 
> And just to show the variation ...
> 
> 
>  dc0:
>      yesterday      8.53 MiB  /    2.61 MiB  /   11.14 MiB
>          today       693 KiB  /     156 KiB  /     849 KiB
> 
>  dc1:
>      yesterday     19.00 MiB  /    1.79 MiB  /   20.78 MiB
>          today       496 KiB  /     103 KiB  /     599 KiB
> 
>  lagg0:
>      yesterday     27.53 MiB  /    3.71 MiB  /   31.24 MiB
>          today      1.16 MiB  /     172 KiB  /    1.33 MiB
> 
> 
> I believe (know) there has been some changes in the LAgg code in
> stable/9 and stable/8 recently so you may want to check into that.
> 
> Given this is LAgg and LACP you will see some variation regardless but I
> recall a point that it seemed like one interface was being favored over
> the other quite repeatedly or obsessively that had me second guessing
> whether it was doing the right thing.
> 
> LACP in Cisco is quite different than how we treat it here in FreeBSD as
> it tends to use the interfaces quite evenly all the time so that also
> has me second guessing whether the right thing is happening here. ( in
> PAgP and LACP modes ).
>

Note that you can configure the way the switch load balances traffic thus:

switch(config)#port-channel load-balance ?
  dst-ip       Dst IP Addr
  dst-mac      Dst Mac Addr
  src-dst-ip   Src XOR Dst IP Addr
  src-dst-mac  Src XOR Dst Mac Addr
  src-ip       Src IP Addr
  src-mac      Src Mac Addr

You may also run simulations on the switch to see what interface from a
port-channel would be used:

switch#test etherchannel load-balance interface po48 ip 10.1.2.3 88.190.45.1
Would select Te1/1/2 of Po48

switch#test etherchannel load-balance interface po48 ip 10.1.2.3 88.190.45.9
Would select Te2/1/2 of Po48

switch#test etherchannel load-balance interface po48 ip 10.1.2.3
88.190.45.10
Would select Te2/1/2 of Po48

Hope this helps.