lacp lagg port flags do not show correctly resulting in poor
traffic distribution/performance
Adarsh Joshi
adarsh.joshi at qlogic.com
Fri Jul 27 16:16:46 UTC 2012
Andrew,
Thanks for the response. Yes, the link is working fine. I am able to see LACPDUs on tcpdump.
regards
Adarsh
________________________________________
From: Andrew Boyer [aboyer at averesystems.com]
Sent: Friday, July 27, 2012 8:11 AM
To: Adarsh Joshi
Cc: freebsd-net at freebsd.org
Subject: Re: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance
Adarsh,
Sorry for the delay.
I'm not an LACP protocol expert, but looking at your logs I don't see ql1 on either node receiving a lacpdu response. Are you certain that link is working?
-Andrew
On Jul 10, 2012, at 1:53 PM, Adarsh Joshi wrote:
> Andrew,
>
> Here are the logs with LACP_DEBUG defined in ieee802.3ad_lacp.c,
>
> after typing
>
> Ifconfig lagg0 create
> ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netmask 255.255.255.0
>
> I compiled it as a standalone driver by the way.
>
> System 1:
>
> # ifconfig -v lagg0
> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
> ether 00:0e:1e:08:05:20
> inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> groups: lagg
> laggproto lacp
> lag id: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),
> (8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
> laggport: ql1 flags=18<COLLECTING,DISTRIBUTING> state=7D
> [(8000,00-0E-1E-08-05-20,01D3,8000,000D),
> (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> laggport: ql0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D
> [(8000,00-0E-1E-08-05-20,01D3,8000,000C),
> (8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
>
>
> System 2:
>
> # ifconfig -v lagg0
> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
> ether 00:0e:1e:04:2c:f0
> inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> groups: lagg
> laggproto lacp
> lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),
> (FFFF,00-00-00-00-00-00,0000,0000,0000)]
> laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D
> [(8000,00-0E-1E-04-2C-F0,0213,8000,000F),
> (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> laggport: ql0 flags=18<COLLECTING,DISTRIBUTING> state=3D
> [(8000,00-0E-1E-04-2C-F0,0213,8000,000E),
> (8000,00-0E-1E-08-05-20,01D3,8000,000C)]
>
>
> System 1 logs :
>
> Jul 10 10:38:49 bsd-14 kernel: lacp_attach[738] : lacp attached
> Jul 10 10:38:49 bsd-14 kernel: lacp_attach[740] : lacp_defined
> Jul 10 10:38:49 bsd-14 kernel: lagg0: link state changed to UP
> Jul 10 10:38:49 bsd-14 kernel: ql0: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1
> Jul 10 10:38:49 bsd-14 kernel: ql0: -> UNSELECTED
> Jul 10 10:38:49 bsd-14 kernel: ql1: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1
> Jul 10 10:38:49 bsd-14 kernel: ql1: -> UNSELECTED
> Jul 10 10:38:49 bsd-14 kernel: lacp_select_tx_port: no active aggregator
> Jul 10 10:38:50 bsd-14 kernel: ql1: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(0000,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator created
> Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 10:38:50 bsd-14 kernel: ql1: mux_state 0 -> 1
> Jul 10 10:38:50 bsd-14 kernel: ql0: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(0000,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator created
> Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 10:38:50 bsd-14 kernel: ql0: mux_state 0 -> 1
> Jul 10 10:38:51 bsd-14 kernel: ql1: lacpdu transmit
> Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D)
> Jul 10 10:38:51 bsd-14 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 10:38:51 bsd-14 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000)
> Jul 10 10:38:51 bsd-14 kernel: partner.state=2<TIMEOUT>
> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu transmit
> Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 10:38:51 bsd-14 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 10:38:51 bsd-14 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000)
> Jul 10 10:38:51 bsd-14 kernel: partner.state=2<TIMEOUT>
> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive
> Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 10:38:51 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION>
> Jul 10 10:38:51 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 10:38:51 bsd-14 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:51 bsd-14 kernel: ql0: old pstate 2<TIMEOUT>
> Jul 10 10:38:51 bsd-14 kernel: ql0: new pstate 5<ACTIVITY,AGGREGATION>
> Jul 10 10:38:51 bsd-14 kernel: ql0: partner timeout changed
> Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive
> Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 10:38:51 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION>
> Jul 10 10:38:51 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 10:38:51 bsd-14 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTED
> Jul 10 10:38:52 bsd-14 kernel: ql1: partner timeout changed
> Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0
> Jul 10 10:38:52 bsd-14 kernel: ql1: collecting disabled
> Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0
> Jul 10 10:38:52 bsd-14 kernel: ql1: mux_state 1 -> 0
> Jul 10 10:38:52 bsd-14 kernel: ql1: lacpdu transmit
> Jul 10 10:38:52 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D)
> Jul 10 10:38:52 bsd-14 kernel: actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED>
> Jul 10 10:38:52 bsd-14 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
> Jul 10 10:38:52 bsd-14 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
> Jul 10 10:38:52 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:52 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0
> Jul 10 10:38:52 bsd-14 kernel: ql0: collecting disabled
> Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0
> Jul 10 10:38:52 bsd-14 kernel: ql0: mux_state 1 -> 0
> Jul 10 10:38:52 bsd-14 kernel: ql0: lacpdu transmit
> Jul 10 10:38:52 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 10:38:52 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION>
> Jul 10 10:38:52 bsd-14 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 10:38:52 bsd-14 kernel: partner.state=5<ACTIVITY,AGGREGATION>
> Jul 10 10:38:52 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:53 bsd-14 kernel: ql1: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator created
> Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 10:38:53 bsd-14 kernel: ql1: mux_state 0 -> 1
> Jul 10 10:38:53 bsd-14 kernel: ql0: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
> Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator created
> Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
> Jul 10 10:38:53 bsd-14 kernel: ql0: mux_state 0 -> 1
> Jul 10 10:38:54 bsd-14 kernel: ql0: lacpdu receive
> Jul 10 10:38:54 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 10:38:54 bsd-14 kernel: actor.state=d<ACTIVITY,AGGREGATION,SYNC>
> Jul 10 10:38:54 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 10:38:54 bsd-14 kernel: partner.state=5<ACTIVITY,AGGREGATION>
> Jul 10 10:38:54 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:54 bsd-14 kernel: ql0: old pstate 5<ACTIVITY,AGGREGATION>
> Jul 10 10:38:54 bsd-14 kernel: ql0: new pstate d<ACTIVITY,AGGREGATION,SYNC>
> Jul 10 10:38:55 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0
> Jul 10 10:38:55 bsd-14 kernel: ql1: collecting disabled
> Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 1 -> 2
> Jul 10 10:38:55 bsd-14 kernel: ql1: collecting enabled
> Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 2 -> 3
> Jul 10 10:38:55 bsd-14 kernel: ql1: enable distributing on aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], nports 0 -> 1
> Jul 10 10:38:55 bsd-14 kernel: lacp_select_active_aggregator:
> Jul 10 10:38:55 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1
> Jul 10 10:38:55 bsd-14 kernel: active aggregator changed
> Jul 10 10:38:55 bsd-14 kernel: old (none)
> Jul 10 10:38:55 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 10:38:55 bsd-14 kernel: Set table 1 with 1 ports
> Jul 10 10:38:55 bsd-14 kernel: lacp_suppress_distributing
> Jul 10 10:38:55 bsd-14 kernel: ql1: marker transmit, port=13, sys=00:0e:1e:08:05:20, id=1
> Jul 10 10:38:55 bsd-14 kernel: ql0: marker transmit, port=12, sys=00:0e:1e:08:05:20, id=1
> Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 3 -> 4
> Jul 10 10:38:55 bsd-14 kernel: ql1: lacpdu transmit
> Jul 10 10:38:55 bsd-14 kernel: ql0: marker response, port=12, sys=00:0e:1e:08:05:20, id=1
> Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D)
> Jul 10 10:38:55 bsd-14 kernel: actor.state=7d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING,DEFAULTED>
> Jul 10 10:38:55 bsd-14 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
> Jul 10 10:38:55 bsd-14 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
> Jul 10 10:38:55 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:55 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], pending 1 -> 0
> Jul 10 10:38:55 bsd-14 kernel: ql0: collecting disabled
> Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 1 -> 2
> Jul 10 10:38:55 bsd-14 kernel: ql0: collecting enabled
> Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 2 -> 3
> Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu transmit
> Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 10:38:55 bsd-14 kernel: actor.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
> Jul 10 10:38:55 bsd-14 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 10:38:55 bsd-14 kernel: partner.state=d<ACTIVITY,AGGREGATION,SYNC>
> Jul 10 10:38:55 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu receive
> Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 10:38:55 bsd-14 kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
> Jul 10 10:38:55 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 10:38:55 bsd-14 kernel: partner.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
> Jul 10 10:38:55 bsd-14 kernel: maxdelay=0
> Jul 10 10:38:55 bsd-14 kernel: ql0: old pstate d<ACTIVITY,AGGREGATION,SYNC>
> Jul 10 10:38:55 bsd-14 kernel: ql0: new pstate 3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
> Jul 10 10:38:56 bsd-14 kernel: ql0: enable distributing on aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], nports 0 -> 1
> Jul 10 10:38:56 bsd-14 kernel: lacp_select_active_aggregator:
> Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1
> Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], speed=10000000000, nports=1
> Jul 10 10:38:56 bsd-14 kernel: active aggregator changed
> Jul 10 10:38:56 bsd-14 kernel: old [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 10:38:56 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
> Jul 10 10:38:56 bsd-14 kernel: Set table 0 with 1 ports
> Jul 10 10:38:56 bsd-14 kernel: lacp_suppress_distributing
> Jul 10 10:38:56 bsd-14 kernel: ql1: marker transmit, port=13, sys=00:0e:1e:08:05:20, id=2
> Jul 10 10:38:56 bsd-14 kernel: ql0: marker transmit, port=12, sys=00:0e:1e:08:05:20, id=2
> Jul 10 10:38:56 bsd-14 kernel: ql0: mux_state 3 -> 4
> Jul 10 10:38:56 bsd-14 kernel: ql0: marker response, port=12, sys=00:0e:1e:08:05:20, id=2
> Jul 10 10:38:59 bsd-14 kernel: lacp_transit_expire
> ^C
>
>
>
>
> System 2 logs :
>
> Jul 10 02:38:24 bsd-15 kernel: lacp_attach[738] : lacp attached
> Jul 10 02:38:24 bsd-15 kernel: lacp_attach[740] : lacp_defined
> Jul 10 02:38:24 bsd-15 kernel: lagg0: link state changed to UP
> Jul 10 02:38:24 bsd-15 kernel: ql0: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1
> Jul 10 02:38:24 bsd-15 kernel: ql0: -> UNSELECTED
> Jul 10 02:38:24 bsd-15 kernel: ql1: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1
> Jul 10 02:38:24 bsd-15 kernel: ql1: -> UNSELECTED
> Jul 10 02:38:24 bsd-15 kernel: lacp_select_tx_port: no active aggregator
> Jul 10 02:38:25 bsd-15 kernel: ql1: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(0000,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator created
> Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 02:38:25 bsd-15 kernel: ql1: mux_state 0 -> 1
> Jul 10 02:38:25 bsd-15 kernel: ql0: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(0000,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator created
> Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 02:38:25 bsd-15 kernel: ql0: mux_state 0 -> 1
> Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu receive
> Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 02:38:26 bsd-15 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 02:38:26 bsd-15 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000)
> Jul 10 02:38:26 bsd-15 kernel: partner.state=2<TIMEOUT>
> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:26 bsd-15 kernel: ql0: lacp_sm_rx_update_ntt: assert ntt
> Jul 10 02:38:26 bsd-15 kernel: ql0: old pstate 2<TIMEOUT>
> Jul 10 02:38:26 bsd-15 kernel: ql0: new pstate 85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 02:38:26 bsd-15 kernel: ql0: partner timeout changed
> Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit
> Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 02:38:26 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION>
> Jul 10 02:38:26 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 02:38:26 bsd-15 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:26 bsd-15 kernel: ql1: lacpdu transmit
> Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F)
> Jul 10 02:38:26 bsd-15 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 02:38:26 bsd-15 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000)
> Jul 10 02:38:26 bsd-15 kernel: partner.state=2<TIMEOUT>
> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:26 bsd-15 kernel: ql0: collecting disabled
> Jul 10 02:38:26 bsd-15 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0
>
> Jul 10 02:38:26 bsd-15 kernel: ql0: mux_state 1 -> 0
> Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit
> Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 02:38:26 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION>
> Jul 10 02:38:26 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 02:38:26 bsd-15 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:27 bsd-15 kernel: ql0: lacpdu receive
> Jul 10 02:38:27 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 02:38:27 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION>
> Jul 10 02:38:27 bsd-15 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 02:38:27 bsd-15 kernel: partner.state=5<ACTIVITY,AGGREGATION>
> Jul 10 02:38:27 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:27 bsd-15 kernel: ql0: old pstate 85<ACTIVITY,AGGREGATION,EXPIRED>
> Jul 10 02:38:27 bsd-15 kernel: ql0: new pstate 5<ACTIVITY,AGGREGATION>
> Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTED
> Jul 10 02:38:27 bsd-15 kernel: ql1: partner timeout changed
> Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0
> Jul 10 02:38:27 bsd-15 kernel: ql1: collecting disabled
> Jul 10 02:38:27 bsd-15 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0
>
> Jul 10 02:38:27 bsd-15 kernel: ql1: mux_state 1 -> 0
> Jul 10 02:38:27 bsd-15 kernel: ql1: lacpdu transmit
> Jul 10 02:38:27 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F)
> Jul 10 02:38:27 bsd-15 kernel: actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED>
> Jul 10 02:38:27 bsd-15 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
> Jul 10 02:38:27 bsd-15 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
> Jul 10 02:38:27 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:27 bsd-15 kernel: ql0: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(8000,00-0E-1E-08-05-20,01D3,8000,000C)]
> Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator created
> Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)]
> Jul 10 02:38:27 bsd-15 kernel: ql0: mux_state 0 -> 1
> Jul 10 02:38:28 bsd-15 kernel: ql1: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator created
> Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 02:38:28 bsd-15 kernel: ql1: mux_state 0 -> 1
> Jul 10 02:38:29 bsd-15 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], pending 1 -> 0
> Jul 10 02:38:29 bsd-15 kernel: ql0: collecting disabled
> Jul 10 02:38:29 bsd-15 kernel: ql0: mux_state 1 -> 2
> Jul 10 02:38:29 bsd-15 kernel: ql0: lacpdu transmit
> Jul 10 02:38:29 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 02:38:29 bsd-15 kernel: actor.state=d<ACTIVITY,AGGREGATION,SYNC>
> Jul 10 02:38:29 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 02:38:29 bsd-15 kernel: partner.state=5<ACTIVITY,AGGREGATION>
> Jul 10 02:38:29 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu receive
> Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 02:38:30 bsd-15 kernel: actor.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
> Jul 10 02:38:30 bsd-15 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 02:38:30 bsd-15 kernel: partner.state=d<ACTIVITY,AGGREGATION,SYNC>
> Jul 10 02:38:30 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:30 bsd-15 kernel: ql0: old pstate 5<ACTIVITY,AGGREGATION>
> Jul 10 02:38:30 bsd-15 kernel: ql0: new pstate 1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
> Jul 10 02:38:30 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0
> Jul 10 02:38:30 bsd-15 kernel: ql1: collecting disabled
> Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 1 -> 2
> Jul 10 02:38:30 bsd-15 kernel: ql1: collecting enabled
> Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 2 -> 3
> Jul 10 02:38:30 bsd-15 kernel: ql1: enable distributing on aggregator [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], nports 0 -> 1
> Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator:
> Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1
> Jul 10 02:38:30 bsd-15 kernel: active aggregator changed
> Jul 10 02:38:30 bsd-15 kernel: old (none)
> Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00 00-00-00,0000,0000,0000)]
> Jul 10 02:38:30 bsd-15 kernel: Set table 1 with 1 ports
> Jul 10 02:38:30 bsd-15 kernel: lacp_suppress_distributing
> Jul 10 02:38:30 bsd-15 kernel: ql1: marker transmit, port=15, sys=00:0e:1e:04:2c:f0, id=1
> Jul 10 02:38:30 bsd-15 kernel: ql0: marker transmit, port=14, sys=00:0e:1e:04:2c:f0, id=1
> Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 3 -> 4
> Jul 10 02:38:30 bsd-15 kernel: ql1: lacpdu transmit
> Jul 10 02:38:30 bsd-15 kernel: ql0: marker response, port=14, sys=00:0e:1e:04:2c:f0, id=1
> Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F)
> Jul 10 02:38:30 bsd-15 kernel: actor.state=7d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING,DEFAULTED>
> Jul 10 02:38:30 bsd-15 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
> Jul 10 02:38:30 bsd-15 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
> Jul 10 02:38:30 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:30 bsd-15 kernel: ql0: collecting enabled
> Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 2 -> 3
> Jul 10 02:38:30 bsd-15 kernel: ql0: enable distributing on aggregator [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], nports 0 -> 1
> Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator:
> Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], speed=10000000000, nports=1
> Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1
> Jul 10 02:38:30 bsd-15 kernel: active aggregator not changed
> Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
> Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 3 -> 4
> Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu transmit
> Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
> Jul 10 02:38:30 bsd-15 kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
> Jul 10 02:38:30 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
> Jul 10 02:38:30 bsd-15 kernel: partner.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
> Jul 10 02:38:30 bsd-15 kernel: maxdelay=0
> Jul 10 02:38:33 bsd-15 kernel: lacp_transit_expire
> ^C
>
> Let me know if you need more info.
>
> Thanks
> Adarsh
>
>
> -----Original Message-----
> From: owner-freebsd-net at freebsd.org [mailto:owner-freebsd-net at freebsd.org] On Behalf Of Adarsh Joshi
> Sent: Tuesday, July 10, 2012 10:08 AM
> To: Andrew Boyer
> Cc: freebsd-net at freebsd.org
> Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance
>
> Andrew,
>
> Thanks for the reply.
>
> The reason for my suspicion on the portflags is thus (extracted from the ifconfig output in my previous mail):
>
> System 1:
> Laggport: ql1 flags = 18 state = 7D
> Laggport: ql0 flags = 1c state = 3D
>
> System 2:
> Laggport: ql1 flags = 1c state = 7D
> Laggport: ql0 flags = 18 state = 3D
>
> I should have explained my setup to you before. Here it is.
> Both the ql0 interfaces of the 2 systems are connected using a single cable and ql1 interfaces of the 2 systems are connected using a single cable.
>
> System 1 System 2
> ql0 <=======> ql0
> ql1 <=======> ql1
>
> With this setup, I don't think it is possible for ports ql0 to talk to their partners (each other) and ql1 ports not getting a response from their partner and still get the lagg configuration I have posted.
>
> I thought the portflags are dependent on the LACP state. But I see different flags for the same LACP state (For the state 7D, ql1 on system 1 shows flags = 18 and ql1 on system 2 shows flags = 1c).
>
> Or is my understanding totally wrong?
>
> I will send the LACP_DEBUG logs within the hour.
>
> Thanks
> Adarsh
>
> From: Andrew Boyer [mailto:aboyer at averesystems.com]
> Sent: Tuesday, July 10, 2012 5:57 AM
> To: Adarsh Joshi
> Cc: freebsd-net at freebsd.org
> Subject: Re: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance
>
>
> On Jul 9, 2012, at 8:38 PM, Adarsh Joshi wrote:
>
>
> Hi,
>
> I am trying to configure lacp lagg interfaces with 2 systems connected back to back as follows:
>
> Ifconfig lagg0 create
> Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netmask 255.255.255.0
>
> Sometimes, the lag interface comes up correctly but sometimes the laggport flags do not show properly. Instead of 1c<ACTIVE,COLLECTING,DISTRIBUTING>, it shows values of 18. I have seen similar issues reported on various forums with no solution.
> Looking at the lagg driver code and reading the standard, I thought the laggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in file ieee8023ad_lacp.h. But the following ifconfig -v output does not make any sense to me.
>
> My concern is that when all the interfaces show flags as 1c, the traffic is distributed across both the interfaces uniformly and I get aggregated throughput. If not, the traffic flows only on 1 interface.
>
> Is this a bug? How do I solve this? Or am I doing something wrong?
>
> I am using Free-BSD 9.0 release.
>
> System 1:
> # ifconfig -v lagg0
> lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000),
> (8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
> laggport: ql1 flags=18<COLLECTING,DISTRIBUTING> state=7D
> [(8000,00-0E-1E-08-05-20,0213,8000,000F),
> (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> laggport: ql0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D
> [(8000,00-0E-1E-08-05-20,0213,8000,000E),
> (8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
>
> System 2:
>
> # ifconfig -v lagg0
> lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),
> (FFFF,00-00-00-00-00-00,0000,0000,0000)]
> laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D
> [(8000,00-0E-1E-04-2C-F0,0213,8000,000F),
> (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
> laggport: ql0 flags=18<COLLECTING,DISTRIBUTING> state=3D
> [(8000,00-0E-1E-04-2C-F0,0213,8000,000E),
> (8000,00-0E-1E-08-05-20,0213,8000,000E)]
>
>
> thanks
> Adarsh
>
> I don't think you have a port flags problem per se; the flags are correctly displaying the state of the lagg. Your problem is that your systems aren't negotiating the correct lagg configuration. Each tuple after the laggport represents the [(actor state),(partner state)]. Ports ql0 have been able to talk to their partners (each other). Neither ql1 port has seen a response from a partner, though.
>
> You could try restarting the state machine on one box with 'ifconfig lagg0 laggproto lacp'. To see the negotiation you'll need to rebuild your kernel with '#define LACP_DEBUG 1' added to the top of sys/net/ieee802.3ad_lacp.c. Or upgrade to a newer stable snapshot that has the net.lacp_debug sysctl and turn it on.
>
> Or just turn off LACP. What does it get you in this configuration?
>
> Hope this helps,
> Andrew
>
> --------------------------------------------------
> Andrew Boyer aboyer at averesystems.com<mailto:aboyer at averesystems.com>
>
>
>
>
>
> ________________________________
> This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
>
> This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
>
--------------------------------------------------
Andrew Boyer aboyer at averesystems.com
This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
More information about the freebsd-net
mailing list