lacp lagg port flags do not show correctly resulting in poor
traffic distribution/performance
Adarsh Joshi
adarsh.joshi at qlogic.com
Wed Jul 11 18:14:59 UTC 2012
I tried to configure lagg with the latest source from SVN head and it did not help.
Regards
Adarsh
-----Original Message-----
From: owner-freebsd-net at freebsd.org [mailto:owner-freebsd-net at freebsd.org] On Behalf Of Adarsh Joshi
Sent: Tuesday, July 10, 2012 10:53 AM
To: Andrew Boyer
Cc: freebsd-net at freebsd.org
Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance
Andrew,
Here are the logs with LACP_DEBUG defined in ieee802.3ad_lacp.c,
after typing
Ifconfig lagg0 create
ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netmask 255.255.255.0
I compiled it as a standalone driver by the way.
System 1:
# ifconfig -v lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
ether 00:0e:1e:08:05:20
inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp
lag id: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),
(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
laggport: ql1 flags=18<COLLECTING,DISTRIBUTING> state=7D
[(8000,00-0E-1E-08-05-20,01D3,8000,000D),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ql0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D
[(8000,00-0E-1E-08-05-20,01D3,8000,000C),
(8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
System 2:
# ifconfig -v lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
ether 00:0e:1e:04:2c:f0
inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp
lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),
(FFFF,00-00-00-00-00-00,0000,0000,0000)]
laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D
[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ql0 flags=18<COLLECTING,DISTRIBUTING> state=3D
[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),
(8000,00-0E-1E-08-05-20,01D3,8000,000C)]
System 1 logs :
Jul 10 10:38:49 bsd-14 kernel: lacp_attach[738] : lacp attached Jul 10 10:38:49 bsd-14 kernel: lacp_attach[740] : lacp_defined Jul 10 10:38:49 bsd-14 kernel: lagg0: link state changed to UP Jul 10 10:38:49 bsd-14 kernel: ql0: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1 Jul 10 10:38:49 bsd-14 kernel: ql0: -> UNSELECTED Jul 10 10:38:49 bsd-14 kernel: ql1: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1 Jul 10 10:38:49 bsd-14 kernel: ql1: -> UNSELECTED Jul 10 10:38:49 bsd-14 kernel: lacp_select_tx_port: no active aggregator Jul 10 10:38:50 bsd-14 kernel: ql1: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator created Jul 10 10:38:50 bsd-14 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:50 bsd-14 kernel: ql1: mux_state 0 -> 1 Jul 10 10:38:50 bsd-14 kernel: ql0: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator created Jul 10 10:38:50 bsd-14 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:50 bsd-14 kernel: ql0: mux_state 0 -> 1 Jul 10 10:38:51 bsd-14 kernel: ql1: lacpdu transmit Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D)
Jul 10 10:38:51 bsd-14 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED>
Jul 10 10:38:51 bsd-14 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000)
Jul 10 10:38:51 bsd-14 kernel: partner.state=2<TIMEOUT> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0 Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu transmit Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 10:38:51 bsd-14 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED>
Jul 10 10:38:51 bsd-14 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000)
Jul 10 10:38:51 bsd-14 kernel: partner.state=2<TIMEOUT> Jul 10 10:38:51 bsd-14 kernel: maxdelay=0 Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 10:38:51 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION>
Jul 10 10:38:51 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 10:38:51 bsd-14 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED>
Jul 10 10:38:51 bsd-14 kernel: maxdelay=0 Jul 10 10:38:51 bsd-14 kernel: ql0: old pstate 2<TIMEOUT> Jul 10 10:38:51 bsd-14 kernel: ql0: new pstate 5<ACTIVITY,AGGREGATION> Jul 10 10:38:51 bsd-14 kernel: ql0: partner timeout changed Jul 10 10:38:51 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:51 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 10:38:51 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION>
Jul 10 10:38:51 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 10:38:51 bsd-14 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED>
Jul 10 10:38:51 bsd-14 kernel: maxdelay=0 Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTED Jul 10 10:38:52 bsd-14 kernel: ql1: partner timeout changed Jul 10 10:38:52 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql1: collecting disabled Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql1: mux_state 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql1: lacpdu transmit Jul 10 10:38:52 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D)
Jul 10 10:38:52 bsd-14 kernel: actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED>
Jul 10 10:38:52 bsd-14 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
Jul 10 10:38:52 bsd-14 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Jul 10 10:38:52 bsd-14 kernel: maxdelay=0 Jul 10 10:38:52 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql0: collecting disabled Jul 10 10:38:52 bsd-14 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql0: mux_state 1 -> 0 Jul 10 10:38:52 bsd-14 kernel: ql0: lacpdu transmit Jul 10 10:38:52 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 10:38:52 bsd-14 kernel: actor.state=5<ACTIVITY,AGGREGATION>
Jul 10 10:38:52 bsd-14 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 10:38:52 bsd-14 kernel: partner.state=5<ACTIVITY,AGGREGATION>
Jul 10 10:38:52 bsd-14 kernel: maxdelay=0 Jul 10 10:38:53 bsd-14 kernel: ql1: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000D),(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator created Jul 10 10:38:53 bsd-14 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:53 bsd-14 kernel: ql1: mux_state 0 -> 1 Jul 10 10:38:53 bsd-14 kernel: ql0: port lagid=[(8000,00-0E-1E-08-05-20,01D3,8000,000C),(8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator created Jul 10 10:38:53 bsd-14 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
Jul 10 10:38:53 bsd-14 kernel: ql0: mux_state 0 -> 1 Jul 10 10:38:54 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:54 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 10:38:54 bsd-14 kernel: actor.state=d<ACTIVITY,AGGREGATION,SYNC>
Jul 10 10:38:54 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 10:38:54 bsd-14 kernel: partner.state=5<ACTIVITY,AGGREGATION>
Jul 10 10:38:54 bsd-14 kernel: maxdelay=0 Jul 10 10:38:54 bsd-14 kernel: ql0: old pstate 5<ACTIVITY,AGGREGATION> Jul 10 10:38:54 bsd-14 kernel: ql0: new pstate d<ACTIVITY,AGGREGATION,SYNC> Jul 10 10:38:55 bsd-14 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 10:38:55 bsd-14 kernel: ql1: collecting disabled Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 1 -> 2 Jul 10 10:38:55 bsd-14 kernel: ql1: collecting enabled Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 2 -> 3 Jul 10 10:38:55 bsd-14 kernel: ql1: enable distributing on aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], nports 0 -> 1 Jul 10 10:38:55 bsd-14 kernel: lacp_select_active_aggregator:
Jul 10 10:38:55 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1 Jul 10 10:38:55 bsd-14 kernel: active aggregator changed Jul 10 10:38:55 bsd-14 kernel: old (none) Jul 10 10:38:55 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:55 bsd-14 kernel: Set table 1 with 1 ports Jul 10 10:38:55 bsd-14 kernel: lacp_suppress_distributing Jul 10 10:38:55 bsd-14 kernel: ql1: marker transmit, port=13, sys=00:0e:1e:08:05:20, id=1 Jul 10 10:38:55 bsd-14 kernel: ql0: marker transmit, port=12, sys=00:0e:1e:08:05:20, id=1 Jul 10 10:38:55 bsd-14 kernel: ql1: mux_state 3 -> 4 Jul 10 10:38:55 bsd-14 kernel: ql1: lacpdu transmit Jul 10 10:38:55 bsd-14 kernel: ql0: marker response, port=12, sys=00:0e:1e:08:05:20, id=1 Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000D)
Jul 10 10:38:55 bsd-14 kernel: actor.state=7d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING,DEFAULTED>
Jul 10 10:38:55 bsd-14 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
Jul 10 10:38:55 bsd-14 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Jul 10 10:38:55 bsd-14 kernel: maxdelay=0 Jul 10 10:38:55 bsd-14 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], pending 1 -> 0 Jul 10 10:38:55 bsd-14 kernel: ql0: collecting disabled Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 1 -> 2 Jul 10 10:38:55 bsd-14 kernel: ql0: collecting enabled Jul 10 10:38:55 bsd-14 kernel: ql0: mux_state 2 -> 3 Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu transmit Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 10:38:55 bsd-14 kernel: actor.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
Jul 10 10:38:55 bsd-14 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 10:38:55 bsd-14 kernel: partner.state=d<ACTIVITY,AGGREGATION,SYNC>
Jul 10 10:38:55 bsd-14 kernel: maxdelay=0 Jul 10 10:38:55 bsd-14 kernel: ql0: lacpdu receive Jul 10 10:38:55 bsd-14 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 10:38:55 bsd-14 kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Jul 10 10:38:55 bsd-14 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 10:38:55 bsd-14 kernel: partner.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
Jul 10 10:38:55 bsd-14 kernel: maxdelay=0 Jul 10 10:38:55 bsd-14 kernel: ql0: old pstate d<ACTIVITY,AGGREGATION,SYNC> Jul 10 10:38:55 bsd-14 kernel: ql0: new pstate 3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Jul 10 10:38:56 bsd-14 kernel: ql0: enable distributing on aggregator [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], nports 0 -> 1 Jul 10 10:38:56 bsd-14 kernel: lacp_select_active_aggregator:
Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1 Jul 10 10:38:56 bsd-14 kernel: [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)], speed=10000000000, nports=1 Jul 10 10:38:56 bsd-14 kernel: active aggregator changed Jul 10 10:38:56 bsd-14 kernel: old [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 10:38:56 bsd-14 kernel: new [(8000,00-0E-1E-08-05-20,01D3,0000,0000),(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
Jul 10 10:38:56 bsd-14 kernel: Set table 0 with 1 ports Jul 10 10:38:56 bsd-14 kernel: lacp_suppress_distributing Jul 10 10:38:56 bsd-14 kernel: ql1: marker transmit, port=13, sys=00:0e:1e:08:05:20, id=2 Jul 10 10:38:56 bsd-14 kernel: ql0: marker transmit, port=12, sys=00:0e:1e:08:05:20, id=2 Jul 10 10:38:56 bsd-14 kernel: ql0: mux_state 3 -> 4 Jul 10 10:38:56 bsd-14 kernel: ql0: marker response, port=12, sys=00:0e:1e:08:05:20, id=2 Jul 10 10:38:59 bsd-14 kernel: lacp_transit_expire ^C
System 2 logs :
Jul 10 02:38:24 bsd-15 kernel: lacp_attach[738] : lacp attached Jul 10 02:38:24 bsd-15 kernel: lacp_attach[740] : lacp_defined Jul 10 02:38:24 bsd-15 kernel: lagg0: link state changed to UP Jul 10 02:38:24 bsd-15 kernel: ql0: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1 Jul 10 02:38:24 bsd-15 kernel: ql0: -> UNSELECTED Jul 10 02:38:24 bsd-15 kernel: ql1: media changed 0x0 -> 0x100033, ether = 1, fdx = 1, link = 1 Jul 10 02:38:24 bsd-15 kernel: ql1: -> UNSELECTED Jul 10 02:38:24 bsd-15 kernel: lacp_select_tx_port: no active aggregator Jul 10 02:38:25 bsd-15 kernel: ql1: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator created Jul 10 02:38:25 bsd-15 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:25 bsd-15 kernel: ql1: mux_state 0 -> 1 Jul 10 02:38:25 bsd-15 kernel: ql0: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator created Jul 10 02:38:25 bsd-15 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:25 bsd-15 kernel: ql0: mux_state 0 -> 1 Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu receive Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 02:38:26 bsd-15 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED>
Jul 10 02:38:26 bsd-15 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000)
Jul 10 02:38:26 bsd-15 kernel: partner.state=2<TIMEOUT> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0 Jul 10 02:38:26 bsd-15 kernel: ql0: lacp_sm_rx_update_ntt: assert ntt Jul 10 02:38:26 bsd-15 kernel: ql0: old pstate 2<TIMEOUT> Jul 10 02:38:26 bsd-15 kernel: ql0: new pstate 85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:26 bsd-15 kernel: ql0: partner timeout changed Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:26 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION>
Jul 10 02:38:26 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 02:38:26 bsd-15 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED>
Jul 10 02:38:26 bsd-15 kernel: maxdelay=0 Jul 10 02:38:26 bsd-15 kernel: ql1: lacpdu transmit Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F)
Jul 10 02:38:26 bsd-15 kernel: actor.state=85<ACTIVITY,AGGREGATION,EXPIRED>
Jul 10 02:38:26 bsd-15 kernel: partner=(0000,00-00-00-00-00-00,0000,0000,0000)
Jul 10 02:38:26 bsd-15 kernel: partner.state=2<TIMEOUT> Jul 10 02:38:26 bsd-15 kernel: maxdelay=0 Jul 10 02:38:26 bsd-15 kernel: ql0: collecting disabled Jul 10 02:38:26 bsd-15 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0
Jul 10 02:38:26 bsd-15 kernel: ql0: mux_state 1 -> 0 Jul 10 02:38:26 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:26 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:26 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION>
Jul 10 02:38:26 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 02:38:26 bsd-15 kernel: partner.state=85<ACTIVITY,AGGREGATION,EXPIRED>
Jul 10 02:38:26 bsd-15 kernel: maxdelay=0 Jul 10 02:38:27 bsd-15 kernel: ql0: lacpdu receive Jul 10 02:38:27 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 02:38:27 bsd-15 kernel: actor.state=5<ACTIVITY,AGGREGATION>
Jul 10 02:38:27 bsd-15 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:27 bsd-15 kernel: partner.state=5<ACTIVITY,AGGREGATION>
Jul 10 02:38:27 bsd-15 kernel: maxdelay=0 Jul 10 02:38:27 bsd-15 kernel: ql0: old pstate 85<ACTIVITY,AGGREGATION,EXPIRED> Jul 10 02:38:27 bsd-15 kernel: ql0: new pstate 5<ACTIVITY,AGGREGATION> Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_rx_timer: EXPIRED -> DEFAULTED Jul 10 02:38:27 bsd-15 kernel: ql1: partner timeout changed Jul 10 02:38:27 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 02:38:27 bsd-15 kernel: ql1: collecting disabled Jul 10 02:38:27 bsd-15 kernel: lacp_aggregator_delref: lagid=[(8000,00-0E-1E-04-2C-F0,0213, 0000,0000),(0000,00-00-00-00-00-00,0000,0000,0000)], refcnt 1 -> 0
Jul 10 02:38:27 bsd-15 kernel: ql1: mux_state 1 -> 0 Jul 10 02:38:27 bsd-15 kernel: ql1: lacpdu transmit Jul 10 02:38:27 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F)
Jul 10 02:38:27 bsd-15 kernel: actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED>
Jul 10 02:38:27 bsd-15 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
Jul 10 02:38:27 bsd-15 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Jul 10 02:38:27 bsd-15 kernel: maxdelay=0 Jul 10 02:38:27 bsd-15 kernel: ql0: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),(8000,00-0E-1E-08-05-20,01D3,8000,000C)]
Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator created Jul 10 02:38:27 bsd-15 kernel: ql0: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)]
Jul 10 02:38:27 bsd-15 kernel: ql0: mux_state 0 -> 1 Jul 10 02:38:28 bsd-15 kernel: ql1: port lagid=[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator created Jul 10 02:38:28 bsd-15 kernel: ql1: aggregator lagid=[(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:28 bsd-15 kernel: ql1: mux_state 0 -> 1 Jul 10 02:38:29 bsd-15 kernel: ql0: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], pending 1 -> 0 Jul 10 02:38:29 bsd-15 kernel: ql0: collecting disabled Jul 10 02:38:29 bsd-15 kernel: ql0: mux_state 1 -> 2 Jul 10 02:38:29 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:29 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:29 bsd-15 kernel: actor.state=d<ACTIVITY,AGGREGATION,SYNC>
Jul 10 02:38:29 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 02:38:29 bsd-15 kernel: partner.state=5<ACTIVITY,AGGREGATION>
Jul 10 02:38:29 bsd-15 kernel: maxdelay=0 Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu receive Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 02:38:30 bsd-15 kernel: actor.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
Jul 10 02:38:30 bsd-15 kernel: partner=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:30 bsd-15 kernel: partner.state=d<ACTIVITY,AGGREGATION,SYNC>
Jul 10 02:38:30 bsd-15 kernel: maxdelay=0 Jul 10 02:38:30 bsd-15 kernel: ql0: old pstate 5<ACTIVITY,AGGREGATION> Jul 10 02:38:30 bsd-15 kernel: ql0: new pstate 1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
Jul 10 02:38:30 bsd-15 kernel: ql1: lacp_sm_mux_timer: aggregator [(8000,00-0E-1E-04-2C-F0, 0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], pending 1 -> 0 Jul 10 02:38:30 bsd-15 kernel: ql1: collecting disabled Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 1 -> 2 Jul 10 02:38:30 bsd-15 kernel: ql1: collecting enabled Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 2 -> 3 Jul 10 02:38:30 bsd-15 kernel: ql1: enable distributing on aggregator [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], nports 0 -> 1 Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator:
Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1 Jul 10 02:38:30 bsd-15 kernel: active aggregator changed Jul 10 02:38:30 bsd-15 kernel: old (none) Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00 00-00-00,0000,0000,0000)] Jul 10 02:38:30 bsd-15 kernel: Set table 1 with 1 ports Jul 10 02:38:30 bsd-15 kernel: lacp_suppress_distributing Jul 10 02:38:30 bsd-15 kernel: ql1: marker transmit, port=15, sys=00:0e:1e:04:2c:f0, id=1 Jul 10 02:38:30 bsd-15 kernel: ql0: marker transmit, port=14, sys=00:0e:1e:04:2c:f0, id=1 Jul 10 02:38:30 bsd-15 kernel: ql1: mux_state 3 -> 4 Jul 10 02:38:30 bsd-15 kernel: ql1: lacpdu transmit Jul 10 02:38:30 bsd-15 kernel: ql0: marker response, port=14, sys=00:0e:1e:04:2c:f0, id=1 Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000F)
Jul 10 02:38:30 bsd-15 kernel: actor.state=7d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING,DEFAULTED>
Jul 10 02:38:30 bsd-15 kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
Jul 10 02:38:30 bsd-15 kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Jul 10 02:38:30 bsd-15 kernel: maxdelay=0 Jul 10 02:38:30 bsd-15 kernel: ql0: collecting enabled Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 2 -> 3 Jul 10 02:38:30 bsd-15 kernel: ql0: enable distributing on aggregator [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], nports 0 -> 1 Jul 10 02:38:30 bsd-15 kernel: lacp_select_active_aggregator:
Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(8000,00-0E-1E-08-05-20,01D3,0000,0000)], speed=10000000000, nports=1 Jul 10 02:38:30 bsd-15 kernel: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)], speed=10000000000, nports=1 Jul 10 02:38:30 bsd-15 kernel: active aggregator not changed Jul 10 02:38:30 bsd-15 kernel: new [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),(FFFF,00-00-00-00-00-00,0000,0000,0000)]
Jul 10 02:38:30 bsd-15 kernel: ql0: mux_state 3 -> 4 Jul 10 02:38:30 bsd-15 kernel: ql0: lacpdu transmit Jul 10 02:38:30 bsd-15 kernel: actor=(8000,00-0E-1E-04-2C-F0,0213,8000,000E)
Jul 10 02:38:30 bsd-15 kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Jul 10 02:38:30 bsd-15 kernel: partner=(8000,00-0E-1E-08-05-20,01D3,8000,000C)
Jul 10 02:38:30 bsd-15 kernel: partner.state=1d<ACTIVITY,AGGREGATION,SYNC,COLLECTING>
Jul 10 02:38:30 bsd-15 kernel: maxdelay=0 Jul 10 02:38:33 bsd-15 kernel: lacp_transit_expire ^C
Let me know if you need more info.
Thanks
Adarsh
-----Original Message-----
From: owner-freebsd-net at freebsd.org [mailto:owner-freebsd-net at freebsd.org] On Behalf Of Adarsh Joshi
Sent: Tuesday, July 10, 2012 10:08 AM
To: Andrew Boyer
Cc: freebsd-net at freebsd.org
Subject: RE: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance
Andrew,
Thanks for the reply.
The reason for my suspicion on the portflags is thus (extracted from the ifconfig output in my previous mail):
System 1:
Laggport: ql1 flags = 18 state = 7D
Laggport: ql0 flags = 1c state = 3D
System 2:
Laggport: ql1 flags = 1c state = 7D
Laggport: ql0 flags = 18 state = 3D
I should have explained my setup to you before. Here it is.
Both the ql0 interfaces of the 2 systems are connected using a single cable and ql1 interfaces of the 2 systems are connected using a single cable.
System 1 System 2
ql0 <=======> ql0
ql1 <=======> ql1
With this setup, I don't think it is possible for ports ql0 to talk to their partners (each other) and ql1 ports not getting a response from their partner and still get the lagg configuration I have posted.
I thought the portflags are dependent on the LACP state. But I see different flags for the same LACP state (For the state 7D, ql1 on system 1 shows flags = 18 and ql1 on system 2 shows flags = 1c).
Or is my understanding totally wrong?
I will send the LACP_DEBUG logs within the hour.
Thanks
Adarsh
From: Andrew Boyer [mailto:aboyer at averesystems.com]
Sent: Tuesday, July 10, 2012 5:57 AM
To: Adarsh Joshi
Cc: freebsd-net at freebsd.org
Subject: Re: lacp lagg port flags do not show correctly resulting in poor traffic distribution/performance
On Jul 9, 2012, at 8:38 PM, Adarsh Joshi wrote:
Hi,
I am trying to configure lacp lagg interfaces with 2 systems connected back to back as follows:
Ifconfig lagg0 create
Ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1 192.168.100.1 netmask 255.255.255.0
Sometimes, the lag interface comes up correctly but sometimes the laggport flags do not show properly. Instead of 1c<ACTIVE,COLLECTING,DISTRIBUTING>, it shows values of 18. I have seen similar issues reported on various forums with no solution.
Looking at the lagg driver code and reading the standard, I thought the laggport flags ( defined in if_lagg.h) are based on the LACP_STATE_BITS in file ieee8023ad_lacp.h. But the following ifconfig -v output does not make any sense to me.
My concern is that when all the interfaces show flags as 1c, the traffic is distributed across both the interfaces uniformly and I get aggregated throughput. If not, the traffic flows only on 1 interface.
Is this a bug? How do I solve this? Or am I doing something wrong?
I am using Free-BSD 9.0 release.
System 1:
# ifconfig -v lagg0
lag id: [(8000,00-0E-1E-08-05-20,0213,0000,0000),
(8000,00-0E-1E-04-2C-F0,0213,0000,0000)]
laggport: ql1 flags=18<COLLECTING,DISTRIBUTING> state=7D
[(8000,00-0E-1E-08-05-20,0213,8000,000F),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ql0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D
[(8000,00-0E-1E-08-05-20,0213,8000,000E),
(8000,00-0E-1E-04-2C-F0,0213,8000,000E)]
System 2:
# ifconfig -v lagg0
lag id: [(8000,00-0E-1E-04-2C-F0,0213,0000,0000),
(FFFF,00-00-00-00-00-00,0000,0000,0000)]
laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=7D
[(8000,00-0E-1E-04-2C-F0,0213,8000,000F),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ql0 flags=18<COLLECTING,DISTRIBUTING> state=3D
[(8000,00-0E-1E-04-2C-F0,0213,8000,000E),
(8000,00-0E-1E-08-05-20,0213,8000,000E)]
thanks
Adarsh
I don't think you have a port flags problem per se; the flags are correctly displaying the state of the lagg. Your problem is that your systems aren't negotiating the correct lagg configuration. Each tuple after the laggport represents the [(actor state),(partner state)]. Ports ql0 have been able to talk to their partners (each other). Neither ql1 port has seen a response from a partner, though.
You could try restarting the state machine on one box with 'ifconfig lagg0 laggproto lacp'. To see the negotiation you'll need to rebuild your kernel with '#define LACP_DEBUG 1' added to the top of sys/net/ieee802.3ad_lacp.c. Or upgrade to a newer stable snapshot that has the net.lacp_debug sysctl and turn it on.
Or just turn off LACP. What does it get you in this configuration?
Hope this helps,
Andrew
--------------------------------------------------
Andrew Boyer aboyer at averesystems.com<mailto:aboyer at averesystems.com>
________________________________
This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
_______________________________________________
freebsd-net at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
_______________________________________________
freebsd-net at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
This message and any attached documents contain information from QLogic Corporation or its wholly-owned subsidiaries that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.
More information about the freebsd-net
mailing list