IB test report : 20121206

[Daichi GOTO]

Hi guys,

================================================================
IB test report. OS and IB cards:

	FreeBSD9-stable r243935
	MHQH19B-XTR	(ConnectX2)
	MCX353A-QCBT	(ConnectX2)

================================================================
TEST1 IBoIP UD (Unreliable Datagram)

Kernel configuration

	include         GENERIC
	ident           INFINIBAND
	options OFED	# Infiniband protocol
	options SDP	# Sockets Direct Protocol for infiniband
	device mlx4ib	# ConnectX Infiniband support
	device mlxen	# ConnectX Ethernet support
	device mthca	# Infinihost cards
	device ipoib	# IP over IB devices

make.conf

	WITH_OFED=              YES

rc.conf (server side)

	opensm_enable="YES"

IP setting

	ifconfig ib0 mtu 2044
	ifconfig ib0 inet 192.168.2.x netmask 0xffffff00

server: iperf bench

	# iperf -s
	------------------------------------------------------------
	Server listening on TCP port 5001
	TCP window size: 64.0 KByte (default)
	------------------------------------------------------------
	[  4] local 192.168.2.1 port 5001 connected with 192.168.2.2 port 12551
	[ ID] Interval       Transfer     Bandwidth
	[  4]  0.0-10.0 sec  6.00 GBytes  5.15 Gbits/sec

client: iperf bench

	# iperf -c 192.168.2.1 -i 1
	------------------------------------------------------------
	Client connecting to 192.168.2.1, TCP port 5001
	TCP window size: 33.1 KByte (default)
	------------------------------------------------------------
	[  3] local 192.168.2.2 port 12551 connected with 192.168.2.1 port 5001
	[ ID] Interval       Transfer     Bandwidth
	[  3]  0.0- 1.0 sec  47.2 MBytes   396 Mbits/sec
	[  3]  1.0- 2.0 sec  49.0 MBytes   411 Mbits/sec
	[  3]  2.0- 3.0 sec   260 MBytes  2.18 Gbits/sec
	[  3]  3.0- 4.0 sec   880 MBytes  7.39 Gbits/sec
	[  3]  4.0- 5.0 sec   926 MBytes  7.77 Gbits/sec
	[  3]  5.0- 6.0 sec   879 MBytes  7.38 Gbits/sec
	[  3]  6.0- 7.0 sec   464 MBytes  3.90 Gbits/sec
	[  3]  7.0- 8.0 sec   947 MBytes  7.95 Gbits/sec
	[  3]  8.0- 9.0 sec   945 MBytes  7.93 Gbits/sec
	[  3]  9.0-10.0 sec   742 MBytes  6.22 Gbits/sec
	[  3]  0.0-10.0 sec  6.00 GBytes  5.15 Gbits/sec

result:

	stable transmission. average 800MB/sec-1000MB/sec
	mtu 2044 is max.

================================================================
TEST2 IBoIP CM

Kernel configuration

	include         GENERIC
	ident           INFINIBAND
	options OFED	# Infiniband protocol
	options SDP	# Sockets Direct Protocol for infiniband
	options	IPOIB_CM#Use connect mode ipoib
	device mlx4ib	# ConnectX Infiniband support
	device mlxen	# ConnectX Ethernet support
	device mthca	# Infinihost cards
	device ipoib	# IP over IB devices

others

	the same

result

	unstable. terrible. average 0MB/sec-1000MB/sec

================================================================

Consideration from this first experimetns

	For OFED1.5.x of FreeBSD and ConnectX2 cards, 
	Unreliable Datagram mode IB over IP looks the only 
	way to use.

	CM is very unstable. It's odd. Needs improvement.
	(We must improve around mtu code? 2044 is dinky)

	I'm wondering that how many developers are using 
	ConnectX2 cards with CM OFED kernel.

================================================================

-- 
Daichi GOTO
Director | BSD Consulting Inc.
3F SS bldg. 3-4-2, Nishishinbashi, Minato, Tokyo
81-80-4174-6161 | daichi at bsd-consulting.jp