[iSCSI] Trying to reach max disk throughput

[Ben RUBSON]

Hello,

I'm facing something strange with iSCSI, I can't manage to reach the expected disk throughput using one (read or write) thread.



### Target : local disk throughput, one thread :

# dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 22.127838 secs (485244798 bytes/sec) - 462MB/s



### Initiator : network throughput to target, one thread :

# iperf -c 192.168.2.2 -t 30 -i 5 -P 1 -l 128KB
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec  19.5 GBytes  33.5 Gbits/sec
[  3]  5.0-10.0 sec  19.7 GBytes  33.9 Gbits/sec
[  3] 10.0-15.0 sec  19.6 GBytes  33.6 Gbits/sec
[  3] 15.0-20.0 sec  19.6 GBytes  33.7 Gbits/sec
[  3] 20.0-25.0 sec  19.8 GBytes  34.0 Gbits/sec
[  3] 25.0-30.0 sec  19.9 GBytes  34.2 Gbits/sec



### Initiator : network latency to target :

# ping -c 10 192.168.2.2 
64 bytes from 192.168.2.2: icmp_seq=0 ttl=64 time=0.025 ms
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=64 time=0.027 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=64 time=0.021 ms
64 bytes from 192.168.2.2: icmp_seq=4 ttl=64 time=0.020 ms
64 bytes from 192.168.2.2: icmp_seq=5 ttl=64 time=0.025 ms
64 bytes from 192.168.2.2: icmp_seq=6 ttl=64 time=0.022 ms
64 bytes from 192.168.2.2: icmp_seq=7 ttl=64 time=0.020 ms
64 bytes from 192.168.2.2: icmp_seq=8 ttl=64 time=0.022 ms
64 bytes from 192.168.2.2: icmp_seq=9 ttl=64 time=0.023 ms
round-trip min/avg/max/stddev = 0.020/0.023/0.027/0.002 ms



### Initiator : iscsi disk throughput :

## dd if=/dev/da8 of=/dev/null bs=$((128*1024)) count=81920
10737418240 bytes transferred in 34.731815 secs (309152234 bytes/sec) - 295MB/s

With 2 parallel dd jobs : 345MB/s
With 4 parallel dd jobs : 502MB/s



### Questions :

Why such a difference ?
Where are the 167MB/s (462-295) lost ?

All CPUs, on both sides, are above 90% idle during these tests.
I tried to increase net.inet.tcp.sendbuf_max, net.inet.tcp.recvbuf_max.
I also increased SOCKBUF_SIZE in iscsid.h and ctld.h.
And tried HTCP as the TCP algorithm.
But with no chance.

Any idea ?

Many thanks !

Ben