Samba read speed performance tuning

[Gary Gatten]

It MAY make a big diff, but make sure during your tests you use unique files or flush the cache or you'll me testing cache speed and not disk speed.

----- Original Message -----
From: owner-freebsd-questions at freebsd.org <owner-freebsd-questions at freebsd.org>
To: freebsd-net at freebsd.org <freebsd-net at freebsd.org>; freebsd-questions at freebsd.org <freebsd-questions at freebsd.org>; FreeBSD-STABLE Mailing List <freebsd-stable at freebsd.org>; freebsd-performance at freebsd.org <freebsd-performance at freebsd.org>
Sent: Fri Mar 19 20:28:02 2010
Subject: Re: Samba read speed performance tuning

On Fri, Mar 19, 2010 at 11:14 PM, Dan Naumov <dan.naumov at gmail.com> wrote:
> On a FreeBSD 8.0-RELEASE/amd64 system with a Supermicro X7SPA-H board
> using an Intel gigabit nic with the em driver, running on top of a ZFS
> mirror, I was seeing a strange issue. Local reads and writes to the
> pool easily saturate the disks with roughly 75mb/s throughput, which
> is roughly the best these drives can do. However, working with Samba,
> writes to a share could easily pull off 75mb/s and saturate the disks,
> but reads off a share were resulting in rather pathetic 18mb/s
> throughput.
>
> I found a threadon the FreeBSD forums
> (http://forums.freebsd.org/showthread.php?t=9187) and followed the
> suggested advice. I rebuilt Samba with AIO support, kldloaded the aio
> module and made the following changes to my smb.conf
>
> From:
> socket options=TCP_NODELAY
>
> To:
> socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
> min receivefile size=16384
> use sendfile=true
> aio read size = 16384
> aio write size = 16384
> aio write behind = true
> dns proxy = no[/CODE]
>
> This showed a very welcome improvement in read speed, I went from
> 18mb/s to 48mb/s. The write speed remained unchanged and was still
> saturating the disks. Now I tried the suggested sysctl tunables:
>
> atombsd# sysctl net.inet.tcp.delayed_ack=0
> net.inet.tcp.delayed_ack: 1 -> 0
>
> atombsd# sysctl net.inet.tcp.path_mtu_discovery=0
> net.inet.tcp.path_mtu_discovery: 1 -> 0
>
> atombsd# sysctl net.inet.tcp.recvbuf_inc=524288
> net.inet.tcp.recvbuf_inc: 16384 -> 524288
>
> atombsd# sysctl net.inet.tcp.recvbuf_max=16777216
> net.inet.tcp.recvbuf_max: 262144 -> 16777216
>
> atombsd# sysctl net.inet.tcp.sendbuf_inc=524288
> net.inet.tcp.sendbuf_inc: 8192 -> 524288
>
> atombsd# sysctl net.inet.tcp.sendbuf_max=16777216
> net.inet.tcp.sendbuf_max: 262144 -> 16777216
>
> atombsd# sysctl net.inet.tcp.sendspace=65536
> net.inet.tcp.sendspace: 32768 -> 65536
>
> atombsd# sysctl net.inet.udp.maxdgram=57344
> net.inet.udp.maxdgram: 9216 -> 57344
>
> atombsd# sysctl net.inet.udp.recvspace=65536
> net.inet.udp.recvspace: 42080 -> 65536
>
> atombsd# sysctl net.local.stream.recvspace=65536
> net.local.stream.recvspace: 8192 -> 65536
>
> atombsd# sysctl net.local.stream.sendspace=65536
> net.local.stream.sendspace: 8192 -> 65536
>
> This improved the read speeds a further tiny bit, now I went from
> 48mb/s to 54mb/s. This is it however, I can't figure out how to
> increase Samba read speed any further. Any ideas?


Oh my god... Why did noone tell me how much of an enormous performance
boost vfs.zfs.prefetch_disable=0 (aka actually enabling prefetch) is.
My local reads off the mirror pool jumped from 75mb/s to 96mb/s (ie.
they are now nearly 25% faster than reading off an individual disk)
and reads off a Samba share skyrocketed from 50mb/s to 90mb/s.

By default, FreeBSD sets vfs.zfs.prefetch_disable to 1 on any i386
systems and on any amd64 systems with less than 4GB of avaiable
memory. My system is amd64 with 4gb ram, but integrated video eats
some of that, so the autotuning disabled the prefetch. I had read up
on it and a fair amount of people seemed to have performance issues
caused by having prefetch enabled and get better results with it
turned off, in my case however, it seems that enabling it gave a
really solid boost to performance.


- Sincerely
Dan Naumov
_______________________________________________
freebsd-questions at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"





<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
 and may contain information that is privileged and/or confidential.
 If you are not the intended recipient, you are hereby notified that
 any review, use, dissemination, disclosure or copying of this email
 and its attachments, if any, is strictly prohibited.  If you have
 received this email in error, please immediately notify the sender by
 return email and delete this email from your system."
</font>