FreeBSD 8.2 Release, ZFS + Samba, running out of memory

[Jeremy Chadwick]

On Thu, Feb 24, 2011 at 12:41:06AM +0100, Henner Heck wrote:
> 
> Here are my testresults:
> 
> *Disabling sendfile in smb.conf *(no patching yet)
> - free memory drops to 570M and stays there
> - copy speed from share 75MB/s, CPU load 50%
>     (why not 100% and higher speed?)

See below.

> *Patched kernel with patch 218795*
> - without sendfile  
>     Copying works.
> - with sendfile  
>    Copy speed often drops to zero after several hundred megabytes,
>      no more data is transmitted, the BSD machine is 100% idle and
> responsive.
>      After some time Total Commander claims that the file cannot be copied,
>      on retry it sometimes copies the file, but the problem will usually
> come up
>      with the same file again or one of the next files.
> 
> The speed in both cases is between 40-70MB/s, CPU load is 30-50%,
>     the free memory drops to about 250M, no running out of memory.

It's important to note that you're copying over the network.  Samba is
well-known throughout the community for not performing "extremely fast",
even on certain distributions of Linux.  The best way to gain speed
improvements is to make use of AIO.  I also noticed that your smb.conf
tunings may be incorrect for AIO, but you will need to adjust things and
try combinations to figure out what works best for you across the
network.

> *Other observations*
> A process named "intr" causes about 20% CPU load when copying.
> I wonder why.

I believe this is an interrupt handler thread in the kernel, which is
quite justified given what all is going on within the system during this
kind of operation.

> *Conclusions*
> When using sendfile the patch replaced a freezing system with unreliable
> copying. Without sendfile there seems to be no problem.
> 
> I am a bit dissapointed with the speed, since the CPU is not at all
> maxed out.
> The NIC is a PCIe Intel CT 1000 (I hadn't mentioned that before.)
> Any idea where the bottleneck might be?

Almost certainly Samba, but there is little-to-nothing one can do about
it.  If you want to verify it's Samba, enable ftpd on your system and do
FTP transfers instead (do both a GET and a PUT for testing speed in both
directions).

Good luck.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |