samba+zfs

Kurt Touet ktouet at gmail.com
Wed Nov 9 05:59:56 UTC 2011 

On Tue, Nov 8, 2011 at 8:16 PM, Dan The Man <dan at sunsaturn.com> wrote:
>
> On Tue, 8 Nov 2011, Kurt Touet wrote:
>
>> On Tue, Nov 8, 2011 at 3:32 AM, Dan The Man <dan at sunsaturn.com> wrote:
>>>
>>>
>>> Sorry I meant it was running fine on beta3 and 8.2 stable, and NOT RC1:
>>> asterisk:~# uname -a
>>> FreeBSD asterisk.sunsaturn.com 9.0-RC1 FreeBSD 9.0-RC1 #0: Mon Oct 31
>>> 19:46:53 CDT 2011
>>> droot at asterisk.sunsaturn.com:/usr/obj/usr/src/sys/MYKERNEL
>>>  amd64
>>> asterisk:~#
>>>
>>>
>>>
>>> Dan.
>>>
>>>
>>> --
>>> Dan The Man
>>> CTO/ Senior System Administrator
>>> Websites, Domains and Everything else
>>> http://www.SunSaturn.com
>>> Email: Dan at SunSaturn.com
>>>
>>> On Tue, 8 Nov 2011, Dan The Man wrote:
>>>
>>>>
>>>> Ok here is some specs: this been running fine on 8.2 stable and i was
>>>> sure
>>>> it was running fine on RC1 as well. I did some testing against samba 34
>>>> 35
>>>> and 36 in the ports collection all with the same slow write problems.
>>>>
>>>> I did further testing mounting drive in question with NFS and it did not
>>>> suffer the same problem, so it seems just samba related here, where
>>>> samba
>>>> would actually outperform my NFS mount before, now its taking 10x as
>>>> long
>>>> to write anything.
>>>>
>>>> This is really most simplistic setup I have, all I want to do is map a
>>>> network drive at the house read/write so my laptop, desktop etc all have
>>>> access to it. I have played with all the smb.conf options, and can't
>>>> seem
>>>> to find where the issue is, further research suggests others are
>>>> experiencing same problems with beta3 from following forum post:
>>>>
>>>> http://forums.freebsd.org/showthread.php?t=27300
>>>>
>>>> Hardware this is running on: I beleive a 4 year old amd chip and board,
>>>> with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 SSD
>>>> as
>>>> UFS boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap
>>>> hitachi drives for my storage space,which is mirrored nightly with rsync
>>>> with another duplicate machine(cause I know someone is going to say why
>>>> not
>>>> use raid5-raidz)
>>>>
>>>> Network specs: machine currently has dedicated IPV4 and gif0 tunneled
>>>> IPV6
>>>> addresses to he.net.
>>>>
>>>> I've played with nearly every option in smb.conf disabling, enabling etc
>>>> and can't seem to find the issue: here are my current config file
>>>> settings
>>>> on machine that could apply to samba:
>>>>
>>>> asterisk:~# cat /boot/loader.conf
>>>> autoboot_delay="5"
>>>> accf_data_load="YES"
>>>> aio_load="YES"
>>>> zfs_load="YES"
>>>> kern.maxbcache=64M
>>>> kern.ipc.maxpipekva=4M
>>>>
>>>> vfs.zfs.prefetch_disable=1
>>>> vm.kmem_size="1844M"
>>>> vfs.zfs.arc_min="1024M"
>>>> vfs.zfs.arc_max="1536M"
>>>> vfs.zfs.vdev.min_pending=2
>>>> vfs.zfs.vdev.max_pending=8
>>>> vfs.zfs.txg.timeout=5
>>>> vfs.zfs.zil_disable="1"
>>>> ahci_load="YES"
>>>> asterisk:~#
>>>>
>>>> asterisk:~# cat /usr/local/etc/smb.conf
>>>> # Global parameters
>>>> [global]
>>>>       workgroup = HOME
>>>>       netbios name = ASTERISK
>>>>       server string = "Primary backups"
>>>>       interfaces = sk0
>>>>       #smb ports = 139
>>>>       #security = USER
>>>>       security = SHARE
>>>>       encrypt passwords = Yes
>>>>       #socket options = TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536
>>>>       domain master = no
>>>>       wins support = yes
>>>>       guest account = root
>>>>       socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
>>>>       use sendfile = no
>>>>       level2 oplocks = True
>>>>       read raw = no
>>>>       write cache size = 262144
>>>>       min receivefile size = 16384
>>>>       aio read size = 16384
>>>>       aio write size = 16384
>>>>       aio write behind = yes
>>>>       dns proxy = no
>>>>       max log size = 50
>>>>       #log file = /dev/null
>>>>       log file = /var/log/samba.log
>>>>       debug level = 1
>>>>       syslog = 0
>>>>
>>>> [data]
>>>>       comment = "Primary backups"
>>>>       path = /data/public
>>>>       read only = No
>>>>       guest ok = Yes
>>>>       valid users = root
>>>> asterisk:~# asterisk:~# cat /etc/sysctl.conf
>>>> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith
>>>> Exp $
>>>> #
>>>> #  This file is read when going to multi-user and its contents piped
>>>> thru
>>>> #  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for
>>>> details.
>>>> #
>>>>
>>>> # Uncomment this to prevent users from seeing information about
>>>> processes
>>>> that
>>>> # are being run under another UID.
>>>> #security.bsd.see_other_uids=0
>>>>
>>>> #raise file descriptors on the system
>>>> kern.maxfiles=204916
>>>> kern.maxfilesperproc=204916
>>>>
>>>> #raise sockets we can accept
>>>> kern.ipc.somaxconn=32768
>>>>
>>>> #http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html
>>>> kern.ipc.maxsockbuf=16777216
>>>> net.inet.tcp.rfc1323=1
>>>> net.inet.tcp.sendbuf_max=16777216
>>>> net.inet.tcp.recvbuf_max=16777216
>>>> net.inet.tcp.sendspace=65536
>>>> net.inet.tcp.recvspace=131072
>>>>
>>>> #NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations
>>>> net.inet.icmp.icmplim=500
>>>> kern.ipc.nmbjumbop=192000
>>>> kern.ipc.nmbclusters=229376
>>>> kern.ipc.maxsockets=204800
>>>> net.inet.tcp.maxtcptw=163840
>>>> #also add following to /boot/loader.conf
>>>> #vm.kmem_size=1844M
>>>> #kern.maxbcache=64M
>>>> #kern.ipc.maxpipekva=4M
>>>>
>>>> #default setting of net.inet.ip.portrange.first is to low, causing us
>>>> problems with bind
>>>> net.inet.ip.portrange.last=65535
>>>> net.inet.ip.portrange.first=1024
>>>>
>>>> #DOS protection
>>>> net.inet.tcp.msl=7500
>>>> net.inet.tcp.blackhole=2
>>>> net.inet.udp.blackhole=1
>>>> net.inet.icmp.icmplim=50
>>>> net.inet.ip.accept_sourceroute=0
>>>> net.inet.ip.sourceroute=0
>>>>
>>>> #some stuff for samba
>>>> kern.ipc.nmbclusters=32768
>>>> kern.maxvnodes=800000
>>>> net.inet.tcp.delayed_ack=0
>>>> net.inet.tcp.inflight.enable=0
>>>> net.inet.tcp.path_mtu_discovery=0
>>>> net.inet.tcp.recvbuf_auto=1
>>>> net.inet.tcp.recvbuf_inc=524288
>>>> net.inet.tcp.sendbuf_auto=1
>>>> net.inet.tcp.sendbuf_inc=524288
>>>> net.inet.udp.maxdgram=57344
>>>> net.inet.udp.recvspace=65535
>>>> net.local.stream.recvspace=65535
>>>> net.local.stream.sendspace=65535
>>>> net.inet.tcp.mssdflt=1460
>>>>
>>>> #IPSEC
>>>> net.inet.ip.forwarding=1
>>>> net.inet6.ip6.forwarding=1
>>>> kern.module_path=/boot/kernel;/boot/modules;/usr/local/modules
>>>>
>>>>
>>>> #NFS--not concerned with data integrity when playing mostly already
>>>> stored
>>>> movies
>>>> vfs.nfsrv.async=1
>>>>
>>>> #JAIL
>>>> #i like to use ping etc inside jail!
>>>> security.jail.allow_raw_sockets=1
>>>> asterisk:~#
>>>>
>>>>
>>>> Here are logs of me trying to mux a DTS mkv file from samba.log on debug
>>>> level 10, I get the following over and over again:
>>>>
>>>> [2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data)
>>>> [2011/11/08 03:24:00.067981,  3] smbd/process.c:1466(switch_message)
>>>>  switch message SMBwriteX (pid 64308) conn 0x805008450
>>>> [2011/11/08 03:24:00.067990,  4] smbd/uid.c:345(change_to_user)
>>>>  Skipping user change - already user
>>>> [2011/11/08 03:24:00.068001, 10]
>>>> locking/locking.c:120(strict_lock_default)
>>>>  is_locked: optimisation - exclusive oplock on file
>>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>>>> [2011/11/08 03:24:00.068010, 10]
>>>> locking/locking.c:162(strict_lock_default)
>>>>  strict_lock_default: flavour = WINDOWS_LOCK brl start=83431665
>>>> len=65536
>>>> unlocked for fnum 49966 file
>>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>>>> [2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfile)
>>>>  default_sys_recvfile: from = 33, to = 39, offset=83431665, count =
>>>> 65536
>>>> [2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file)
>>>>  real_write_file (torrent_downloads_finished/Point.Break.1991.720p
>>>> (1).mkv): pos = 83431665, size = 65536, returned 65536
>>>> [2011/11/08 03:24:00.069013,  3] smbd/reply.c:4639(reply_write_and_X)
>>>>  writeX fnum=49966 num=65536 wrote=65536
>>>> [2011/11/08 03:24:00.069038, 10]
>>>> lib/util_sock.c:516(read_smb_length_return_keepalive)
>>>>  got smb length of 65600
>>>> [2011/11/08 03:24:00.069052, 10]
>>>> smbd/reply.c:4459(is_valid_writeX_buffer)
>>>>  is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536
>>>> [2011/11/08 03:24:00.069062,  6] smbd/process.c:1659(process_smb)
>>>>  got message type 0x0 of len 0x3f
>>>> [2011/11/08 03:24:00.069072,  3] smbd/process.c:1661(process_smb)
>>>>  Transaction 15398 of length 67 (65536 toread)
>>>> [2011/11/08 03:24:00.069081,  5] lib/util.c:332(show_msg)
>>>> [2011/11/08 03:24:00.069087,  5] lib/util.c:342(show_msg)
>>>>  size=63
>>>>  smb_com=0x2f
>>>>  smb_rcls=0
>>>>  smb_reh=0
>>>>  smb_err=0
>>>>  smb_flg=24
>>>>  smb_flg2=51207
>>>>  smb_tid=1
>>>>  smb_pid=65279
>>>>  smb_uid=0
>>>>  smb_mid=36032
>>>>  smt_wct=14
>>>>  smb_vwv[ 0]=  255 (0xFF)
>>>>  smb_vwv[ 1]=57054 (0xDEDE)
>>>>  smb_vwv[ 2]=49966 (0xC32E)
>>>>  smb_vwv[ 3]= 4337 (0x10F1)
>>>>  smb_vwv[ 4]= 1274 (0x4FA)
>>>>  smb_vwv[ 5]=65535 (0xFFFF)
>>>>  smb_vwv[ 6]=65535 (0xFFFF)
>>>>  smb_vwv[ 7]=    0 (0x0)
>>>>  smb_vwv[ 8]=    0 (0x0)
>>>>  smb_vwv[ 9]=    1 (0x1)
>>>>  smb_vwv[10]=    0 (0x0)
>>>>  smb_vwv[11]=   64 (0x40)
>>>>  smb_vwv[12]=    0 (0x0)
>>>>  smb_vwv[13]=    0 (0x0)
>>>>  smb_bcc=0
>>>> [2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data)
>>>> [2011/11/08 03:24:00.069170,  3] smbd/process.c:1466(switch_message)
>>>>  switch message SMBwriteX (pid 64308) conn 0x805008450
>>>> [2011/11/08 03:24:00.069179,  4] smbd/uid.c:345(change_to_user)
>>>>  Skipping user change - already user
>>>> [2011/11/08 03:24:00.069188, 10]
>>>> locking/locking.c:120(strict_lock_default)
>>>>  is_locked: optimisation - exclusive oplock on file
>>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>>>> [2011/11/08 03:24:00.069197, 10]
>>>> locking/locking.c:162(strict_lock_default)
>>>>  strict_lock_default: flavour = WINDOWS_LOCK brl start=83497201
>>>> len=65536
>>>> unlocked for fnum 49966 file
>>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>>>> [2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfile)
>>>>  default_sys_recvfile: from = 33, to = 39, offset=83497201, count =
>>>> 65536
>>>> [2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file)
>>>>  real_write_file (torrent_downloads_finished/Point.Break.1991.720p
>>>> (1).mkv): pos = 83497201, size = 65536, returned 65536
>>>> [2011/11/08 03:24:00.070004,  3] smbd/reply.c:4639(reply_write_and_X)
>>>>  writeX fnum=49966 num=65536 wrote=65536
>>>> [2011/11/08 03:24:00.070030, 10]
>>>> lib/util_sock.c:516(read_smb_length_return_keepalive)
>>>>  got smb length of 65600
>>>> [2011/11/08 03:24:00.070044, 10]
>>>> smbd/reply.c:4459(is_valid_writeX_buffer)
>>>>  is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536
>>>> [2011/11/08 03:24:00.070053,  6] smbd/process.c:1659(process_smb)
>>>>  got message type 0x0 of len 0x3f
>>>> [2011/11/08 03:24:00.070063,  3] smbd/process.c:1661(process_smb)
>>>>  Transaction 15399 of length 67 (65536 toread)
>>>> [2011/11/08 03:24:00.070072,  5] lib/util.c:332(show_msg)
>>>> [2011/11/08 03:24:00.070077,  5] lib/util.c:342(show_msg)
>>>>  size=63
>>>>  smb_com=0x2f
>>>>  smb_rcls=0
>>>>  smb_reh=0
>>>>  smb_err=0
>>>>  smb_flg=24
>>>>  smb_flg2=51207
>>>>  smb_tid=1
>>>>  smb_pid=65279
>>>>  smb_uid=0
>>>>  smb_mid=36102
>>>>  smt_wct=14
>>>>  smb_vwv[ 0]=  255 (0xFF)
>>>>  smb_vwv[ 1]=57054 (0xDEDE)
>>>>  smb_vwv[ 2]=49966 (0xC32E)
>>>>  smb_vwv[ 3]= 4337 (0x10F1)
>>>>  smb_vwv[ 4]= 1275 (0x4FB)
>>>>  smb_vwv[ 5]=65535 (0xFFFF)
>>>>  smb_vwv[ 6]=65535 (0xFFFF)
>>>>  smb_vwv[ 7]=    0 (0x0)
>>>>  smb_vwv[ 8]=    0 (0x0)
>>>>  smb_vwv[ 9]=    1 (0x1)
>>>>  smb_vwv[10]=    0 (0x0)
>>>>  smb_vwv[11]=   64 (0x40)
>>>>  smb_vwv[12]=    0 (0x0)
>>>>  smb_vwv[13]=    0 (0x0)
>>>>  smb_bcc=0
>>>>
>>>>
>>>> Hopefully maybe someone can shine some light on this....
>>>>
>>>>
>>>> Dan.
>>>>
>>>>
>>>> --
>>>> Dan The Man
>>>> CTO/ Senior System Administrator
>>>> Websites, Domains and Everything else
>>>> http://www.SunSaturn.com
>>>> Email: Dan at SunSaturn.com
>>>>
>>>> On Fri, 28 Oct 2011, Garrett Cooper wrote:
>>>>
>>>>> On Thu, Oct 27, 2011 at 6:42 PM, Dan <dan at sunsaturn.com> wrote:
>>>>>>
>>>>>>
>>>>>> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs
>>>>>> its taking over an hour to just mux in things like DTS english, where
>>>>>> it
>>>>>> was
>>>>>> 15 minutes on beta3.
>>>>>
>>>>> Hi Dan,
>>>>> - Can you do more deterministic / scientific benchmarks?
>>>>> - Did you upgrade Samba?
>>>>> - What is your system's operating hardware profile?
>>>>> Thanks!
>>>>> -Garrett
>>>>>
>>
>> I had noticed a similar problem with doing large writes from my
>> windows workstation to my freebsd ZFS array.  Previously I had been
>> able to write at around 30MB/s (limited by a slow SATA controller),
>> and those speeds had dropped to 3-5MB/s with long pauses between
>> writes to the array (monitoring with zpool iostat).   However, I just
>> updated to stable/9 r227357 and the issue seems to have gone away; I'm
>> back up to 30MB/s writes.
>>
>> Hope you find the same sort of thing,
>> -kurt
>>
>
> Yep your right, on RC2 my samba ZFS write speeds are back.
> Thanks for heads up Kurt.
>
>
> Dan.
>
>
> --
> Dan The Man
> CTO/ Senior System Administrator
> Websites, Domains and Everything else
> http://www.SunSaturn.com
> Email: Dan at SunSaturn.com
>


I may have spoken too soon.  With the speeds back up to normal, I
moved a bunch of content that I'd been storing temporarily on my
workstation to the server.  After a few hours and moving 40gb+ of
content (not all at once), I noticed the same write problems cropping
up again.  I rebooted the server and speeds came back to normal.

The oddity that I notice is that the array seems to be delaying writes
to a single disc in my raidz1 vdev, and then writing out very slowly
to it (1-2MB/s).   The drive that it is doing it with has absolutely
zero read/write/seek errors in SMART, and works just fine after the
box is rebooted (so it shouldn't be a drive issue).  The only thing
interesting about this drive is that it replaced a failed drive a few
weeks ago.

Is anyone else seeing problems like this with samba/zfs ?    Perhaps
it's not exclusive to samba, either?
-kurt

More information about the freebsd-current mailing list