Abysmally slow write to geom class volume over network

[Michael Osipov]

Am 2016-10-12 um 19:08 schrieb Jan Bramkamp:
 >
 >
 > On 12/10/2016 17:25, Michael Osipov wrote:
 >> Am 2016-10-06 um 20:34 schrieb Michael Osipov:
 >>> Hi folks,
 >>>
 >>> I am experiecing a performance breakdown when writing to some geom 
class
 >>> volume over network. I have tried SCP/SFTP, SMB, nc, ggated/ggatec from
 >>> Windows 10 and FreeBSD 10.3-RELEASE and I receive at most 500 kB/s.
 >>> Reading is superfast as well as writing to a regular, non-geom volume,
 >>> network is fully saturated.
 >>>
 >>> I am on:
 >>> # uname -a
 >>> FreeBSD bsd1home 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11
 >>> 18:37:29 UTC 2016
 >>> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  i386
 >>>
 >>> Any idea what the problem is? How can I investigate on the penality in
 >>> performance?
 >>
 >> Seriously no one able to help?
 >
 > Not with the description you've given.

Probably, yes though I wasn't even sure where to start.

 > Please describe the network setup, how you access remote block devices,
 > what you mean by "geom volume".

Simply put, I have this:

$ uname -a
FreeBSD bsd1home 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 
18:37:29 UTC 2016 
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  i386

atapci0 at pci0:2:13:0:    class=0x018000 card=0x00051103 chip=0x00041103 
rev=0x03 hdr=0x00
     vendor     = 'HighPoint Technologies, Inc.'
     device     = 'HPT366/368/370/370A/372/372N'
     class      = mass storage
bfe0 at pci0:2:5:0:        class=0x020000 card=0x80a81043 chip=0x440114e4 
rev=0x01 hdr=0x00
     vendor     = 'Broadcom Corporation'
     device     = 'BCM4401 100Base-T'
     class      = network
     subclass   = ethernet

# camcontrol devlist | grep WDC
<WDC WD2000JB-00GVA0 08.02D08>     at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD2000JB-00FUA0 15.05R15>     at scbus0 target 1 lun 0 (ada1,pass1)
<WDC WD2000JB-00EVA0 15.05R15>     at scbus1 target 0 lun 0 (ada2,pass2)

# graid3 label -rv data ada0 ada1 ada2
Metadata value stored on ada0.
Metadata value stored on ada1.
Metadata value stored on ada2.
Done.

# newfs -j /dev/raid3/data
/dev/raid3/data: 381564.4MB (781443928 sectors) block size 32768, 
fragment size 4096
         using 610 cylinder groups of 626.22MB, 20039 blks, 80256 inodes.
         with soft updates
...
Using inode 4 in cg 0 for 33554432 byte journal
newfs: soft updates journaling set

# mount /dev/raid3/data /mnt

After the filesystem has been mounted, I have copied huge files, 
hundreds of megabytes each, via SFTP and nc(1) to that /mnt from another 
machine:

$ stat Image.bin
   File: 'Image.bin'
   Size: 654302880       Blocks: 638968     IO Block: 65536  regular file
Device: 3041c3c6h/809616326d    Inode: 562949953455390  Links: 1
Access: (0644/-rw-r--r--)  Uid: (197609/ mosipov)   Gid: (197121/ UNKNOWN)
Access: 2016-01-23 09:30:50.807257500 +0100
Modify: 2010-10-10 12:35:43.609375000 +0200
Change: 2010-10-10 12:35:43.609375000 +0200
  Birth: 2016-01-23 09:30:50.807257500 +0100

$ scp Image.bin 192.168.1.7:/mnt
Password for mosipov at bsd1home:
Image.bin                                       0% 5888KB  52.8KB/s 
3:19:57 ETA

Now let's copy to non-geom filesystem:
$ scp Image.bin 192.168.1.7:/usr/home/mosipov
Password for mosipov at bsd1home:
Password for mosipov at bsd1home:
Image.bin                                     100%  624MB  10.6MB/s   00:59

It it neither the network card nor the controller as you can see, 
non-geom is fast and local writes too:

# dd if=/dev/zero bs=1M count=10240 | pv -s 10G -treab --progress > 
/mnt/file.bin
10240+0 records out
10737418240 bytes transferred in 216.084333 secs (49690869 bytes/sec)
   10GiB 0:03:36 [47.4MiB/s] [47.4MiB/s] [==========================>] 100%

As if there is a bottleneck between socket read and geom write to FS.

Is that better?

Best regards,

Michael

PS: iSCSI won't be an option because the machine will be access either 
from Windows via SMB or SFTP, alternatively via DNLA (HTTP) from phones 
and TV.