Abysmally slow write to geom class volume over network

[Fabian Keil]

Michael Osipov <1983-01-06 at gmx.net> wrote:

> Am 2016-10-14 um 11:43 schrieb Fabian Keil:
> > Michael Osipov <1983-01-06 at gmx.net> wrote:
> >  
> >> Am 2016-10-13 um 03:41 schrieb John-Mark Gurney:  
> >>> Michael Osipov wrote this message on Wed, Oct 12, 2016 at 20:54
> >>> +0200:  
> >>>> As if there is a bottleneck between socket read and geom write to
> >>>> FS.
> >>>>
> >>>> Is that better?  
> >>>
> >>> Have you run gstat on the system to see if there is an IO bottle
> >>> neck?  Since you are using graid3, you want to look to see if
> >>> it's %busy is ~100, while the underlying components are not.  
> >>
> >> This is hardly impossible because as soon as I start some SFTP
> >> transfer, all of my SSH sessions free or receive connetion
> >> timeout/abort.  Doing a SFTP from FreeBSD to FreeBSD gives me on both
> >> physical disks and RAID3 volume a busy of zero to one perfect. In
> >> other terms, the drives are bored.  
> >
> > Try checking the FAIL and SLEEP columns in the "vmstat -z" output.  
> 
> I assume that you expect a rise on those numbers. I have made several 
> runs. Rebooted the machine and then started SFTP transfer. After seconds 
> my SSH sessions locked up. The transfer was aborted manually after 10 
> minutes which should have saturated the entire connection. After that, I 
> reran vmstat -z, no or minimal rise in FAIL and SLEEP.

IIRC the SLEEP column only showns currently sleeping requests,
therefore you may want to run "vmstat -z" multiple times while
the transfer is ongoing. Having said that, a custom DTrace script
would probably be a better tool to diagnose the issue anyway.

> Interesting to say that this happens if is is a UFS volume on 
> gconcat/graid3/gvinum/gstripe configuration. Regular gpart with GPT has 
> no performance penalty. Additionally, it is not limited to SSH but 
> virtually everything with sockets: nc, ggate, smb.
> 
> > This could be related to:
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209680#c2  
> 
> It pretty much sounds like it, though I do not use ipfw, pf or any NAT 
> stuff. I will try your first patch and let you know.
> 
> Do you want me to add my usecase to the issue?

If the patch helps, that could be useful once a committer
finds the time to look at the PR.

Fabian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-geom/attachments/20161015/5098585f/attachment.sig>