Single-threaded bottleneck in geli

[Alan Somers]

I might give this a shot.  What is the best way tell if geli ought to use
direct dispatch?  Is there a generic "are crypto instructions available"
macro that would cover aesni as well as other platform-specific
instructions?

-Alan

On Sat, Jul 4, 2020, 2:55 PM Paweł Jakub Dawidek <pawel at dawidek.net> wrote:

> Direct dispatch would be great for geli, especially that geli can use own
> (multiple) threads when necessary (eg. using crypto cards). With AES-NI you
> could go straight to the disk.
>
> --
> Paweł Jakub Dawidek
>
>
>
> > On Jul 3, 2020, at 13:22, Alan Somers <asomers at freebsd.org> wrote:
> >
> > I don't.  What I meant was that a single thread (geom) is limiting the
> > performance of the system overall.  I'm certain, based on top, gstat, and
> > zpool iostat, that geom is the limiting factor on this system.
> > -Alan
> >
> >> On Fri, Jul 3, 2020 at 2:18 PM Paweł Jakub Dawidek <pawel at dawidek.net>
> >> wrote:
> >>
> >> Hi Alan,
> >>
> >> why do you think it will hurt single-threaded performance?
> >>
> >> --
> >> Paweł Jakub Dawidek
> >>
> >>
> >>
> >>>> On Jul 3, 2020, at 12:30, Alan Somers <asomers at freebsd.org> wrote:
> >>>
> >>> I'm using geli, gmultipath, and ZFS on a large system, with hundreds
> of
> >>> drives.  What I'm seeing is that under at least some workloads, the
> >> overall
> >>> performance is limited by the single geom kernel process.  procstat and
> >>> kgdb aren't much help in telling exactly why this process is using so
> >> much
> >>> CPU, but it certainly must be related to the fact that over 15,000 IOPs
> >> are
> >>> going through that thread.  What can I do to improve this situation?
> >> Would
> >>> it make sense to enable direct dispatch for geli?  That would hurt
> >>> single-threaded performance, but probably improve performance for
> highly
> >>> multithreaded workloads like mine.
> >>>
> >>> Example top output:
> >>> PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU
> COMMAND
> >>>  13 root         -8    -     0B    96K CPU46   46  82.7H  70.54%
> >>> geom{g_down}
> >>>  13 root         -8    -     0B    96K -        9  35.5H  25.32%
> >>> geom{g_up}
> >>>
> >>> -Alan
> >>> _______________________________________________
> >>> freebsd-geom at freebsd.org mailing list
> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-geom
> >>> To unsubscribe, send any mail to "freebsd-geom-unsubscribe at freebsd.org
> "
> >>
> >>
> > _______________________________________________
> > freebsd-geom at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-geom
> > To unsubscribe, send any mail to "freebsd-geom-unsubscribe at freebsd.org"
>
>