Single-threaded bottleneck in geli

[Alan Somers]

I'm using geli, gmultipath, and ZFS on a large system, with hundreds of
drives.  What I'm seeing is that under at least some workloads, the overall
performance is limited by the single geom kernel process.  procstat and
kgdb aren't much help in telling exactly why this process is using so much
CPU, but it certainly must be related to the fact that over 15,000 IOPs are
going through that thread.  What can I do to improve this situation?  Would
it make sense to enable direct dispatch for geli?  That would hurt
single-threaded performance, but probably improve performance for highly
multithreaded workloads like mine.

Example top output:
  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   13 root         -8    -     0B    96K CPU46   46  82.7H  70.54%
geom{g_down}
   13 root         -8    -     0B    96K -        9  35.5H  25.32%
geom{g_up}

-Alan