zfs, iSCSI and volmode=dev

[Edward Napierala]

2017-08-30 11:45 GMT+02:00 Eugene M. Zheganin <emz at norma.perm.ru>:

> Hi,
>
>
> I have an iSCSI production system that exports a large number of zvols as
> the iSCSI targets. System is running FreeBSD 11.0-RELEASE-p7 and initially
> all of the zvols were confugured with default volmode. I've read that it's
> recommended to use them in dev mode, so the system isn't bothered with all
> of these geom structures, so I've switched all of the zvols to dev mode,
> then I exported/imported the pools back. Surprisingly, the performance has
> fallen down like 10 times (200-300 Mbits/sec against 3-4 Gbits/sec
> previously). After observing for 5 minutes the ESXes trying to boot up, and
> doing this extremely slowly, I switched the volmode back to default, then
> again exported/imported the pools. The performance went back to normal.
>
>
> So... why did this happen ? The result seems to be counter-intuitive. At
> least not obvious to me.


I don't really have an answer - mav@ would be the best person to ask.  Based
on his description, "ZVOLs in GEOM mode don't support DPO/FUA cache control
bits, had to chunk large I/Os into MAXPHYS-sized pieces and go through
GEOM."
There also used to be so that TRIM was only supported in the "dev" mode, but
that changed a while ago.