zfs, iSCSI and volmode=dev

[Eugene M. Zheganin]

Hi,


I have an iSCSI production system that exports a large number of zvols 
as the iSCSI targets. System is running FreeBSD 11.0-RELEASE-p7 and 
initially all of the zvols were confugured with default volmode. I've 
read that it's recommended to use them in dev mode, so the system isn't 
bothered with all of these geom structures, so I've switched all of the 
zvols to dev mode, then I exported/imported the pools back. 
Surprisingly, the performance has fallen down like 10 times (200-300 
Mbits/sec against 3-4 Gbits/sec previously). After observing for 5 
minutes the ESXes trying to boot up, and doing this extremely slowly, I 
switched the volmode back to default, then again exported/imported the 
pools. The performance went back to normal.


So... why did this happen ? The result seems to be counter-intuitive. At 
least not obvious to me.


Thanks.

Eugene.