Bhyve storage improvements

Alexander Motin mav at
Fri Mar 27 17:43:14 UTC 2015

On 27.03.2015 18:47, John Nielsen wrote:
> Does anyone have plans (or know about any) to implement virtio-scsi support in bhyve? That API does support TRIM and should retain most or all of the low-overhead virtio goodness.

I was thinking about that (not really a plans yet, just some thoughts),
but haven't found a good motivation and understanding of whole possible

I am not sure it worth to emulate SCSI protocol in addition to already
done ATA in ahci-hd and simple block in virtio-blk just to get another,
possibly faster then AHCI, block storage with TRIM/UNMAP.  Really good
SCSI disk emulation in CTL in kernel takes about 20K lines of code. It
is pointless to duplicate it, and may be complicated for administration
to just interface to it.  Indeed I've seen virtio-blk being faster then
ahci-hd in some tests, but those tests were highly synthetic.  I haven't
tested it on real workloads, but I have feeling that real difference may
be not that large.  If somebody wants to check -- more benchmarks are
highly welcome!  From the theoretical side I'd like to notice that both
ATA and SCSI protocols on guests go through additional ATA/SCSI
infrastructure (CAM in FreeBSD), absent in case pure block virtio-blk,
so they have some more overhead by definition.

Main potential benefit I see from using virtio-scsi is a possibility to
pass through to client not a block device, but some real SCSI device. It
can be some local DVD writer, or remote iSCSI storage. The last would be
especially interesting for large production installations. But the main
problem I see here is booting. To make user-level loader boot the kernel
from DVD or iSCSI, bhyve has to implement its own SCSI initiator, like
small second copy of CAM in user-level. Booting kernel from some other
local block storage and then attaching to remote iSCSI storage for data
can be much easier, but it is not convenient. It is possible to nt
connect to iSCSI directly from user-level, but to make kernel CAM do it,
and then make CAM provide both block layer for booting and SCSI layer
for virtio-scsi, but I am not sure that it is very good from security
point to make host system to see virtual disks. Though may be it could
work if CAM could block kernel/GEOM access to them, alike it is done for
ZVOLs now, supporting "geom" and "dev" modes. Though that complicates
CAM and the whole infrastructure.

Alexander Motin

More information about the freebsd-virtualization mailing list