sa(4) 9.2->10.1, nsa0.0: request ptr 0x803135040 is not on a page boundary; cannot split request

[Kenneth D. Merry]

On Sat, Oct 25, 2014 at 20:53:50 +0300, Konstantin Belousov wrote:
> On Fri, Oct 24, 2014 at 05:07:26PM -0600, Kenneth D. Merry wrote:
> > On Thu, Oct 23, 2014 at 20:53:06 +0200, Harald Schmalzbauer wrote:
> > >  Hello,
> > > 
> > > I read about the changes in sa(4) regarding large-block-split changes
> > > and the transitional 'kern.cam.sa.allow_io_split' workarround.
> > > 
> > > I'm using bacula (7.0.5) and my previous neccessarry multi-blocking
> > > adjustmets like "Minimum block size = 2097152" obviously didn't work
> > > with FreebSD 10.1 anymore.
> > > Good news is, they are not needed any more!
> > > With the default of 126 blocks (64512) I get 60-140MB/s with btape(8)'s
> > > speed test on my LTO4 (HH) drive and another quick test showed that
> > > using mbuffer(1) for zfs(8) 'send' isn't needed anymore (| dd
> > > of=/dev/nsa0 bs=64512 seems to max out LTO4 speed). [with FreeBSD 9 the
> > > transfer rates were some magnitudes lower with these block size settings!]
> > > 
> > > Not so good news is, that bacula can't read the tape's label.
> > > 'Labeling a tape (with 'label' at bconsole(8) or btape(8)) is
> > > successful, and btape(8)'s 'readlabel' partially displays the correct
> > > label, but not the very beginning of the label:
> > > Volume Label:
> > > Id : **error**VerNo
> > > ?rest OK
> > > 
> > > While it should read:
> > > Volume Label:
> > > Id : Bacula 1.0 immortal
> > > VerNo : 11
> > > ?
> > > 
> > > When btape(8) starts to read the label, the _subject's error is reported_:
> > > *nsa0.0: request ptr 0x803135040 is not on a page boundary; cannot split
> > > request*
> > 
> > What blocksize are you using with btape(8)?
> > 
> > What kind of controller are you using?
> > 
> > The reason you get that error message is that the sa(4) driver goes through
> > physio(9) to get buffers from userland into the kernel.  physio(9) relies
> > on the vmapbuf()/vunmapbuf() routines to map buffers in and out of the
> > kernel.
> > 
> > vmapbuf() operates with a page granularity.  The address to be mapped has
> > to start on a page boundary.  It also uses kernel virtual address segments
> > that are MAXPHYS in size.  On x86 boxes at least, MAXPHYS is 128KB.
> > 
> > So if you use a blocksize of 128KB, but pass in a pointer that doesn't
> > start on a page boundary, vmapbuf() will have to map 33 pages instead of
> > 32.  In your case, it will have to start at page address 0x803135000, and
> > will need 33 4KB pages, which is greater than 128KB.
> I want to disable unaligned physio at all.
> 
> See https://reviews.freebsd.org/D888 for yet another case where this beats.
> 
> Obvious thing which stops us from doing this is binary compatibility.
> I need some form of wide support to make this change.

I don't see how we can remove support for non-page-aligned I/O through the
various block and character devices.  It would break all kinds of things, and
Bacula is just the start.

Perhaps another way to handle it would be that requests with unaligned
buffers would just go through a non-optimal path.  We could just malloc
buffers and copy the data in and out.

If the application sends in aligned buffers, it can go through the optimal
path.

Ken
-- 
Kenneth Merry
ken at FreeBSD.ORG