Patch: sym(4) "VTOBUS FAILED" panics on amd64, amd64/89550

[Scott Long]

Stefan Esser wrote:
> Scott Long schrieb:
> 
>>Jan Mikkelsen wrote:
>>
>>
>>>Hi,
>>>
>>>Doug White wrote:
>>>
>>>
>>>>On Fri, 22 Sep 2006, Jan Mikkelsen wrote:
>>>>
>>>>
>>>>
>>>>>Quick summary:  sym(4) assumes on amd64 that virtual 
>>>>
>>>>addresses provided by
>>>>
>>>>
>>>>>bus_dmamem_alloc() have the same alignment as the physical 
>>>>
>>>>addresses (in
>>>>
>>>>
>>>>>this case, 2*PAGE_SIZE).  They don't, and stuff breaks.  
>>>>
>>>>This patch works
>>>>
>>>>
>>>>>around that.
>>>>
>>>>Why is this? busdma supports alignment constraints; why not just set
>>>>the alignment to what you need it set at? I realize sym has its own
>>>>hand rolled DMA management craziness but alignment is something
>>>>busdma can take care of easily.
>>>
>>>
>>>sym has the alignment requirement on the virtual address because of the
>>>buddy memory allocation algorithm; changing how sym allocates memory
>>>internally would remove the requirement.  The buddy algorithm with 2^13
>>>bytes aligned on a 2^12 byte (but not a 2^13 byte) boundary can
>>>provide two
>>>chunks of 2^12 bytes but nothing greater than 2^12 bytes.
>>>
>>>The VTOBUS failure is caused by the buddy implementation making alignment
>>>assumptions which aren't true, and then getting the virtual addresses
>>>wrong.
>>>
>>>Perhaps I'm just doing something wrong with bus_dma.  I believe I set the
>>>alignment requirements to be 2*PAGE_SIZE, and this is what I see for the
>>>physical address.  However the virtual address seems to only be page
>>>aligned.
>>>
>>>I can't see any mention of virtual address alignment in the bus_dma man
>>>page.  Can it take care of virtual address alignment?  If so, how?
>>> 
>>
>>busdma makes no guarantees on virtual addresses.
>>
>>Sigh, sorry I never got this fixed.  The custom memory allocator made me
>>unhappy, and I never had time to dig into it.  Do real docs on sym exist
>>somewhere?  I'm not against sitting down and re-writing the physical
>>memory handling to both work and conform to the FreeBSD APIs.
> 
> 
> I've been the co-author of the ncr SCSI driver, on which sym is based
> (though not that particular code fragment). Since I know the structure
> and principals of the driver (and since I have and know the docs up to
> the 53c875, possibly also the 53c895), I'd probably be in a position
> to work on this with the least effort to get started. Only problem is
> that I do not have an amd64 system for testing ...
> 
> I changed the private allocator in the sym driver to use contigmalloc,
> some time ago, but now I understand that there are stricter alignment
> requirements. For a start, a work-around could be committed, IMHO (even
> if it is ugly). The better approach is of course an extension of busdma
> to support aligned physical chunks as required by the driver.
> 
> But I could also try to find a clean fix for the affected driver code.
> 
> Is the Symbios SCSI controller still used that much that the effort
> required for a "clean" fix is well spent?
> 
> Regards, STefan

What does the driver need that busdma doesn't provide now?
bus_dmamem_alloc() is just a wrapper around contigmalloc.  All alignment
and exclusion arguments that you can give contigmalloc are honored by
busdma.  I guess that's what frustrated me when I looked at this driver;
I couldn't figure out why all the effort had been made to have a custom
allocator based on contigmalloc (which should be an interface hidden
from most drivers and available only to the MD portion of the kernel,
btw) when busdma works fine for nearly every other driver out there.
I'm not being critical here, it's just that I couldn't figure out if
there was some subtle detail that I was missing.  If there are
facilities that busdma doesn't provide but that you need, I'd be very
happy to work with you on developing them.

Scott