6.0-RELEASE/AMD64 Ram Capacity?

[Jacques Caron]

Hi,

At 12:06 16/03/2006, Robert Leftwich wrote:
>>Scott Long writes:
>>
>>>The ATA driver is still a bit flakey with more than 4GB of RAM, so 
>>>make sure that you are using either a modern SCSI controller or a 
>>>RAID controller.
>
>Would you mind clarifying if this applies to *only* > 4GB of ram or 
>if it includes exactly 4GB? Also does PAE impact this flakey-ness? 
>Are there any particular chipsets that are flaky/non-flaky? Are all 
>FreeBSD 6.x versions affected?
>
>I guess what it boils down to is that I'm running an asus a8n-sli 
>premium with 4GB ram and 4*160GB SATA2 drives in raid 0+1 (via the 
>onboard nVidia (pseudo) raid) and currently using 6.1beta3 (as 6.0x 
>does not recognise all 4GB of ram) - should I be worried that the 
>drives are going to be flaky?

It will affect any machine with more than 2 (S)ATA drives where some 
of the memory is above the 32-bit limit, since it requires the busdma 
code to allocate bounce buffers under the 32-bit limit, and the 
busdma code has a hard limit on the amount of bounce buffers it 
allocates for a given type in some conditions (in 5.4 at least, 
haven't checked what the status is in 6.0 and later), and the ATA 
driver wants more.

To check if things work, simply do (on a box that you're ready to see panic):
dd if=/dev/adX of=/dev/null bs=512k &
dd if=/dev/adY of=/dev/null bs=512k &
dd if=/dev/adZ of=/dev/null bs=512k &
dd if=/dev/adT of=/dev/null bs=512k &

where X,Y,Z,T are the device numbers of your different drives. If 
your box is still alive after your have entered the 4th line you 
should be safe (on 5.4 you actually won't even get to the 4th line, a 
panic will interfere with your day).

You may also want to check sysctl -a | grep busdma before/while you 
run the test:

hw.busdma.zone1.total_bpages: 320
hw.busdma.zone1.free_bpages: 320
hw.busdma.zone1.reserved_bpages: 0
hw.busdma.zone1.active_bpages: 0
hw.busdma.zone1.total_bounced: 763930088
hw.busdma.zone1.total_deferred: 0
hw.busdma.zone1.lowaddr: 0xffffffff
hw.busdma.zone1.alignment: 2
hw.busdma.zone1.boundary: 65536

On a stock 5.4 total_bpages will be much lower than 320, and 
active_bpages will reach the limit pretty soon (once the 2nd dd is 
running) and kaboom (once your launch the 3rd dd) since the driver 
expects the pages to be there (busdma tells it they are).

I run my 5.4 boxes with the following patch:

diff -c sys/dev/ata/ata-dma.c.orig sys/dev/ata/ata-dma.c
*** sys/dev/ata/ata-dma.c.orig  Fri Sep  2 16:14:19 2005
--- sys/dev/ata/ata-dma.c       Thu Oct 27 12:40:12 2005
***************
*** 111,117 ****
                            BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR,
                            NULL, NULL, 256 * DEV_BSIZE,
                            ATA_DMA_ENTRIES, ch->dma->max_iosize,
!                          BUS_DMA_ALLOCNOW, NULL, NULL, &ch->dma->ddmatag))
         goto error;

       if (bus_dmamem_alloc(ch->dma->cdmatag, (void **)&ch->dma->dmatab, 0,
--- 111,117 ----
                            BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR,
                            NULL, NULL, 256 * DEV_BSIZE,
                            ATA_DMA_ENTRIES, ch->dma->max_iosize,
!                          0, NULL, NULL, &ch->dma->ddmatag))
         goto error;

       if (bus_dmamem_alloc(ch->dma->cdmatag, (void **)&ch->dma->dmatab, 0,


It is not the "right" fix (the problem is in busdma, not ata, or at 
least it used to be in 5.4), but it's a workaround that works pretty 
well: busdma will gladly allocate the bounce buffers and ata will 
happily use them.

Note again that I really haven't checked if things have changed in 
post-5.4 versions, so maybe it works now. I believe Scott made at 
least one change somewhere in there, but I haven't had the 
opportunity to look at it or test it. And obviously I can't tell you 
if the above patch works with anything other than 5.4.

Final note: bounce buffer allocation is *very* slow, so with the 
above patch it takes a few seconds for each ata driver to initialize.

Hope that helps,

Jacques.