E4500 with 24GB RAM

[Doug White]

On Tue, 7 Jun 2005, Hiroki Sato wrote:

> Scott Long <scottl at samsco.org> wrote
>   in <429A7B4D.3080102 at samsco.org>:
>
> sc> I'd guess that you're the first to have access to so much memory
> sc> and a machine to hold it.  The error means that vm_map_find()
> sc> returned KERN_NO_SPACE.  It could be that there is a 64-bit bug
> sc> in the code, or it could be that the page tables to index so much
> sc> memory consume all available space in the kernel map.
>
>  Thanks.  Jake gave me an advice about kern.maxbcache tunable, and
>  I finally make it boot with kern.maxbcache=524288000.
>  However, the hme driver seems to have a problem:
>
>  |panic: iommu_enter: XXX: physical address too large (0x5e3fcc000)

This panic occurs since the IOMMU can only handle physical addresses up to
34 bits (16GB), it appears. From src/sys/sparc64/include/iommureg.h:

 67 #define IOMMU_BITS              34
 68 #define IOMMU_MAXADDR           (1UL << IOMMU_BITS)

Since the panic is in a KASSERT thats why it only pops up under
INVARIANTS.  However the bus_dma tags specify that hme can only handle 32
bit addresses so it should be using a bounce buffer. I'm guessing that
bus_dma needs to be taught about this limitation and use bounce buffers
for the IOMMU on >16GB systems.

Someone should also research this value a bit more ... I wonder if its
from some older system (E450?), and newer machines have larger limits. I
would hope they'd build E4500s with IOMMUs that can address all of
physical memory.

>  |cpuid = 0
>  |KDB: enter: panic
>  |[thread pid 442 tid 100215 ]
>  |Stopped at      kdb_enter+0x3c: ta              %xcc, 1
>  |db> tr
>  |Tracing pid 442 tid 100215 td 0xfffff800af657b80
>  |panic() at panic+0x16c
>  |iommu_enter() at iommu_enter+0x3c
>  |iommu_dvmamap_load_buffer() at iommu_dvmamap_load_buffer+0x118
>  |iommu_dvmamap_load_mbuf() at iommu_dvmamap_load_mbuf+0x170
>  |hme_load_txmbuf() at hme_load_txmbuf+0x68
>  |hme_start_locked() at hme_start_locked+0x1bc
>  |hme_start() at hme_start+0x2c
>  |if_start() at if_start+0xa0
>  |ether_output_frame() at ether_output_frame+0x244
>  |ether_output() at ether_output+0x474
>  |ip_output() at ip_output+0xa98
>  |udp_output() at udp_output+0x5b4
>  |udp_send() at udp_send+0x14
>  |sosend() at sosend+0x654
>
>  When INVARIANTS enabled this panic occurs instantly.
>
>

-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite at gumbysoft.com          |  www.FreeBSD.org