G5 Bridge-mode MMU

[Peter Grehan]

Hi Nathan,

  Great to see you taking on this work !

> I'm trying to get G5 support going, and have run into an interesting 
> problem while bootstrapping the MMU. Firmware loads the kernel in 
> protected mode, and we have no BAT facility. Thus, the bootstrap 
> allocator fails because low physical memory isn't 1:1 mapped. And we 
> can't add this region to the page table, because the page table is what 
> we are trying to allocate. And we can't even use the MMU's large page 
> facility in bridge mode.

  You really can't use the existing MMU code. The G5 was the driving 
factor to create the pmap indirection.

  My thoughts were to abandon the existing method of trying to keep the 
OFW mappings the same as the kernel mappings, and completely switch 
address spaces when calling back into OFW, rather than just changing 
segment registers. This would require an indirection in the OFW code to 
jumpto to a routine that switches stacks, but code similar to that 
already exists in NetBSD for older G3's where OFW runs in real-mode.

> One solution is to drop back to data real mode while the page table is 
> allocated and set up. This seems to work ok, but requires that the 
> kernel stack is mapped 1:1, which seems to be the case on my iMac G5. 
> I'm not sure how reliable this assumption is, though. It's also 
> irritating because any open firmware calls from real mode (e.g. 
> printf()) make the machine hang up.

  It's a good enough assumption to make in pmap_bootstrap(). Calls to 
OFW usually copy the parameters into BSS, though there are some that don't.

> The other solution is to prepare a temporary  statically-allocated page 
> table with enough mappings to map the kernel, OF, and the real page 
> table, then switch to the dynamically allocated one.

  I think you want to allocate the real page table, put enough mappings 
in there for the kernel text+data+bss (and the frame buffer if that is 
being used for the console) and the page table itself, and modify OFW so 
that there is a big context switch when calling into it. Use the 
existing call into OFW to get it's mappings to make sure those pages are 
excluded from kernel use.

There are some other gotchas that you may know about:

- the OFW calls to determine the amount of memory don't use two cells 
for addresses, required on a G5. That's a simple fix (which you might 
already have :). You might wonder why I did that work for the loader and 
never submitted it for the kernel :) Memory should be restricted to <= 
4G. There's no need for a PAE-style mode to get more memory. Instead, 
that should be the target of a true 64-bit PPC port.

- interrupts switch the CPU to 64-bit mode. You have to immediately drop 
to 32-bit mode in the interrupt handler before any branch is taken. My 
plan was to have a macro in the exception asm that did this, and also 
expanded to 'rfid' for the return. The asm would then be included into 
trap_subr.S and expanded to 32/64-bit, in a similar fashion to how the 
kernel does elf32 and elf64 via macro expansion. Installing the vectors 
would also then become CPU-model specific, but that was inevitable to 
support other CPU models with different vector requirements.

- the G3/4 pmap code relies on 1:1 BAT mapping for page-zeroing. The G5 
code will have to use an i386-style temporary mapping for this.

- another 1:1 relic is the use of UMA_MD_SMALL_ALLOC. I talked with Alan 
Cox a long time back about being able to dynamically determine whether 
this could be used, though that is a fair amount of work. I would like 
to see a single kernel for G3/4/5, but that may be asking too much up 
front, especially given my lack of progress in achieving this goal :)

- the G5 uses two cascaded OpenPIC interrupt controllers. The interrupt 
code needs to be reworked to support this concept.

  .. and probably many more.

later,

Peter.