MTRR failure revisited (nVidia) 8-STABLE/RELEASE

Sun Dec 13 14:05:15 UTC 2009

On Sat, 2009-12-12 at 20:08 -0800, Chris H wrote:
> On Sat, December 12, 2009 6:36 am, Robert Noland wrote:
> > On Sat, 2009-12-12 at 03:47 -0800, Chris H wrote:
> >
> >> Greetings,
> >> I brought this same error to the list back in May 2009.
> >> Under: failed to set mtrr: invalid argument.
> >> Well, I'm back using the same card:
> >> GeForce4 MX 440-SE - VideoRam 65536 - BusID PCI:1:3:0.
> >> The driver is different, I'm using: nvidia-driver-96.43.13 out of ports on a
> >> custom 8-STABLE kernel. Xorg starts up, and produces a desktop. But it's "dog
> >> slow"
> UPDATE:
> Disabling HAL /greatly/ increased performance
> eg; hal_enable="YES" --> hal_enable="NO" in /etc/rc.conf
> More specifically, response times are now closer to what one would anticipate/
> expect now that HAL has been dis-abled in rc.conf.
> >> , and the nvidia driver emits the following error: NVIDIA: failed to set
> >> MTRR 0xf0000000, 0M (write-combining)
> >> several times. I understand John Baldwin provided some "invaluable" help some
> >> time ago:
> >> http://lists.freebsd.org/pipermail/freebsd-hackers/2006-June/016995.html
> >> and I was wondering if anyone has gained any further "insight" with these
> >> cards, and how to better "interface" them in BSD. Last I spoke on the topic, I
> >> was informed that the memory was basically "untouchable" - or perhaps in other
> >> words; can't be manipulated. Has this changed? Surely someone else has had to
> >> deal with this besides me. It seems crazy to spend a "boat load" of $$ on
> >> these high performers, and not be able to use them on a high performing OS -
> >> no? :) Sure, the one I'm working with now is "legacy". But I have 3 near new,
> >> top of their line cards, and thus far it appears that if I ever hope to use
> >> them, I'll be forced to... hack, choke.. spin up a WIN CD. :(
> >>
> >> Thank you for all your time, consideration, and insight.
> >>
> Greetings Robert, and thank you for taking the time to respond.
> >
> > The mtrr issue has to do with the system / bios, not the graphics card.
> > While I've not used the blob driver, the issue in Nouveau and other drm
> > drivers is that on many systems if you run "memcontrol list", you will see a line
> > something like this:
> >
> > 0x0/0x100000000 BIOS write-back set-by-firmware active
> I see the following (condensed for brevity):
> 0x0/0x10000 BIOS write-back fixed-base fixed-length set-by-firmware active
> 0x10000/0x10000-0x70000/0x10000 BIOS write-back fixed-base fixed-length
> set-by-firmware active
> 0x80000/0x4000 BIOS-0x9c000/0x4000 write-back fixed-base fixed-length
> set-by-firmware active
> 0xa0000/0x4000-0xbc000/0x4000 BIOS uncacheable fixed-base fixed-length
> set-by-firmware active
> 0xc0000/0x10000xc7000/0x1000 BIOS write-protect fixed-base fixed-length
> set-by-firmware active
> 0xc8000/0x1000-0xff000/0x1000 BIOS uncacheable fixed-base fixed-length
> set-by-firmware active
> 0x0/0x40000000 BIOS write-back set-by-firmware active

The above entry is the one that causes setting write-combine MTRR to
fail.

> 0xe0000000/0x20000000 BIOS uncacheable set-by-firmware active
> 
> While I could pull the BIOS out of it's socket after POST. I don't suppose
> I could read it's contents to file, and then allow manipulation of the
> regions currently "off limits"?

This is more easily achieved in our MTRR code I expect, certainly than
BIOS hacking all of the effected machines.  Frankly, all of my more
modern machines have this issue.

> >
> > This says that all of memory defaults to write-back.  We aren't allowed
> > to overlap write-combined on top of write-back, so the setting of mtrr fails.
> Isn't it /best/ to choose write-back, so as to mark the memory dirty? I /could/
> choose write-ahead, or write-through.
> > I've looked at ways to try to fix this in the past, but
> > generally found it more practical to use PAT than try to override/fix bios
> > behavior.
> Marius Nünnerich also mentioned this in a response to this thread. Would you be
> willing to share any additional information, based on your experiences using PAT?

PAT and MTRR both allow for the setting of cache attributes for a range
of memory.  For MTRR, there are a certain number of variable range
registers in a given CPU which allow the setting of global cache
attributes for a given memory range.  There are a number of rules about
what memory types may overlap though, so having that entry that covers
all of memory means that setting MTRR to anything other than uncached
will always fail.

PAT is less restrictive in this regard and a table exists in the intel
documentation showing the effective cache method for various
combinations of MTRR/PAT.  The difficulty with PAT is that it is an
attribute of the individual page mapping and so all mappings of the same
physical region of memory need to have the same PAT type set.  jhb@,
alc@ and I (my contribution has mostly consisted of whining and testing)
have recently (over the last year at least) been adding the ability to
handle allowing userspace mappings of memory regions to be mapped with
consistent PAT attributes.  This was also one of the key features that
nvidia wanted before they would deliver an amd64 blob driver.  Much of
the needed infrastructure is in place now, however there is still some
work to be done on bus_dma to make use of the features.  I have some
local hacks to bus_dma that I am using in my trees to allow me to more
easily manipulate the PAT attributes, but they aren't practical to
commit.  We need to revisit this topic and decide what is the correct
way to proceed.

robert.

> > I've been told that linux does apply some BIOS fixups to address this
> > issue, which I might look into again, but I make no promises.
> Is there anything I could do that would help you in this regard?
> There are also a
> > very limited number of variable mtrr registers (7 on most hardware, iirc) for
> > managing caching.
> 
> Thank you again for taking the time to respond.
> 
> --Chris H
> >
> > robert.
> >
> >> --Chris H
> >>
> >>
> >>
> >> _______________________________________________
> >> freebsd-stable at freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> >>
> > --
> > Robert Noland <rnoland at FreeBSD.org>
> > FreeBSD
> >
> >
> > _______________________________________________
> > freebsd-stable at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> >
> >
> 
> 
-- 
Robert Noland <rnoland at FreeBSD.org>
FreeBSD