svn commit: r205221 - head/sys/dev/bge

[Andrew Gallatin]

Bruce Evans wrote:
 > On Wed, 17 Mar 2010, Andrew Gallatin wrote:
 >
 >> Pyun YongHyeon wrote:
 >>
 >>>   Revert r205090.
 >>>   It's hard to know when the mail box register write will get 
flushed to
 >>>   the hardware and it may take longer.
 >>>     Pointed out by:    scottl
 >>
 >> I may be mis-reading the code, but it looks like the mailbox
 >> register is in memory space, which should be flushed immediately
 >> unless write-combining is enabled on the region.  The bge
 >> driver does not seem to be setting up write combining.
 >> Is the concern that something may enable write combining
 >> behind your back?  In that case, a wmb() could act as a
 >> serializing instruction and flush the WC buffers.
 >
 > We want writes to the PCI bus to be efficient.  Normally (?) writes
 > to bge registers appear to be several times faster than reads.  I don't
 > know if this depends on write combining but think it depends on write
 > buffering which can delay the write to the hardware by about the
 > difference between the read time and the time to write to the bufer.
 > Any forcing of serialization or timing would presumably lose the
 > benefits of the buffer.

What buffer? On the host side of the PCI{e} bus or the NIC side??

For i386/amd64, only write-combining (MTRR or PAT) pio mappings are
buffered on the host side.  Every other PIO behaves as if implicitly
surrounded by a serializing instruction before and after for legacy
reasons.  On non i386/amd64, there may be buffering; it depends on the
platform.  Or are you talking about buffering on the bge side of the
PCI{e} bus?

The point I was trying to make is that PIO reads are very
(hopelessly?) slow because the CPU stalls while waiting for the device
to return some data, and you're at the mercy of the device.  In
general, when you're given a choice as to what to use to force a write
across the PCI{e} bus (in the case of WC being enabled, or working on
a non-x86/amd64 arch), the best choice by far is a serializing
instruction (wmb()) and not a PIO read.  Unfortunately, it looks like
a PIO read is intimately entwined with correct operation of this
device, so doing it the old way is probably safest/best.

Sorry for the noise..

Drew