nfs-server silent data corruption

[Erik Trulsson]

On Mon, Apr 21, 2008 at 08:43:33AM -0700, Jeremy Chadwick wrote:
> On Mon, Apr 21, 2008 at 04:52:55PM +0200, Arno J. Klaassen wrote:
> > Kris Kennaway <kris at FreeBSD.ORG> writes:
> > > Uh, you're getting server-side data corruption, it could definitely be
> > > because of the memory you added.
> > 
> > yop, though I'm still not convinced the memory is bad (the very same
> > Kingston ECC as the 2*1G in use for about half a year already) :
> 
> Can you download and run memtest86 on this system, with the added 2G ECC
> insalled?  memtest86 doesn't guarantee showing signs of memory problems,
> but in most cases it'll start spewing errors almost immediately.
> 
> One thing I did notice in the motherboard manual below is something
> called "Hammer Configuration".  It appears to default to 800MHz, but
> there's an "Auto" choice.  Does using Auto fix anything?
> 
> > I added it directly to the 2nd CPU (diagram on page 9 of
> >  http://www.tyan.com/manuals/m_s2895_101.pdf) and the problem
> > seems to be the interaction between nfe0 and powerd .... :
> 
> That board is the weirdest thing I've seen in years.
> 
> Two separate CPUs using a single (shared) memory controller,

No. Each CPU contains its own memory controller (just like all AMD's
Opteron/Athlon64 CPUs does.)


> two
> separate (and different!) nVidia chipsets,

More like one chipset consisting of several physical chips. (Which is
actually quite common. The most common division is a
"nortbridge/southbridge" division, but other ways are possible too.)

The only unusual thing is that there are several chips connected directly
to the CPUs, instead of having the CPUs talk to a single chip which in
turn talks to another chip which can easily create bottlenecks.  


> a SMSC I/O controller
> probably used for serial and parallel I/O

Just like almost all other motherboards.

>, two separate nVidia NICs with
> Marvell PHYs (yet somehow you can bridge the two NICs and PHYs?)

What is so wierd about that?  If you want to have more than one ethernet
connection, then you normally have more than one NIC.
Bridging can easily (and commonly) be done over separate NICs.


>, two
> separate PCI-e busses (each associated with a separate nVidia chipset),

Since it is always the case that each PCI-E slot or PCI-E device sits
on its own bus I fail to see anything strange about that.
(And it is actually very common to have the PCI-E slots on motherboards
be connected to different chips.)

> two separate PCI-X busses... the list continues.

Having more than one PCI-X bus used to be fairly common on server boards for
performance reasons.  Nowadays PCI-X is slowly being replaced with PCI-E
so on the latest generation of serverboards there are usually no more than
one PCI-X bus.


> 
> I know you don't need opinions at this point, but what a behemoth.  I
> can't imagine that thing running reliably.

I would rather say it is a quite elegant design for a high-end motherboard
intended for server/workstation installations.

It is a dual-socket Opteron board.  Each Opteron has its own
memory-controller and uses HyperTransport to connect to other components.
Each dual-socket Opteron has three HyperTransport links available.
One from each CPU will be needed to the other CPU, leaving two links from
each CPU available to connect to other chips.  From that starting point it
is a fairly obvious design.
To maximise the available bandwidth one would want to spread out the chips
over these links, which this motherboard does fairly well, using three
of the four available links.
(And hanging the most important things from CPU0, so you can actually use
the board even if you have only one CPU installed.)

As for reliability I see no particular reason for that board to be less
reliable than any other multi-CPU board.



-- 
<Insert your favourite quote here.>
Erik Trulsson
ertr1013 at student.uu.se