Update on data corruption with Tyan/3Ware

[Erik Trulsson]

On Tue, Sep 25, 2007 at 07:09:16AM -0700, Ted Mittelstaedt wrote:
> 
> 
> > -----Original Message-----
> > From: Erik Trulsson [mailto:ertr1013 at student.uu.se]
> > Sent: Tuesday, September 25, 2007 5:27 AM
> > To: Ted Mittelstaedt
> > Cc: Bart Silverstrim; Chris Boyd; freebsd-questions at freebsd.org
> > Subject: Re: Update on data corruption with Tyan/3Ware
> > 
>  
> > > Bart and Chris,
> > > 
> > >   The problem might be that the 3ware driver uses a 32 bit "int" to
> > > represent
> > > a file size.
> > 
> > The 3ware driver handles blocks on the device. It knows nothing
> > about files or their sizes.  That is the job of the filesystem 
> > which resides
> > at a higher level in the OS.
> > 
> > It is true that the 3ware twe(4) driver uses 32-bit ints to represent the
> > *disk* size in number of disk-blocks.  With the standard 512-byte 
> > block size
> > this gives a maximum disk size of (512*4G==) 2TB, which is a known
> > limitation of the twe(4) driver as well as many other disk controllers.
> > Anyways the twa(4) driver is supposed to handle volumes larger than 2TB.
> > 
> > (twe(4) handles 3ware 7000 and 8000-series controllers, while the twa(4)
> > driver is for the 9000-series.)
> > 
> 
> Your right, I missed that.
> 
> > >  In FreeBSD, stat() ftruncate() lseek() and
> > > friends which are based on "strut stat" had this limitation 
> > under FreeBSD
> > > 4.xx.
> > > 
> > >   Note line# 821 of twe_freebsd.c the driver:
> > > 
> > >     sc->twed_disk->d_maxsize = (TWE_MAX_SGL_LENGTH - 1) * PAGE_SIZE;
> > >     sc->twed_disk->d_sectorsize = TWE_BLOCK_SIZE;
> > >     sc->twed_disk->d_mediasize = TWE_BLOCK_SIZE *
> > > (off_t)sc->twed_drive->td_size;
> > >     sc->twed_disk->d_fwsectors = sc->twed_drive->td_sectors;
> > > 
> > > that off_t also appears elsewhere.
> > 
> > Since off_t is a 64-bit type, and since none of that code fragment has
> > anything to do with *file* sizes, I am not sure what your point is.
> >
> 
> This isn't true, it is OS dependent.  On FreeBSD 64 bit it may be
> a long long but it isn't on all UNIXes.  (particularly old ones)
> That is why GNU/Linux has (or had) off64_t

Well, yes, but since the code fragment above comes from a file with
FreeBSD-specific code it really does not matter in this case how other
systems define off_t.

> 
> It's my understanding the twa/twe drivers have a binary "blob" that they
> loads. 

No, they don't.  What you might be thinking of is that you used to be able
to include a firmware image for the card in the driver.  That firmware is
code which is executed *on* the controller card and is normally included
on the card.  Including a firmware image in the driver is no longer
supported since there nowadays is support for updating the firmware via
a userland program.

> You don't know what architecture this blob was compiled under.

And if they are even half-way competent at setting up a cross-compile
environment it does not matter what system they use to compile the code.

> I merely pointed out use of this code fragment to illustrate a
> point, not to state that this was a bug in the driver to indicate
> that it may be possible the blob as a limitation of some kind.
> 
> Since Bart Silverstrim says he is storing >4GB files without
> trouble under Linux, I would ASSUME the limitation ISN'T in
> the 3ware hardware.
> 
> Apparently you missed the post from Chris that states:
> 
> "...FreeBSD 6.2 AMD64 with all patches as of 9-21-2007..."
> 
> "...We've narrowed the problem down to files that are > 4GB.  Anytime we
> have a file that's > 4GB, we get inconsistent checksums, can't
> uncompress it, etc.  Files < 4GB are fine..."

I missed none of that.  I just note that the 3ware driver and card knows
nothing about files.  It has no way of knowing whether the blocks it is
reading and writing belongs to one large file or several small files. 
Therefore if there are problems only with *files* larger than 4GB it seems
unlikely that the problem is with the card or its driver.

> 
> So as I already stated the VERY FIRST RESPONSE that Chris needs to
> go to 3ware and ask them what is going on.  Unless your going to continue
> to say that FreeBSD64 has a 4GB filesize limitation?

No, I know very well that FreeBSD does not have any 4GB filesize limitation.
It can have bugs in the filesystem or virtual memory system though.
The userland programs reading and writing the file might also have bugs
for that matter.


The first things I would check in such a situation is if the same problem
happens with some other disk controller in the same system.
I would also check the RAM carefully with Memtest86 or similar. (Bad RAM
can cause all kinds of very strange behaviour.)



-- 
<Insert your favourite quote here.>
Erik Trulsson
ertr1013 at student.uu.se