rare, random issue with read(), mmap() failing to read entire file

[Adam Vande More]

On Fri, Nov 15, 2013 at 8:56 PM, John Refling <netbsdrat at gmail.com> wrote:

>
>
> I'm having some very insidious issues with copying and verifying
> (identical)
> data from several hard disks.  This might be a hardware issue or something
> very deep in the disk / filesystem code.  I have verified this with several
> disks and motherboards.  It corrupts 0.0096% of my files, different files
> each time!
>
>
>
> Background:
>
>
>
> 1.  I have a 500 GB USB hard disk (the new 4,096 [4k] sector size) which I
> have been using to store a master archive of over 70,000 files.
>
>
>
> 2.  To make a backup of the USB disk, I copied everything over to a 500 GB
> SATA hard disk.  [Various combinations of `cp -r', `scp -r', `tar -cf - . |
> rsh ... tar -xf -', etc.]
>
>
>
> 3.  To verify that the copy was correct, I did sha256 sums of all files on
> both disks.
>
>
>
> 4.  When comparing the sha256 sums on both drives, I discovered that 6 or
> so
> files did not compare OK from one drive to the other.
>
>
>
> 5.  When I checked the files individually, the files compared OK, and even
> when I recomputed their individual sha256 sums, I got DIFFERENT sha256 sums
> which were correct this time!
>
>
>
> The above lead me to investigate further, and using ONLY the USB disk, I
> recomputed the sha256 sums for all files ON THAT DISK.  A small number
> (6-12) of files ON THE SAME DISK had different sha256 sums than previously
> computed!  The disk is read-only so nothing could have changed.
>
>
>
> To try to get to the bottom of this, I took the sha256 code and put it in
> my
> own file reading routine, which reads-in data from the file using read().
> On summing up the total bytes read in the read() loop, I discovered that on
> the files that failed to compare, the read() returned EOF before the actual
> EOF. According to the manual page this is impossible.  I compared the total
> number of bytes read by the read() loop to the stat() file length value,
> and
> they were different!  Obviously, the sha256 sum will be different since not
> all the file is read.
>
>
>
> This happens consistently on 6 to 12 files out of 70,000+ *every* time, and
> on DIFFERENT files *every* time.  So things work 99.9904% of the time.
>
>
>
> But something fails 0.0096% (one hundredth of one percent) of the time,
> which with a large number of files is significant!
>
>
>
> Instead of read(), I tried mmap()ing chunks of the file.  Using mmap() to
> access the data in the file instead of read() resulted in a (different)
> sha256 sum than the read() version!  The mmap() version was correct, except
> in ONE case where BOTH versions were WRONG, when compared to a 3rd and 4th
> run!
>
>
>
> Using `diff -rq disk1 disk2` resulted in similar issues.  There were always
> a few files that failed to compare.  Doing another `diff -rq disk1 disk2`
> resulted in a few *other* files that failed to compare, while the ones that
> didn't compare OK the first time, DID compare OK the second time.  This
> happened to 6-12 files out of 70,000+.
>
>
>
> Whatever is affecting my use of read() in my sha256 routine seems to also
> affect system utilities such as diff!
>
>
>
> This gets really insidious because I don't know if the original `cp -r
> disk1
> disk2` did these short reads on a few files while copying the files, thus
> corrupting my archive backup (on 6-12 files)!
>
>
>
> Some of the files that fail are small (10KB) and some are huge (8GB).
>
>
>
> HELP!
>
>
>
> It takes 7 hours to recompute the sha256 sums of the files on the disk so
> random experiments are time consuming, but I'm willing to try things that
> are suggested.
>
>
>
> System details:
>
>
>
> This is observed with the following disks:
>
>
>
> Western Digital 500GB SATA 512 byte sectors
>
> Hitachi 500GB SATA 512 byte sectors
>
> Iomega RPHD-UG3 500GB USB 4096 byte sectors
>
>
>
> in combination with these motherboards:
>
>
>
> P4M800Pro-M V2.0: Pentium D 2.66 GHz, 2GB memory
>
> HP/Compaq Evo: Pentium 4, 2.8 GHz, 2GB memory
>
>
>
> OP System version:
>
> Freebsd: 9.1 RELEASE #0
>
>
>
> no hardware errors noted in /var/log/messages during the file reading
>
>
>
> did Spinrite on disks to freshen (re-read/write) all sectors, with no
> errors.
>
>
>
> The file systems were built using:
>
>
>
> dd if=/dev/zero of=/dev/xxx bs=2m
>
> newfs -m0 /dev/xxx
>
>
>
> Looked through the mailing lists and bug reports but can't see anything
> similar.
>
>
>
> Thanks for your help,
>
>
>
> John Refling
>

Try recoverdisk(1)



-- 
Adam