Can't read a full block, only got 8193 bytes.

[Ask Bjørn Hansen]

Hi everyone,

We're recycling an old database server with room for 16 disks as a backup server (our old database servers had 12-20 15k disks; the new ones one or two SSDs and they're faster).

We have a box running FreeBSD 8.2 with 7 disks in a ZFS raidz2 (and a spare).  It's using an older 3ware card with all the disks (2TB WD green "ears" ones) setup as a "single" unit on the 3ware controller and though slow is basically working great.  We have a small program to smartly purge old snapshots that I wrote after a year and tens of thousands of snapshots: https://github.com/abh/zfs-snapshot-cleaner

The new box is running 9.0 with a 3ware 9690SA-4I4E card with the latest firmware (4.10.00.024).  We're using Seagate 3TB barracuda disks (big and cheap; good for backups).

Now for the problem: When running bonnie++ we get a few ZFS checksum errors and (weirder) we get this error from bonnie:

"Can't read a full block, only got 8193 bytes."

This seems to only be when testing a single ZFS disk or a UFS partition.  Testing a raidz1 we just get checksum errors noted in zpool status, but no errors reading (though read speeds are ~10MB/second across four disks -- writing sequentially was ~230MB/second).

Any ideas where to start look?

Our best guess is that the 3ware controller can't play nicely with the disks; we're planning to try some older/smaller disks on Monday and then trying the same system and disks with Linux to see if the 3ware driver there works differently.



Ask