FreeBSD 10.2-RELEASE #0 r286666: Panic and crash

[Shawn Bakhtiar]

Hi all!

http://pastebin.com/niXrjF0D

Please refer to full output from crash above.

This morning our IMAP server decided to go belly up. I could not remote in, and the machine would not respond to any pings.

Checking the physical console I had the following worrisome messages on screen:

• g_vfs_done():da1p1[READ(offset=7265561772032, length=32768)]error = 5
• g_vfs_done():da1p1[WRITE(offset=7267957735424, length=131072)]error = 16
• /mnt/USBBD: got error 16 while accessing filesystem
• panic: softdep_deallocate_dependencies: unrecovered I/O error
• cpuid = 5

/mnt/USBDB is a MyBook USB 8TB drive that we use for daily backups of the IMAP data using rsync. Everything so far has worked without issue.

I also noticed a bunch of:

• fstat: can't read file 2 at 0x4000000001fffff
• fstat: can't read file 4 at 0x780000ffff
• fstat: can't read file 5 at 0x600000000
• fstat: can't read file 1 at 0x200007fffffffff
• fstat: can't read file 2 at 0x4000000001fffff
• fstat: can't read file 4 at 0x780000ffff
• fstat: can't read file 5 at 0x600000000


but I have no idea what these are from.

df -h output:
/dev/da0p2    1.8T    226G    1.5T    13%    /
devfs         1.0K    1.0K      0B   100%    /dev
/dev/da1p1    7.0T    251G    6.2T     4%    /mnt/USBBD


da0p2 is a RAID level 5 on an HP Smart Array

Here is the output of dmsg after reboot:
http://pastebin.com/rHVjgZ82

Obviously both the RAID and USB drive did not walk away from the crash cleaning. Should I be running a fsck at this point on both from single user mode to verify and clean up. My concern is the:
WARNING: /: mount pending error: blocks 0 files 26
when mounting /dev/da0p2

For some reason I was under the impression that fsck was run automatically on reboot.

Any help in this matter would be greatly appreciated. I'm a little concerned that a backup strategy that has worked for us for many MANY years would so easily throw the OS into panic. If an I/O error occurred on the USB Drive I would frankly think it should just back out, without panic. Or am I missing something?

Any recommendations / insights would be most welcome.
Shawn