8.2 Problem with creating Snapshots on large file systems? Cannot backup anymore.

[Karl M. Joch]

Hi,

since a few days i have trubles on one server when running the daily amanda backup. This was running fine since over 6 Month and now dump, called by amanda, always returns:

FAIL dumper localhost.sbg.kmjeuro.com mirror/CTSs1d 20111219230324 0 [dump (16603) /sbin/dump returned 1]
  sendbackup: start [localhost.sbg.kmjeuro.com:mirror/CTSs1d level 0]
  sendbackup: info BACKUP=/sbin/dump
  sendbackup: info RECOVER_CMD=/usr/bin/gzip -dc |/sbin/restore -xpGf - ...
  sendbackup: info COMPRESS_SUFFIX=.gz
  sendbackup: info end
  ? mksnap_ffs: Cannot create snapshot /usr/.snap/dump_snapshot: Resource temporarily unavailable
  | dump: Cannot create /usr/.snap/dump_snapshot: No such file or directory
  |
  ? dump (16603) /sbin/dump returned 1
  sendbackup: error [dump (16603) /sbin/dump returned 1]

and Amanda fails. 

System is:

kern.ostype: FreeBSD
kern.osrelease: 8.2-STABLE
kern.osrevision: 199506
kern.version: FreeBSD 8.2-STABLE #1: Sun Mar 27 20:08:58 CEST 2011

FS:

Filesystem            Size    Used   Avail Capacity  Mounted on
/dev/mirror/CTSs1a    1.9G    362M    1.4G    20%    /
devfs                 1.0K    1.0K      0B   100%    /dev
/dev/mirror/CTSs1f    4.8G    2.0M    4.5G     0%    /tmp
/dev/mirror/CTSs1d    969G    301G    590G    34%    /usr
/dev/mirror/CTSs1e     48G    2.3G     42G     5%    /var
/dev/mirror/CTSs1g    779G    4.5G    712G     1%    /usr/l1
devfs                 1.0K    1.0K      0B   100%    /var/named/dev

I tried alot of things meanwhile:

- Powered of, running fsck -y in Single User Mode

- Replaced one Harddisk which had some (recoverable) read errors and resynced gmirror

- Googled alot and found some other posts with the same problem, but no real answer.

I want to stay with ufs2 and Amanda and dont want to switch to ZFS at the moment. I have some other Servers which does the same thing without any problem, as this one has done it for month too. I prepared an rsync backup to one of our NAS to have a temp backup but i need to have tapes soon again. Any help or idea would be very appreciated.

Thank you,

Karl