Problem with amanda backup (cannot allocate memory) on FreeBSD-9.1-RELEASE amd64

[Белогуров Святослав]

Hello everyone. I've posted this problem on forums.zmanda.com, but 
without any luck. I think, maybe it's something FreeBSD-specific, or 
specific to it's version. So, here it is, any help will be greatly 
appreciated.


I have a backup server with amanda-2.4.4p3, which is backup'ing numerous 
servers with different FreeBSD versions (from 6.x to 8.x, i386, amd64), 
some Linux machines. Everything works fine.

Recently I've upgraded some servers to FreeBSD 9.1-amd64. I've installed 
amanda-2.4.4p3 on these servers, but when I tried to do backups, they 
complete with errors, on disk volumes that are larger than 500-1000mb 
approximately. The error message on client machines was: sendbackup: 
critical (fatal): index tee cannot write [Broken pipe]

I decided, that I'm using too old version of amanda, and it's time to 
upgrade. I've installed amanda 3.3.4 to our backup server (as an another 
amanda version, alternative to working one), and to FreeBSD 9.1 clients. 
I've also installed GNU tar 1.27 on my clients.


But, this does not help. I still receive errors on volumes that are 
larger than 500-1000mb, and backup does not complete.

I only get error on /usr volume on problem host, / and /var are 
backuping normally. I suppose that's because of their size (/ - 740M 
used, /var - 42M used, /usr - 4.7G used). When I use disklist to exclude 
some directories on /usr volume (that are using most of it's space) - 
backup also completes normally.

In amgtar.*.debug on client:

Sat Dec 7 14:39:28 2013: thd-0x802c170c0: amgtar: Spawning 
"/usr/local/gnutar-1.27/bin/tar /usr/local/gnutar-1.27/bin/tar --create 
--verbose --file - --directory /usr --one-file-system 
--listed-incremental 
/usr/local/amanda-3.3.4/var/amanda/gnutar-lists/problem-host_usr_0.new 
--sparse --ignore-failed-read --totals ." in pipeline
Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: 84: size(|): Total 
bytes written: 60026880 (58MiB, ?/s)
Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: 0: strange(?): 
/usr/local/gnutar-1.27/bin/tar: -: Cannot write: Broken pipe
Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: 0: strange(?): 
/usr/local/gnutar-1.27/bin/tar: Error is not recoverable: exiting now
Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: after 
/usr/local/gnutar-1.27/bin/tar /usr wait
Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: amgtar: 
/usr/local/gnutar-1.27/bin/tar: pid 19987
Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: 
/usr/local/gnutar-1.27/bin/tar exited with status 2: see 
/tmp/amanda/client/org/amgtar.20131207143928000.debugSat Dec 7 14:39:33 
2013: thd-0x802c170c0: amgtar: sendbackup: size 58620
Sat Dec 7 14:39:33 2013: thd-0x802c170c0: amgtar: sendbackup: end


In amandad.*.debug on clients I get strange error "Cannot allocate memory":

Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: opening security 
stream for fd 50
Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: opening security 
stream for fd 51
Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: opening security 
stream for fd 52
Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: 
security_close(handle=0x802ca3280, driver=0x800c83680 (BSDTCP))
Sat Dec 7 14:39:33 2013: thd-0x802c17100: amandad: 
security_stream_close(0x802d23000)
Sat Dec 7 14:39:44 2013: thd-0x802c17100: amandad: tcpm_send_token: data 
is still flowing
Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: 
security_stream_seterr(0x802d35000, write error to: Cannot allocate memory)
Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: sending NAK pkt:
<<<<<
ERROR write error on stream 499999: write error to: Cannot allocate memory
 >>>>>
Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: 
security_stream_close(0x802d35000)
Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: 
security_stream_close(0x802d3e000)
Sat Dec 7 14:39:52 2013: thd-0x802c17100: amandad: 
security_stream_close(0x802d47000)
Sat Dec 7 14:39:53 2013: thd-0x802c17100: amandad: pid 19988 finish time 
Sat Dec 7 14:39:53 2013


Strange, but that's not explicitly amanda problem. When I use SSH to 
login to problem machine's console, everything works OK. But when I 
start to compile some program package from source, or rebuild kernel 
(when there is much output from "make" to the console), sometimes SSH 
connection breaks, and when I connect again, I can see message from sshd 
in logs: "cannot allocate memory"


So, maybe it's some bug with memory allocation on my FreeBSD box? How 
can I check this, or, maybe, someone had already encountered anything 
similar?

-- 
С уважением,
Белогуров Святослав
http://relant.ru