scp: Write Failed: Cannot allocate memory

Tue Jul 5 07:00:26 UTC 2011

Hi all,

just as an addition: an upgrade to last Friday's FreeBSD-Stable and to  
VirtualBox 4.0.8 does not fix the problem.

I will experiment a bit more tomorrow after hours and grab some statistics.

Regards
Peter

Quoting "Peter Ross" <Peter.Ross at bogen.in-berlin.de>:

> Hi all,
>
> I noticed a similar problem last week. It is also very similar to  
> one reported last year:
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058708.html
>
> My server is a Dell T410 server with the same bge card (the same  
> pciconf -lvc output as described by Mahlon:
>
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058711.html
>
> Yours, Scott, is a em(4)..
>
> Another similarity: In all cases we are using VirtualBox. I just  
> want to mention it, in case it matters. I am still running  
> VirtualBox 3.2.
>
> Most of the time kstat.zfs.misc.arcstats.size was reaching  
> vfs.zfs.arc_max then, but I could catch one or two cases then the  
> value was still below.
>
> I added vfs.zfs.prefetch_disable=1 to sysctl.conf but it does not help.
>
> BTW: It looks as ARC only gives back the memory when I destroy the  
> ZFS (a cloned snapshot containing virtual machines). Even if nothing  
> happens for hours the buffer isn't released..
>
> My machine was still running 8.2-PRERELEASE so I am upgrading.
>
> I am happy to give information gathered on old/new kernel if it helps.
>
> Regards
> Peter
>
> Quoting "Scott Sipe" <cscotts at gmail.com>:
>
>>
>> On Jul 2, 2011, at 12:54 AM, jhell wrote:
>>
>>> On Fri, Jul 01, 2011 at 03:22:32PM -0700, Jeremy Chadwick wrote:
>>>> On Fri, Jul 01, 2011 at 03:13:17PM -0400, Scott Sipe wrote:
>>>>> I'm running 8.2-RELEASE and am having new problems with scp. When scping
>>>>> files to a ZFS directory on the FreeBSD server -- most notably  
>>>>> large files
>>>>> -- the transfer frequently dies after just a few seconds. In my  
>>>>> last test, I
>>>>> tried to scp an 800mb file to the FreeBSD system and the  
>>>>> transfer died after
>>>>> 200mb. It completely copied the next 4 times I tried, and then  
>>>>> died again on
>>>>> the next attempt.
>>>>>
>>>>> On the client side:
>>>>>
>>>>> "Connection to home closed by remote host.
>>>>> lost connection"
>>>>>
>>>>> In /var/log/auth.log:
>>>>>
>>>>> Jul  1 14:54:42 freebsd sshd[18955]: fatal: Write failed: Cannot allocate
>>>>> memory
>>>>>
>>>>> I've never seen this before and have used scp before to transfer  
>>>>> large files
>>>>> without problems. This computer has been used in production for  
>>>>> months and
>>>>> has a current uptime of 36 days. I have not been able to notice  
>>>>> any problems
>>>>> copying files to the server via samba or netatalk, or any problems in
>>>>> apache.
>>>>>
>>>>> Uname:
>>>>>
>>>>> FreeBSD xeon 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Sat Feb 19 01:02:54 EST
>>>>> 2011     root at xeon:/usr/obj/usr/src/sys/GENERIC  amd64
>>>>>
>>>>> I've attached my dmesg and output of vmstat -z.
>>>>>
>>>>> I have not restarted the sshd daemon or rebooted the computer.
>>>>>
>>>>> Am glad to provide any other information or test anything else.
>>>>>
>>>>> {snip vmstat -z and dmesg}
>>>>
>>>> You didn't provide details about your networking setup (rc.conf,
>>>> ifconfig -a, etc.).  netstat -m would be useful too.
>>>>
>>>> Next, please see this thread circa September 2010, titled "Network
>>>> memory allocation failures":
>>>>
>>>> http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/thread.html#58708
>>>>
>>>> The user in that thread is using rsync, which relies on scp by default.
>>>> I believe this problem is similar, if not identical, to yours.
>>>>
>>>
>>> Please also provide your output of ( /usr/bin/limits -a ) for the server
>>> end and the client.
>>>
>>> I am not quite sure I agree with the need for ifconfig -a but some
>>> information about the networking driver your using for the interface
>>> would be helpful, uptime of the boxes. And configuration of the pool.
>>> e.g. ( zpool status -a ;zfs get all <poolname> ) You should probably
>>> prop this information up somewhere so you can reference by URL whenever
>>> needed.
>>>
>>> rsync(1) does not rely on scp(1) whatsoever but rsync(1) can be made to
>>> use ssh(1) instead of rsh(1) and I believe that is what Jeremy is
>>> stating here but correct me if I am wrong. It does use ssh(1) by
>>> default.
>>>
>>> Its a possiblity as well that if using tmpfs(5) or mdmfs(8) for /tmp
>>> type filesystems that rsync(1) may be just filling up your temp ram area
>>> and causing the connection abort which would be expected. ( df -h ) would
>>> help here.
>>
>> Hello,
>>
>> I'm not using tmpfs/mdmfs at all. The clients yesterday were 3  
>> different OSX computers (over gigabit). The FreeBSD server has 12gb  
>> of ram and no bce adapter. For what it's worth, the server is  
>> backed up remotely every night with rsync (remote FreeBSD uses  
>> rsync to pull) to an offsite (slow cable connection) FreeBSD  
>> computer, and I have not seen any errors in the nightly rsync.
>>
>> Sorry for the omission of networking info, here's the output of the  
>> requested commands and some that popped up in the other thread:
>>
>> http://www.cap-press.com/misc/
>>
>> In rc.conf:  ifconfig_em1="inet 10.1.1.1 netmask 255.255.0.0"
>>
>> Scott
>>
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>>
>
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
>