Bumping up a default net.graph.maxdata to avoid "Write failed: Cannot allocate memory"
Mark.Martinec+freebsd at ijs.si
Thu Jul 24 13:11:40 UTC 2014
Syncing zfs snapshots across the net using 'zfs send' over ssh started
failing one day with ssh reporting "Write failed: Cannot allocate
after transferring about 15 to 25 GB of data (as it turned out this
snapshot was larger than usual). Neither of the two hosts were
particularly low on memory, the receiving end was running 10.0-STABLE
amd64, network between the two was a gigabit ethernet, interface em.
The problem was repeatable at will. Simplifying the experiment to a:
ssh <remote-host> zfs send <snapshot> | wc -c
also ended up with the same "Write failed: Cannot allocate memory"
on the receiving side after transferring about 20 GB. Doing some other
experiments ruled out a potential blame from zfs send.
As it turned out (luckily for me, after banging my had over it)
I'm not the only one with this problem - it was reported three years
And yes, I too had a smallish virtual host running under VirtualBox
on this receiving machine, which was mostly idling. That virtual host
was not involved in any of these experiments.
So the problem is that NetGraph was running out of space for data queue
and net.graph.maxdata needed to be bumped up from a default of 512.
My current setting is net.graph.maxdata=2048 and this suffices for
reliable transfer of huge files (> 60 GB) over ssh. After one such
transfer the vmstat -z reports:
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
NetGraph items: 72, 4123, 0, 527, 28, 0, 0
NetGraph data items: 72, 2077, 0, 1023,65650892, 0, 0
while previously the FAIL count was in the hundreds, just as in the
problem report from July 2011. And the issue is not limited to ssh,
others have reported the same over ftp.
Btw, the 'ngctl list' shows:
There are 5 total nodes:
Name: ngctl78040 Type: socket ID: 00000010 Num hooks:
Name: em0 Type: ether ID: 00000001 Num hooks:
Name: em1 Type: ether ID: 00000002 Num hooks:
Name: vboxnet0 Type: ether ID: 00000003 Num hooks:
Name: vboxnetflt_em0 Type: vboxnetflt ID: 00000004 Num hooks:
Unfortunately the 2011 thread remained suspended in the air, with no
action, neither in documentation nor bumping up the default queue limit.
So to save more people from bumping into the same problem, puzzled
over a mystery "Write failed: Cannot allocate memory" failure,
I'd like to suggest bumping up the default value of net.graph.maxdata,
or at least documenting the fact in the handbook.
More information about the freebsd-net