Bumping up a default net.graph.maxdata to avoid "Write failed: Cannot allocate memory"

Mark Martinec Mark.Martinec+freebsd at ijs.si
Thu Jul 24 13:11:40 UTC 2014

Syncing zfs snapshots across the net using 'zfs send' over ssh started
failing one day with ssh reporting "Write failed: Cannot allocate 
after transferring about 15 to 25 GB of data (as it turned out this
snapshot was larger than usual). Neither of the two hosts were
particularly low on memory, the receiving end was running 10.0-STABLE
amd64, network between the two was a gigabit ethernet, interface em.

The problem was repeatable at will. Simplifying the experiment to a:

   ssh <remote-host> zfs send <snapshot> | wc -c

also ended up with the same "Write failed: Cannot allocate memory"
on the receiving side after transferring about 20 GB. Doing some other
experiments ruled out a potential blame from zfs send.

As it turned out (luckily for me, after banging my had over it)
I'm not the only one with this problem - it was reported three years 


And yes, I too had a smallish virtual host running under VirtualBox
on this receiving machine, which was mostly idling. That virtual host
was not involved in any of these experiments.

So the problem is that NetGraph was running out of space for data queue
and net.graph.maxdata needed to be bumped up from a default of 512.
My current setting is net.graph.maxdata=2048 and this suffices for
reliable transfer of huge files (> 60 GB) over ssh. After one such
transfer the  vmstat -z  reports:

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
NetGraph items:          72,   4123,       0,     527,      28,   0,   0
NetGraph data items:     72,   2077,       0,    1023,65650892,   0,   0

while previously the FAIL count was in the hundreds, just as in the
problem report from July 2011. And the issue is not limited to ssh,
others have reported the same over ftp.

Btw, the 'ngctl list' shows:

There are 5 total nodes:
   Name: ngctl78040      Type: socket          ID: 00000010   Num hooks: 
   Name: em0             Type: ether           ID: 00000001   Num hooks: 
   Name: em1             Type: ether           ID: 00000002   Num hooks: 
   Name: vboxnet0        Type: ether           ID: 00000003   Num hooks: 
   Name: vboxnetflt_em0  Type: vboxnetflt      ID: 00000004   Num hooks: 

Unfortunately the 2011 thread remained suspended in the air, with no
action, neither in documentation nor bumping up the default queue limit.

So to save more people from bumping into the same problem, puzzled
over a mystery "Write failed: Cannot allocate memory" failure,
I'd like to suggest bumping up the default value of net.graph.maxdata,
or at least documenting the fact in the handbook.


More information about the freebsd-net mailing list