Network memory allocation failures

[Mahlon E. Smith]

On Tue, Sep 07, 2010, Jeremy Chadwick wrote:
> 
> I figured there might memory exhaustion of sorts, possibly in the bce(4)
> driver itself, that could cause the OP's problem.  bce(4) might not be
> the problem at all.  But the OP's issue seems to only occur when
> transmitting data, not receiving:
> 
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058708.html

More information:

Looks like 100M wasn't enough of a test burst to tickle the problem in
my original message... 10G is, though.  It's definitely happening in
both directions.

Upgraded to -STABLE on one of the two machines last night, running
GENERIC.

FreeBSD obb 8.1-STABLE FreeBSD 8.1-STABLE #0: Tue Sep  7 19:48:55 PDT 2010     root at obb:/usr/obj/usr/src/sys/GENERIC  amd64


Outgoing:

    obb# scp testfile root at holp:/usr/local/tmp/
    testfile                            8%  856MB  37.6MB/s   04:09 ETA
    Write failed: Cannot allocate memory
    lost connection
    obb# scp testfile root at holp:/usr/local/tmp/
    testfile                            0%   72MB  34.3MB/s   04:56 ETA
    Write failed: Cannot allocate memory
    lost connection

Incoming:

    obb# scp root at holp:/usr/local/tmp/testfile .
    testfile                            6%  670MB  31.9MB/s   04:59 ETA
    Write failed: Cannot allocate memory
    lost connection
    obb# scp root at holp:/usr/local/tmp/testfile .
    testfile                            1%  118MB  39.3MB/s   04:17 ETA
    Write failed: Cannot allocate memory
    lost connection
    obb# scp root at holp:/usr/local/tmp/testfile .
    testfile                           15% 1613MB  29.0MB/s   04:57 ETA
    Write failed: Cannot allocate memory
    lost connection



> The 2nd-to-last paragraph there is worth noting, specifically how
> limiting maximum addressable memory to 32GB via loader.conf seems to
> work around the issue.

I'd no longer consider this a coincidence, limiting the memory to 16G
eliminates the issue completely.  I'll retest with 32G today.

Incoming:

    obb# scp root at holp:/usr/local/tmp/testfile testfile2
    testfile                    100%   10GB  17.8MB/s   09:35
    obb# scp root at holp:/usr/local/tmp/testfile testfile2
    testfile                    100%   10GB  17.0MB/s   10:02

Outgoing:

    obb# scp testfile root at holp:/usr/local/tmp/testfile2
    testfile                    100%   10GB  35.7MB/s   04:47
    obb# scp testfile root at holp:/usr/local/tmp/testfile2
    testfile                    100%   10GB  35.4MB/s   04:49

 
> There were other problems with the systems in question back in July, it
> seems.  I assume these got hammered out somehow:
> 
> http://www.mail-archive.com/freebsd-stable@freebsd.org/msg111408.html

To a degree -- the initial install and cpu count problems are all fixed
up, thanks to help from the list.  The Intel 10G panics were stifled
with a newer driver from Intel's site, but I ran out of time to do
any serious testing with it, and just ended up using the broadcoms to
satisfy my time constraint.

--
Mahlon E. Smith  
http://www.martini.nu/contact.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 155 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20100908/9ebeabcf/attachment.pgp