FreeBSD 10.1 Memory Exhaustion
Christopher Forgeron
csforgeron at gmail.com
Mon Jul 13 12:48:40 UTC 2015
Hey, thanks, that's sounding exactly like my issue.
I'll patch and build that kernel this week and have it up for testing on at
least one of my boxes.
While I agree that ZFS is the instigator of this behaviour, shouldn't the
kernel protect against this type of behaviour?
What would happen if a web server started chewing memory aggressively due
to a DoS attack?
On Mon, Jul 13, 2015 at 8:58 AM, Karl Denninger <karl at denninger.net> wrote:
> Put this on your box and see if the problem goes away.... :-)
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594
>
> The 2015-02-10 refactor will apply against 10.1-STABLE and 10.2-PRE (the
> latter will give you a 10-line fuzz in one block but applies and works.)
>
> I've been unable to provoke misbehavior with this patch in and I run a
> cron job that does auto-snapshotting. There are others that have run
> this patch with similarly positive results.
>
> On 7/13/2015 06:48, Christopher Forgeron wrote:
> > TL;DR Summary: I can run FreeBSD out of memory quite consistently, and
> it’s
> > not a TOS/mbuf exhaustion issue. It’s quite possible that ZFS is the
> > culprit, but shouldn’t the pager be able to handle aggressive memory
> > requests in a low memory situation gracefully, without needing custom
> > tuning of ZFS / VM?
> >
> >
> > Hello,
> >
> > I’ve been dealing with some instability in my 10.1-RELEASE and
> > STABLEr282701M machines for the last few months.
> >
> > These machines are NFS/iSCSI storage machines, running on Dell M610x or
> > similar hardware, 96 Gig Memory, 10Gig Network Cards, dual Xeon
> Processors
> > – Fairly beefy stuff.
> >
> > Initially I thought it was more issues with TOS / jumbo mbufs, as I had
> > this problem last year. I had thought that this was properly resolved,
> but
> > setting my MTU to 1500, and turning off TOS did give me a bit more
> > stability. Currently all my machines are set this way.
> >
> > Crashes were usually represented by loss of network connectivity, and the
> > ctld daemon scrolling messages across the screen at full speed about lost
> > connections.
> >
> > All of this did seem like more network stack problems, but with each
> crash
> > I’d be able to learn a bit more.
> >
> > Usually there was nothing of any use in the logfile, but every now and
> then
> > I’d get this:
> >
> > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97
> > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory
> > Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80
> > bytes
> > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97
> > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory
> > Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80
> > bytes
> > Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97
> > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory
> > ---------
> > Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80
> > bytes
> > Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate 80
> > bytes
> > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97
> > (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory
> > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97
> > (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping
> > connection
> > Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97
> > (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping
> > connection
> > Jun 4 03:03:10 san0 kernel: WARNING: 172.16.0.97
> > (iqn.1998-01.com.vmware:esx5a-3387a188): waiting for CTL to terminate
> > tasks, 1 remaining
> > Jun 4 06:04:27 san0 syslogd: kernel boot file is /boot/kernel/kernel
> >
> > So knowing that it seemed to be running out of memory, I started leaving
> > leaving ‘vmstat 5’ running on a console, to see what it was displaying
> > during the crash.
> >
> > It was always the same thing:
> >
> > 0 0 0 1520M 4408M 15 0 0 0 25 19 0 0 21962 1667
> 91390
> > 0 33 67
> > 0 0 0 1520M 4310M 9 0 0 0 2 15 3 0 21527 1385
> 95165
> > 0 31 69
> > 0 0 0 1520M 4254M 7 0 0 0 14 19 0 0 17664 1739
> 72873
> > 0 18 82
> > 0 0 0 1520M 4145M 2 0 0 0 0 19 0 0 23557 1447
> 96941
> > 0 36 64
> > 0 0 0 1520M 4013M 4 0 0 0 14 19 0 0 4288 490 34685
> > 0 72 28
> > 0 0 0 1520M 3885M 2 0 0 0 0 19 0 0 11141 1038
> 69242
> > 0 52 48
> > 0 0 0 1520M 3803M 10 0 0 0 14 19 0 0 24102 1834
> 91050
> > 0 33 67
> > 0 0 0 1520M 8192B 2 0 0 0 2 15 1 0 19037 1131
> 77470
> > 0 45 55
> > 0 0 0 1520M 8192B 0 22 0 0 2 0 6 0 146 82
> 578 0
> > 0 100
> > 0 0 0 1520M 8192B 1 0 0 0 0 0 0 0 130 40
> 510 0
> > 0 100
> > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 143 40
> 501 0
> > 0 100
> > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 201 62
> 660 0
> > 0 100
> > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 101 28
> 404 0
> > 0 100
> > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 97 27
> 398 0
> > 0 100
> > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 93 28
> 377 0
> > 0 100
> > 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 92 27
> 373 0
> > 0 100
> >
> >
> > I’d go from a decent amount of free memory to suddenly having none.
> Vmstat
> > would stop outputting, console commands would hang, etc. The whole system
> > would be useless.
> >
> > Looking into this, I came across a similar issue;
> >
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199189
> >
> > I started increasing v.v_free_min, and it helped – My crashes went from
> > being ~every 6 hours to every few days.
> >
> > Currently I’m running with vm.v_free_min=1254507 – That’s (1254507 *
> 4KiB)
> > , or 4.78GiB of Reserve. The vmstat above is of a machine with that
> > setting still running to 8B of memory.
> >
> > I have two issues here:
> >
> > 1) I don’t think I should ever be able to run the system into the ground
> on
> > memory. Deny me new memory until the pager can free more.
> > 2) Setting ‘min’ doesn’t really mean ‘min’ as it can obviously go below
> > that threshold.
> >
> >
> > I have plenty of local UFS swap (non-ZFS drives)
> >
> > Adrian requested that I output a few more diagnostic items, and this is
> > what I’m running on a console now, in a loop:
> >
> > vmstat
> > netstat -m
> > vmstat -z
> > sleep 1
> >
> > The output of four crashes are attached here, as they can be a bit long.
> > Let me know if that’s not a good way to report them. They will each start
> > mid-way through a vmstat –z output, as that’s as far back as my terminal
> > buffer allows.
> >
> >
> >
> > Now, I have a good idea of the conditions that are causing this: ZFS
> > Snapshots, run by cron, during times of high ZFS writes.
> >
> > The crashes are all nearly on the hour, as that’s when crontab triggers
> my
> > python scripts to make new snapshots, and delete old ones.
> >
> > My average FreeBSD machine has ~ 30 zfs datasets, with each pool having
> ~20
> > TiB used. These all need to snapshot on the hour.
> >
> > By staggering the snapshots by a few minutes, I have been able to reduce
> > crashing from every other day to perhaps once a week if I’m lucky – But
> if
> > I start moving a lot of data around, I can cause daily crashes again.
> >
> > It’s looking to be the memory demand of snapshotting lots of ZFS datasets
> > at the same time while accepting a lot of write traffic.
> >
> > Now perhaps the answer is ‘don’t do that’ but I feel that FreeBSD should
> be
> > robust enough to handle this. I don’t mind tuning for now to
> > reduce/eliminate this, but others shouldn’t run into this pain just
> because
> > they heavily load their machines – There must be a way of avoiding this
> > condition.
> >
> > Here are the contents of my /boot/loader.conf and sysctl.conf, so show my
> > minimal tuning to make this problem a little more bearable:
> >
> > /boot/loader.conf
> > vfs.zfs.arc_meta_limit=49656727553
> > vfs.zfs.arc_max = 91489280512
> >
> > /etc/sysctl.conf
> > vm.v_free_min=1254507
> >
> >
> > Any suggestions/help is appreciated.
> >
> > Thank you.
> >
> >
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>
> --
> Karl Denninger
> karl at denninger.net <mailto:karl at denninger.net>
> /The Market Ticker/
> /[S/MIME encrypted email preferred]/
>
More information about the freebsd-fs
mailing list