M_NOWAIT failure handling -- not so very rosy picture

[Robert Watson]

I'm running a diskless system with the attached patch; the results have
not been so very pleasing.  I'm collecting a set of panics and traces to
mail out to relevant developers, but it does give one pause.  A related
patch for the mbuf allocator would probably also give some interesting
results.  The patch is far from perfect, but has been enough to result in
some interesting scenarios.

# sysctl debug.malloc_failure_rate=10		# Fail one in ten

Some things to try that I've bumped into so far:

# sysctl -a > /dev/null

# mdconfig -a -s 5m -t malloc
# dd if=/dev/zero of=/dev/md0
# newfs /dev/md0

Both of these seem to be storage-related and I've e-mailed phk about them,
but I suspect there are a lot of others hanging around. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Network Associates Laboratories

Index: kern_malloc.c
===================================================================
RCS file: /home/ncvs/src/sys/kern/kern_malloc.c,v
retrieving revision 1.119
diff -u -r1.119 kern_malloc.c
--- kern_malloc.c	10 Mar 2003 20:24:54 -0000	1.119
+++ kern_malloc.c	26 Mar 2003 19:27:28 -0000
@@ -138,6 +138,20 @@
 /* time_uptime of last malloc(9) failure */
 static time_t t_malloc_fail;
 
+#ifdef MALLOC_MAKE_FAILURES
+/*
+ * Cause malloc failures ever (n) mallocs with M_NOWAIT.  If set to 0,
+ * don't cause failures.
+ */
+static int malloc_failure_rate;
+static int malloc_nowait_count;
+static int malloc_failure_count;
+SYSCTL_INT(_debug, OID_AUTO, malloc_failure_rate, CTLFLAG_RW,
+    &malloc_failure_rate, 0, "Every (n) mallocs with M_NOWAIT will fail");
+SYSCTL_INT(_debug, OID_AUTO, malloc_failure_count, CTLFLAG_RD,
+    &malloc_failure_count, 0, "Number of imposed malloc failures");
+#endif
+
 int
 malloc_last_fail(void)
 {
@@ -187,6 +201,15 @@
 #if 0
 	if (size == 0)
 		Debugger("zero size malloc");
+#endif
+#ifdef MALLOC_MAKE_FAILURES
+	if ((flags & M_NOWAIT) && (malloc_failure_rate != 0)) {
+		atomic_add_int(&malloc_nowait_count, 1);
+		if ((malloc_nowait_count % malloc_failure_rate) == 0) {
+			atomic_add_int(&malloc_failure_count, 1);
+			return (NULL);
+		}
+	}
 #endif
 	if (flags & M_WAITOK)
 		KASSERT(curthread->td_intr_nesting_level == 0,