kern/64378: exec of linux binary in pxeboot diskless nfs root system causes panic

[Kris Kennaway]

The following reply was made to PR kern/64378; it has been noted by GNATS.

From: Kris Kennaway <kris at obsecurity.org>
To: freebsd-gnats-submit at FreeBSD.org
Cc:  
Subject: Re: kern/64378: exec of linux binary in pxeboot diskless nfs root system causes panic
Date: Wed, 17 Mar 2004 19:05:52 -0800

 Adding to audit trail
 
 ----- Forwarded message from mark <mark at node.to> -----
 
 X-Original-To: kkenn at localhost
 Delivered-To: kkenn at localhost.obsecurity.org
 Delivered-To: kris at freebsd.org
 Date: Thu, 18 Mar 2004 03:01:18 +0000 (GMT)
 From: mark <mark at node.to>
 Subject: Re: kern/64378: exec of linux binary in pxeboot diskless nfs root
  system causes panic
 In-reply-to: <20040317230753.GA70724 at xor.obsecurity.org>
 X-X-Sender: mark at mister.mcgoonet.com
 To: Kris Kennaway <kris at obsecurity.org>
 Reply-To: mark at node.to
 X-UIDL: /ML!!mY9!!4V`!!GR+!!
 X-Bogosity: No, tests=bogofilter, spamicity=0.000000, version=0.16.4
 
 
 I did a bunch more investigation. I can't seem to get the nfs boot system
 to dump to its speficied dumpdev, despite it having an internal disk
 device for swap (not swap over nfs), with dumpon set.
 
 However:
 
 Stupidly, I realized I had tuned some memory settings on the kernel
 running over NFS, but not the system I was comparing it too.
 
 I had raised the "KVA_PAGES" limit above the default. Removing this caused
 the system to be able to execute linux binaries without panic.
 
 I had raised this limit because this system had panicked earlier under
 high proc count/ memory utilization. It's a 2x2.6ghz xeon with 2.5 gig
 ram. I saw a thread in the maillists saying that KVA_PAGES could be raised
 to prevent the panic I had seen earlier on systems that had a lot of RAM.
 
 However, it obviously had ramifications on linux emulation (no pun
 intended).
 
 I confess to a not totally clear understanding of the intracacies of
 tuning these vars:
 KVA_PAGES
 VM_KMEM_SIZE_MAX
 VM_KMEM_SIZE_SCALE
 
 I tried to do adequate research before submitting this problem as a bug,
 but I guess I didn't do enough.
 
 --mark
 
   mark at node.to  http://node.to/~mark   7123 3F7B 10EC 7122 2F8B
   http://node.to/keys/mark.asc         B474 B09D 6ED7 3FB0 09E8
 
 
 On Wed, 17 Mar 2004, Kris Kennaway wrote:
 
 > On Wed, Mar 17, 2004 at 09:37:41AM -0800, mark wolgemuth wrote:
 >
 > > FreeBSD nonuts 5.2.1-RELEASE-p1 FreeBSD 5.2.1-RELEASE-p1 #9: Tue Mar 16 12:45:00 EST 2004     root at demon.tek.eease.com:/usr/src/sys/i386/compile/NETBOOT  i386
 > > >Description:
 > > Kernel is loaded over pxeboot + nfs. Root and usr are mounted over NFS from Netapp.
 > > Modules linux and linprocfs are loaded via loader.conf.
 > > System comes up ok, but execution of a linux binary, eg. "/compat/linux/sbin/ldconfig" causes panic. Instruction ptr is at "Xpage".
 > > I have union mounted /compat/linux under a read/write mounted fs (md in this case) so that /compat/linux/etc/ld.so.cache can be written, to rule out that problem. Since /compat/linux is writeable, so is /compat/linux/var also.
 > > >How-To-Repeat:
 > > Boot a diskless i386 system using pxeboot and root and usr on read-only nfs.
 > > Load linux kernel module. Try to run /compat/linux/ldconfig.
 >
 > Sounds like it could be a few things:
 >
 > * Stale kernel modules; they need to be built from the same sources as your kernel
 >
 > * Known unionfs bugs (see the manpage).  It's not clear whether you're
 > using unionfs or the union mount option.
 >
 > * Something else :)
 >
 > Please obtain a debugging traceback of the panic as described in:
 >
 >   http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html
 >
 > This is required in order to proceed with evaluating this PR.
 >
 > Kris
 >
 
 ----- End forwarded message -----