stack hogs in kernel

[Peter Wemm]

On Tue, Apr 15, 2008 at 12:38 AM, Jeff Roberson <jroberson at jroberson.net> wrote:
> On Tue, 15 Apr 2008, Andrew Reilly wrote:
> > On Sat, Apr 12, 2008 at 08:16:01PM +0200, Roman Divacky wrote:
> > > On Sat, Apr 12, 2008 at 07:14:21PM +0100, Robert Watson wrote:
> > > > On Fri, 11 Apr 2008, Julian Elischer wrote:
> > > > > 0xc05667e3 kldstat [kernel]:                            2100
> > > > > 0xc07214f8 sendsig [kernel]:                            1416
> > > > > 0xc04fb426 ugenread [kernel]:                           1200
> > > > > 0xc070616b ipmi_smbios_identify [kernel]:               1136
> > > > > 0xc050bd26 usbd_new_device [kernel]:                    1128
> > > > > 0xc0525a83 pfs_readlink [kernel]:                       1092
> > > > > 0xc04fb407 ugenwrite [kernel]:                          1056
> > > > > 0xc055ea33 prison_enforce_statfs [kernel]:              1044
> > > > >
> > > >
> > > > This one, at least, is due to an issue Roman pointed out on hackers@
> in the
> > > > last 24 hours -- a MAXPATHLEN sized buffer on the stack.  Looks like
> > > > pfs_readlink() has the same issue.
> > > >
> > >
> > > I plan to look at some of the MAXPATHLEN usage... I guess we can shave a
> few
> > > tens of KBs from the kernel (static size and runtime size).
> > >
> >
> > Why are single-digit kilobytes of memory space interesting, in this
> > context?  Is the concern about L1 data cache footprint, for performance
> > reasons?  If that is the case, the MAXPATHLEN bufffer will only really
> > occupy the amount of cache actually touched.
> >
> > I've long wondered about the seemingly fanatical stack size concern in
> > kernel space.  In other domains (where I have more experience) you can
> > get good performance benefits from the essentially free memory management
> > and good cache re-use that comes from putting as much into the
> > stack/call-frame as possible.
> >
>
>  There is a small fixed kernel stack per-thread.  It has to be allocated
> up-front out of kernel memory.  There isn't really enough KVA to just allow
> kernel stacks to grow unbounded.  Also consider that most of the time this
> memory is just unused.
>
>  Right now on amd64 we allocate 4 pages for kernel stacks!  This is just
> huge.  It makes allocation slower and more likely to fail since we have to
> find 5 contiguous pages (one for a guard page).

I'd like to see this reduced by one or two.  We should be able to get
away with the same stack size as i386 - the pcb size isn't that much
different.  At the very least, 3 pages + 1 guard page would get the
size down to a power of two.  I don't know if that'll help the kva
allocator, but it might.

I originally chose a KSTACK_PAGES of 4, simply out of conservatism -
not by measurement.  I just didn't want to have to worry about it at
the time.  It is fairly likely that shrinking it by a page will Just
Work since i386 is running with KSTACK_PAGES = 2.

Also, kernel stacks are allocated out of paged kva, so on amd64 this
means they come out of the 2GB kernel area, not direct map.

-- 
Peter Wemm - peter at wemm.org; peter at FreeBSD.org; peter at yahoo-inc.com
"All of this is for nothing if we don't go to the stars" - JMS/B5
"If Java had true garbage collection, most programs would delete
themselves upon execution." -- Robert Sewell
**WANTED TO BUY: Garmin Streetpilot 2650 or 2660. Not later model! **