kern/53350: fill up a malloc md-disk on 5.1-R causes panic
Jay Kuri
jay at oneway.com
Sun Jun 15 09:20:05 PDT 2003
>Number: 53350
>Category: kern
>Synopsis: fill up a malloc md-disk on 5.1-R causes panic
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sun Jun 15 09:20:03 PDT 2003
>Closed-Date:
>Last-Modified:
>Originator: Jay Kuri
>Release: FreeBSD 5.1-RELEASE i386
>Organization:
oneway.com
>Environment:
System: FreeBSD 5.1-RELEASE i386
>Description:
If you have a 5.1-R system with, say, 256M of ram
and you mdconfig one or several malloc md-devices totalling around 100M,
if you attempt to use them to capacity, the machine will panic. The
message given is:
panic: kmem_malloc(4096): kmem_map too small: 108347392 total allocated
I have taken some time to track down how to repeat the problem at will...
so perhaps someone can figure it out. The problem does not occur if you
have more memory, or at least, you may have to use larger md drives to do it.
I tried it on the exact same configuration, only with 4G
of ram, and there is no panic. I have also tried it on several machines,
with the same results. I would guess that it has something to do with
caching/buffering of the copied files, but that's just a guess.
I have attached a shell script that will trigger the error. I've run this
several times, usually in single-user mode, and without fail on a 256M
machine it will panic. set 'cache_dirs' to any set of directories that
will total 100M or more. On my specific config I have a default 5.1R
install with a minimal install+catman, the dirs listed in the script total
around 110M.
The script that got me started on this whole thing worked fine on
5.0-Current (Apr 30), but panic'd the machine as of 5.1R. (it's a
different script than the one below, but I removed the non-essential
parts of the script to make it less confusing) I should also mention
that the kernel is identical to 'GENERIC' except that 'options POLLING'
was included.
Attached is a script that reliably causes the problem. Below are the
results of running the script on a machine with 4G and on a machine with 256M
RAM.
- 4G log -
# df -k
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/acd0 345120 345120 0 100% /
devfs 1 1 0 100% /dev
/dev/da0s1 253888 84416 169472 33% /opt
# sh -x test.sh
+ PATH=/bin:/sbin:/usr/bin:/usr/sbin
+ cache_dirs=/bin /sbin /usr/bin /usr/sbin /usr/lib
+ sysctl+ awk {print int($2 / 1024) - 100000}
hw.physmem
+ space_in_k=3928684
+ echo Ram available to us: 3928684
Ram available to us: 3928684
+ totaldirsize=0
+ type=malloc
+ /sbin/mdconfig -a -t malloc -s 1M
+ mddev=md0
+ /sbin/newfs /dev/md0
/dev/md0: 1.0MB (2048 sectors) block size 16384, fragment size 2048
using 2 cylinder groups of 0.50MB, 32 blks, 64 inodes.
super-block backups (for fsck -b #) at:
160, 1184
+ /sbin/mount /dev/md0 /mnt
+ mkdir /mnt/copy
+ /bin/cp /bin/cp /mnt/
+ /usr/bin/du+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
-sk /bin
+ dirsize=12050
+ /bin/expr 0 + 12050
+ totaldirsize=12050
+ [ 12050 -le 3928684 ]
+ echo /bin
+ dirs_mirrored=/bin
+ /sbin/mdconfig -a -t malloc -s 12050k
+ mddevice=md1
+ /sbin/newfs -i 1024 -U /dev/md1
+ /sbin/tunefs -m 2 /dev/md1
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md1 /mnt/copy
+ /mnt/cp -Rp /bin/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/bin
+ /sbin/mount /dev/md1 /mnt/bin
+ /usr/bin/du -sk /sbin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=33913
+ /bin/expr 12050 + 33913
+ totaldirsize=45963
+ [ 45963 -le 3928684 ]
+ echo /bin /sbin
+ dirs_mirrored=/bin /sbin
+ /sbin/mdconfig -a -t malloc -s 33913k
+ mddevice=md2
+ /sbin/newfs -i 1024 -U /dev/md2
+ /sbin/tunefs -m 2 /dev/md2
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md2 /mnt/copy
+ /mnt/cp -Rp /sbin/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/sbin
+ /sbin/mount /dev/md2 /mnt/sbin
+ /usr/bin/du -sk /usr/bin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=34375
+ /bin/expr 45963 + 34375
+ totaldirsize=80338
+ [ 80338 -le 3928684 ]
+ echo /bin /sbin /usr/bin
+ dirs_mirrored=/bin /sbin /usr/bin
+ /sbin/mdconfig -a -t malloc -s 34375k
+ mddevice=md3
+ /sbin/newfs -i 1024 -U /dev/md3
+ /sbin/tunefs -m 2 /dev/md3
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md3 /mnt/copy
+ /mnt/cp -Rp /usr/bin/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/bin
+ /sbin/mount /dev/md3 /mnt/usr/bin
+ /usr/bin/du -sk /usr/sbin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=10420
+ /bin/expr 80338 + 10420
+ totaldirsize=90758
+ [ 90758 -le 3928684 ]
+ echo /bin /sbin /usr/bin /usr/sbin
+ dirs_mirrored=/bin /sbin /usr/bin /usr/sbin
+ /sbin/mdconfig -a -t malloc -s 10420k
+ mddevice=md4
+ /sbin/newfs -i 1024 -U /dev/md4
+ /sbin/tunefs -m 2 /dev/md4
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md4 /mnt/copy
+ /mnt/cp -Rp /usr/sbin/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/sbin
+ /sbin/mount /dev/md4 /mnt/usr/sbin
+ /usr/bin/du -sk /usr/lib
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=49560
+ /bin/expr 90758 + 49560
+ totaldirsize=140318
+ [ 140318 -le 3928684 ]
+ echo /bin /sbin /usr/bin /usr/sbin /usr/lib
+ dirs_mirrored=/bin /sbin /usr/bin /usr/sbin /usr/lib
+ /sbin/mdconfig -a -t malloc -s 49560k
+ mddevice=md5
+ /sbin/newfs -i 1024 -U /dev/md5
+ /sbin/tunefs -m 2 /dev/md5
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md5 /mnt/copy
+ /mnt/cp -Rp /usr/lib/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/lib
+ /sbin/mount /dev/md5 /mnt/usr/lib
# df -k
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/acd0 345120 345120 0 100% /
devfs 1 1 0 100% /dev
/dev/da0s1 253888 84416 169472 33% /opt
/dev/md0 846 110 670 14% /mnt
/dev/md1 10304 9528 570 94% /mnt/bin
/dev/md2 29414 28644 182 99% /mnt/sbin
/dev/md3 29812 29088 128 100% /mnt/usr/bin
/dev/md4 8866 8270 420 95% /mnt/usr/sbin
/dev/md5 43142 43074 -794 102% /mnt/usr/lib
# top
last pid: 136; load averages: 0.01, 0.08, 0.06 up 0+00:05:06 15:40:18
2 processes: 1 running, 1 sleeping
Mem: 56M Active, 17M Inact, 168M Wired, 42M Cache, 69M Buf, 3608M Free
Swap:
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
44 root 8 0 896K 444K wait 0:00 0.00% 0.00% sh
136 root 96 0 2136K 1300K RUN 0:00 0.00% 0.00% top
- test log 256M -
# top
last pid: 54; load averages: 0.35, 0.24, 0.10 up 0+00:02:05 16:17:12
2 processes: 1 running, 1 sleeping
Mem: 1184K Active, 268K Inact, 8328K Wired, 3664K Buf, 235M Free
Swap:
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
54 root 96 0 2136K 780K RUN 0:00 0.00% 0.00% top
48 root 8 0 900K 320K wait 0:00 0.00% 0.00% sh
# df -k
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/acd0 345120 345120 0 100% /
devfs 1 1 0 100% /dev
/dev/da0s1 253888 84416 169472 33% /opt
# sh -x test.sh
+ PATH=/bin:/sbin:/usr/bin:/usr/sbin
+ cache_dirs=/bin /sbin /usr/bin /usr/sbin /usr/lib
+ sysctl+ awk {print int($2 / 1024) - 100000}
hw.physmem
+ space_in_k=154980
+ echo Ram available to us: 154980
Ram available to us: 154980
+ totaldirsize=0
+ type=malloc
+ /sbin/mdconfig -a -t malloc -s 1M
+ mddev=md0
+ /sbin/newfs /dev/md0
/dev/md0: 1.0MB (2048 sectors) block size 16384, fragment size 2048
using 2 cylinder groups of 0.50MB, 32 blks, 64 inodes.
super-block backups (for fsck -b #) at:
160, 1184
+ /sbin/mount /dev/md0 /mnt
+ mkdir /mnt/copy
+ /bin/cp /bin/cp /mnt/
+ /usr/bin/du+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
-sk /bin
+ dirsize=12050
+ /bin/expr 0 + 12050
+ totaldirsize=12050
+ [ 12050 -le 154980 ]
+ echo /bin
+ dirs_mirrored=/bin
+ /sbin/mdconfig -a -t malloc -s 12050k
+ mddevice=md1
+ /sbin/newfs -i 1024 -U /dev/md1
+ /sbin/tunefs -m 2 /dev/md1
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md1 /mnt/copy
+ /mnt/cp -Rp /bin/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/bin
+ /sbin/mount /dev/md1 /mnt/bin
+ /usr/bin/du -sk /sbin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=33913
+ /bin/expr 12050 + 33913
+ totaldirsize=45963
+ [ 45963 -le 154980 ]
+ echo /bin /sbin
+ dirs_mirrored=/bin /sbin
+ /sbin/mdconfig -a -t malloc -s 33913k
+ mddevice=md2
+ /sbin/newfs -i 1024 -U /dev/md2
+ /sbin/tunefs -m 2 /dev/md2
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md2 /mnt/copy
+ /mnt/cp -Rp /sbin/ /mnt/copy
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/sbin
+ /sbin/mount /dev/md2 /mnt/sbin
+ /usr/bin/du -sk /usr/bin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=34375
+ /bin/expr 45963 + 34375
+ totaldirsize=80338
+ [ 80338 -le 154980 ]
+ echo /bin /sbin /usr/bin
+ dirs_mirrored=/bin /sbin /usr/bin
+ /sbin/mdconfig -a -t malloc -s 34375k
+ mddevice=md3
+ /sbin/newfs -i 1024 -U /dev/md3
+ /sbin/tunefs -m 2 /dev/md3
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md3 /mnt/copy
+ /mnt/cp -Rp /usr/bin/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/bin
+ /sbin/mount /dev/md3 /mnt/usr/bin
+ /usr/bin/du -sk /usr/sbin
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=10420
+ /bin/expr 80338 + 10420
+ totaldirsize=90758
+ [ 90758 -le 154980 ]
+ echo /bin /sbin /usr/bin /usr/sbin
+ dirs_mirrored=/bin /sbin /usr/bin /usr/sbin
+ /sbin/mdconfig -a -t malloc -s 10420k
+ mddevice=md4
+ /sbin/newfs -i 1024 -U /dev/md4
+ /sbin/tunefs -m 2 /dev/md4
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md4 /mnt/copy
+ /mnt/cp -Rp /usr/sbin/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
+ /sbin/umount /mnt/copy
+ /bin/mkdir -p /mnt/usr/sbin
+ /sbin/mount /dev/md4 /mnt/usr/sbin
+ /usr/bin/du -sk /usr/lib
+ /usr/bin/awk {print int(1024 + ($1 * 1.20))}
+ dirsize=49560
+ /bin/expr 90758 + 49560
+ totaldirsize=140318
+ [ 140318 -le 154980 ]
+ echo /bin /sbin /usr/bin /usr/sbin /usr/lib
+ dirs_mirrored=/bin /sbin /usr/bin /usr/sbin /usr/lib
+ /sbin/mdconfig -a -t malloc -s 49560k
+ mddevice=md5
+ /sbin/newfs -i 1024 -U /dev/md5
+ /sbin/tunefs -m 2 /dev/md5
tunefs: minimum percentage of free space changes from 8% to 2%
tunefs: should optimize for space with minfree < 8%
+ /sbin/mount /dev/md5 /mnt/copy
+ /mnt/cp -Rp /usr/lib/ /mnt/copy
/mnt/copy: optimization changed from TIME to SPACE
panic: kmem_malloc(4096): kmem_map too small: 108347392 total allocated
syncing disks, buffers remaining...
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x14
fault code = supervisor write, page not present
instruction pointer = 0x8:0xc02d1ad9
stack pointer = 0x10:0xd24fb544
frame pointer = 0x10:0xd24fb578
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 137 (cp)
trap number = 12
panic: page fault
Uptime: 3m12s
Terminate ACPI
Automatic reboot in 15 seconds - press a key on the console to abort
>How-To-Repeat:
On a 5.1-R machine with 256M of ram, boot single-user and run this script
(ensure that the size of the dirs in cache_dirs is 100M or more)
#!/bin/sh
## quick script to show the mdconfig -t malloc + fill = panic bug on 5.1-R
PATH=/bin:/sbin:/usr/bin:/usr/sbin
# on a machine with 256M of ram, this is enough to crash it. 112M roughly on
# my machine. Use any combination of directories that will push the limit
# up over 100M
cache_dirs="/bin /sbin /usr/bin /usr/sbin /usr/lib"
# say we have 100M less than we actually have, to make sure we don't
# try to use more memory than we have
space_in_k=`sysctl hw.physmem|awk '{print int($2 / 1024) - 100000}'`
echo Ram available to us: $space_in_k
totaldirsize=0
type="malloc"
## This gives us a writable directory so that we can run this off of the 5.1
## install or live cd.
mddev=`/sbin/mdconfig -a -t malloc -s 1M`
/sbin/newfs /dev/${mddev}
/sbin/mount /dev/${mddev} /mnt
mkdir /mnt/copy
/bin/cp /bin/cp /mnt/
for dir in ${cache_dirs}
do
dirsize=`/usr/bin/du -sk ${dir}|/usr/bin/awk '{print int(1024 + ($1 * 1.20
))}'`
totaldirsize=`/bin/expr ${totaldirsize} + ${dirsize}`
if [ ${totaldirsize} -le ${space_in_k} ]; then
dirs_mirrored=`echo ${dirs_mirrored} $dir`
mddevice=`/sbin/mdconfig -a -t ${type} -s ${dirsize}k`
/sbin/newfs -i 1024 -U /dev/${mddevice} >/dev/null
/sbin/tunefs -m 2 /dev/${mddevice}
/sbin/mount /dev/${mddevice} /mnt/copy >/dev/null
/mnt/cp -Rp ${dir}/ /mnt/copy
/sbin/umount /mnt/copy >/dev/null
/bin/mkdir -p /mnt${dir}
/sbin/mount /dev/${mddevice} /mnt${dir} >/dev/null
fi
done
>Fix:
None that I know of.
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list