Weird kernel mode data abort panic on Zedboard.
Thomas Skibo
ThomasSkibo at sbcglobal.net
Tue Mar 5 00:49:40 UTC 2013
Hello, again.
I thought I'd describe this kernel panic and see if anyone has seen
anything similar. This is on the Zedboard (ARM Cortex-A9 armv6).
I can consistently panic the kernel by ftp'ing files to a local
filesystem. It crashes every fourth time or so in the same manner. The
specific fault is a 'Permission Fault (P)' on the first page of data in
a file buf.
The fault occurs in ffs_write() when it attempts to uiomove/copyin data
from user space to the file buf. The first write faults but copyin()
catches the fault and then calls vfs_bio_clrbuf() which tries to zero
out the buf data. It's the second fault that causes the panic.
Using Xilinx's XMD tool, I can walk the page table in physical memory
and find the PTE. It is always okay and has priviledged read and write
permissions. Curiously, sometimes I get another permission fault if I
try to read from the fault address in the debugger. That tells me there
is a stale TLB entry without read permissions either. The other buf
pages seem okay.
Any ideas? I can duplicate this with l2 cache turned off. I can also
duplicate it using a USB-ethernet interface instead of the Zynq cgem
ethernet interface so it's not that driver.
Thanks,
--Thomas
============================================================
(ftp data from a remote system to Zedboard...)
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd tftpboot
250 Directory successfully changed.
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||30574|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
0% | | 0 0.00 KiB/s
--:-- ETA100% |***********************************| 4078 KiB 27.55
MiB/s 00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.45 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||32024|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
0% | | 0 0.00 KiB/s
--:-- ETA100% |***********************************| 4078 KiB 27.79
MiB/s 00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.71 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||59472|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
0% | | 0 0.00 KiB/s
--:-- ETA100% |***********************************| 4078 KiB 27.81
MiB/s 00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.73 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||18618|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
0% | | 0 0.00 KiB/s
--:-- ETA100% |***********************************| 4078 KiB 27.90
MiB/s 00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.81 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||8385|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
0% | | 0 0.00 KiB/s
--:-- ETA100% |***********************************| 4078 KiB 27.83
MiB/s 00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.75 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||45403|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
0% | | 0 0.00 KiB/s
--:-- ETA100% |***********************************| 4078 KiB 27.92
MiB/s 00:00 ETA
226 Transfer complete.
4176120 bytes received in 00:00 (27.83 MiB/s)
ftp> get kernel.bin
local: kernel.bin remote: kernel.bin
229 Entering Extended Passive Mode (|||34019|).
150 Opening BINARY mode data connection for kernel.bin (4176120 bytes).
0% | | 0 0.00 KiB/s
--:-- ETA
vm_fault(0xc0610000, cdf2c000, 2, 0) -> 2
Fatal kernel mode data abort: 'Permission Fault (P)'
trapframe: 0xd67eda68
FSR=0000080f, FAR=cdf2c000, spsr=20000013
r0 =cdf2c000, r1 =00000f80, r2 =00000000, r3 =00000000
r4 =cdf2c000, r5 =00000000, r6 =00000000, r7 =cdf2d000
r8 =000000ff, r9 =00000000, r10=cd385660, r11=d67edae0
r12=cdf2c000, ssp=d67edab4, slr=c02d27a8, pc =c043039c
[ thread pid 619 tid 100040 ]
Stopped at memset+0x48: undge 0xa0cc20f8
db> show buffer $r10
===== faulty address is always first page of buf =====
buf at 0xcd385660
b_flags = 0x20000000<vmio>, b_xflags=0x2<clean>, b_vflags=0x0
b_error = 0, b_bufsize = 32768, b_bcount = 32768, b_resid = 0
b_bufobj = (0xc3028954), b_data = 0xcdf2c000, b_blkno = 2432, b_lblkno =
24, b_dep = 0
b_npages = 8, pages(OBJ, IDX, PA): (0xc301dc24, 0xc0,
0x402a000),(0xc301dc24, 0xc1, 0x402b000),(0xc301dc24, 0xc2,
0x4250000),(0xc301dc24, 0xc3, 0x4251000),(0xc301dc24, 0xc4,
0x4252000),(0xc301dc24, 0xc5, 0x4253000),(0xc301dc24, 0xc6,
0x4274000),(0xc301dc24, 0xc7, 0x4275000)
lock type bufwait: EXCL by thread 0xc2ef7000 (pid 619, ftp, tid 100040)
db> bt
Tracing pid 619 tid 100040 td 0xc2ef7000
db_trace_self() at db_trace_self+0xc
scp=0xc04236fc rlv=0xc0423748 (db_trace_thread+0x38)
rsp=0xd67ed77c rfp=0xd67ed788
db_trace_thread() at db_trace_thread+0xc
scp=0xc042371c rlv=0xc012b708 (db_command_init+0x354)
rsp=0xd67ed78c rfp=0xd67ed7a8
db_command_init() at db_command_init+0x27c
scp=0xc012b630 rlv=0xc012b10c (db_skip_to_eol+0x4a0)
rsp=0xd67ed7ac rfp=0xd67ed850
r5=0x00000000 r4=0xc04d382c
db_skip_to_eol() at db_skip_to_eol+0x1d4
scp=0xc012ae40 rlv=0xc012b278 (db_command_loop+0x60)
rsp=0xd67ed854 rfp=0xd67ed860
r10=0x60000093 r8=0x0000080f
r7=0x00000000 r6=0xcdf2c000 r5=0xc04d3af4 r4=0xd67ed86c
db_command_loop() at db_command_loop+0xc
scp=0xc012b224 rlv=0xc012d758 (X_db_sym_numargs+0xf4)
rsp=0xd67ed864 rfp=0xd67ed980
X_db_sym_numargs() at X_db_sym_numargs+0x14
--More-- scp=0xc012d678 rlv=0xc02897a4 (kdb_trap+0xa4)
rsp=0xd67ed984 rfp=0xd67ed9a8
r4=0xd67eda68
kdb_trap() at kdb_trap+0xc
scp=0xc028970c rlv=0xc04329b4 (badaddr_read+0x284)
rsp=0xd67ed9ac rfp=0xd67ed9c8
r10=0x00000000 r8=0xd67eda68
r7=0xc2ef7000 r6=0xcdf2c000 r5=0x0000080f r4=0xd67eda68
badaddr_read() at badaddr_read+0xfc
scp=0xc043282c rlv=0xc0432ef0 (data_abort_handler+0x4e4)
rsp=0xd67ed9cc rfp=0xd67eda64
r6=0xd67edef8 r5=0xc2eec8ac
r4=0xcdf2c000
data_abort_handler() at data_abort_handler+0xc
scp=0xc0432a18 rlv=0xc0424f00 (address_exception_entry+0x50)
rsp=0xd67eda68 rfp=0xd67edae0
r10=0xcd385660 r9=0x00000000
r8=0x000000ff r7=0xcdf2d000 r6=0x00000000 r5=0x00000000
r4=0xcdf2c000
vfs_bio_clrbuf() at vfs_bio_clrbuf+0xc
--More-- scp=0xc02d2594 rlv=0xc03d7970 (ffs_syncvnode+0xb4c)
rsp=0xd67edae4 rfp=0xd67edb5c
r10=0x00008000 r9=0x00000000
r8=0x00008000 r7=0x0000000f r6=0x00000000 r5=0x00008000
r4=0xc2f61000
ffs_syncvnode() at ffs_syncvnode+0x6bc
scp=0xc03d74e0 rlv=0xc04448c8 (VOP_WRITE_APV+0x104)
rsp=0xd67edb60 rfp=0xd67edc08
r10=0x00000000 r9=0x000c0000
r8=0x00000000 r7=0x00000000 r6=0x00000000 r5=0xd67edc24
r4=0xc04f5f00
VOP_WRITE_APV() at VOP_WRITE_APV+0xc
scp=0xc04447d0 rlv=0xc02ff4b8 (vn_extattr_get+0x2e0)
rsp=0xd67edc0c rfp=0xd67edc6c
r8=0xc2f25ce8 r7=0xc30288e0
r6=0x00000000 r5=0xd67edc24 r4=0x00000000
vn_extattr_get() at vn_extattr_get+0x158
scp=0xc02ff330 rlv=0xc02fd9a0 (foffset_lock_uio+0x1f8)
rsp=0xd67edc70 rfp=0xd67edd40
r10=0xc30288e0 r9=0x00010000
--More-- r8=0x00000000 r7=0x00000000 r6=0xc2f25ce8
r5=0xd67eddb4
r4=0x7fffffff
foffset_lock_uio() at foffset_lock_uio+0x38
scp=0xc02fd7e0 rlv=0xc029ce48 (sys_ioctl+0x210)
rsp=0xd67edd44 rfp=0xd67edd70
r10=0x00010000 r9=0x00000005
r8=0x00000000 r7=0xc2f25ce8 r6=0xc2ef7000 r5=0xd67eddb4
r4=0xffffffff
sys_ioctl() at sys_ioctl+0x174
scp=0xc029cdac rlv=0xc029d190 (kern_writev+0x60)
rsp=0xd67edd74 rfp=0xd67edda8
r10=0x00000000 r9=0x00000000
r8=0xc2ef7000 r7=0xd67eddb4 r6=0x00000005 r5=0x00000000
r4=0x00000000
kern_writev() at kern_writev+0xc
scp=0xc029d13c rlv=0xc029d230 (sys_write+0x58)
rsp=0xd67eddac rfp=0xd67edde0
r8=0x00000000 r7=0x00000000
r6=0x00000000 r5=0xc2ef7000 r4=0xc2eec8ac
sys_write() at sys_write+0xc
--More-- scp=0xc029d1e4 rlv=0xc0433570 (swi_handler+0x49c)
rsp=0xd67edde4 rfp=0xd67edea4
swi_handler() at swi_handler+0xc
scp=0xc04330e0 rlv=0xc0424d34 (swi_entry+0x30)
rsp=0xd67edea8 rfp=0xbfffe2e0
r7=0x00000000 r6=0xd67edeac
r5=0x00010000 r4=0x2089d000
fiqvector() at 0x170f8
scp=0x000170f8 rlv=0x00019f94 (0x19f94)
rsp=0xbfffe2e4 rfp=0xbfffe3d8
r10=0x20803130 r9=0x00036614
r8=0x00024058 r7=0x00000000 r6=0x00000001 r5=0x00000000
r4=0x00000004
fiqvector() at 0x19824
scp=0x00019824 rlv=0x00010424 (0x10424)
rsp=0xbfffe3dc rfp=0xbfffe884
r7=0x20803130 r6=0x00000001
r5=0x00000000 r4=0xbfffe8a0
fiqvector() at 0x101b4
scp=0x000101b4 rlv=0x00010634 (0x10634)
--More-- rsp=0xbfffe888 rfp=0xbfffe894
r10=0x00000000 r9=0x00000000
r8=0x00000000 r7=0xbfffee64 r6=0x20808400 r5=0x00030048
r4=0xbfffe8a0
fiqvector() at 0x10614
scp=0x00010614 rlv=0x0001c82c (0x1c82c)
rsp=0xbfffe898 rfp=0xbfffe8c8
fiqvector() at 0x1c4a0
scp=0x0001c4a0 rlv=0x0001d72c (0x1d72c)
rsp=0xbfffe8cc rfp=0xbfffed10
r6=0x00000001 r5=0x00037354
r4=0x00034bfc
fiqvector() at 0x1c94c
scp=0x0001c94c rlv=0x0000aa58 (0xaa58)
rsp=0xbfffed14 rfp=0xbfffed34
r10=0x2004eb90 r8=0x00000000
r7=0x00000000 r6=0xbfffed54 r5=0xbfffed48 r4=0x00000002
fiqvector() at 0xa928
scp=0x0000a928 rlv=0x20038144 (0x20038144)
rsp=0xbfffed38 rfp=0x00000000
--More-- r8=0x00000000 r7=0x00000000
r6=0xbfffed44 r5=0x20037db8 r4=0x20053000
====== (now debugger faults trying to read the same address) ======
db> x 0xcdf2c000
0xcdf2c000:
vm_fault(0xc0610000, cdf2c000, 2, 0) -> 2
Fatal kernel mode data abort: 'Permission Fault (P)'
trapframe: 0xd67ed6f0
FSR=0000000f, FAR=cdf2c000, spsr=60000093
r0 =00000000, r1 =c2ef7000, r2 =00000004, r3 =c051bac8
r4 =00000004, r5 =cdf2c000, r6 =00000000, r7 =d67ed760
r8 =cdf2c000, r9 =00000002, r10=00000010, r11=d67ed75c
r12=00000001, ssp=d67ed73c, slr=c0429c04, pc =c0423174
panic: Fatal abort
Uptime: 49s
Automatic reboot in 15 seconds - press a key on the console to abort
--
--------
Thomas Skibo
ThomasSkibo at sbcglobal.net
More information about the freebsd-arm
mailing list