Random Kernel Panic on Dreamplug (FS related)

[John-Mark Gurney]

Mattia Rossi wrote this message on Mon, Sep 29, 2014 at 10:42 +0200:
> 
> Am 29.09.2014 06:01, schrieb John-Mark Gurney:
> >Mattia Rossi wrote this message on Fri, Sep 26, 2014 at 14:19 +0200:
> >>This might be part of the weird FFS issues the Dreamplug has and no-one
> >>knows why they're happening.
> >Are you running w/ FFS journaling?  If so, try turning it off, but
> >keeping softupdates on..
> No journaling, no softupdates. I'll try enabling softupdates next time. 
> don't know if it will panic though
> >
> >>data_abort_handler() at data_abort_handler+0x5c0
> >>          pc = 0xc0de7a28  lr = 0xc0dd711c (exception_exit)
> >>          sp = 0xde019898  fp = 0xde019a20
> >>          r4 = 0xffffffff  r5 = 0xffff1004
> >>          r6 = 0xc3f3f6c0  r7 = 0x00001000
> >>          r8 = 0xc443e880  r9 = 0x00000000
> >>         r10 = 0xc3d69000
> >>exception_exit() at exception_exit
> >>          pc = 0xc0dd711c  lr = 0xc0d53828 (ffs_truncate+0xaa8)
> >>          sp = 0xde0198e8  fp = 0xde019a20
> >>          r0 = 0xd0238120  r1 = 0x00000e60
> >>          r2 = 0x00000000  r3 = 0x00000000
> >>          r4 = 0x00000120  r5 = 0x00000000
> >>          r6 = 0xc3f3f6c0  r7 = 0x00001000
> >>          r8 = 0xc443e880  r9 = 0x00000000
> >>         r10 = 0xc3d69000 r12 = 0xd0238120
> >>memset() at memset+0x48
> >>          pc = 0xc0de521c  lr = 0xc0d53828 (ffs_truncate+0xaa8)
> >>          sp = 0xde0198e8  fp = 0xde019a20
> >>Unwind failure (no registers changed)
> >No more beyond this?   If you could run addr2line on 0xc0d53828 so
> >that we know where in ffs_truncate it's failing, that'd be very
> >nice...
> So I was trying to save the coredump in  order to reboot and run 
> addr2line, but that failed:
> 
> Physical memory: 504 MB
> Dumping 67 MB:(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 d5 1f 20 
> 00 00 01 00
> (da0:umass-sim0:0:0:0): CAM status: Resource Unavailable
> (da0:umass-sim0:0:0:0): Error 5, Retries exhausted
> Aborting dump due to I/O error.
> 
> ** DUMP FAILED (ERROR 5) **
> 
> So I guess this error is related to the CAM errors I'm getting from time 
> to time. I was hoping that those errors were related to the INVARIANTS 
> option that slowed down the system and thus might have triggered CAM 
> errors, but obviously the SD Card seems to be the real issue here.
> So no crashdump for further analysis.

That's fine.. w/ the addr2line we have some lines to explore...

> Interestingly the CAM errors didn't show up on the terminal as other 
> times, the kernel just panicked straight away.

Hmm.. that is odd.. someone who knows the SD card layer should look
at this part...  It could be that the SD card driver doesn't handle
dumping (there is this global flag that gets set) properly and the driver
needs to behave differently when it's set...

> But I've got the addr2line output, even though I'm not sure it makes any 
> difference:
> 
> addr2line -f -e /mnt/kernel.debug 0xc0d53828
> 
> ffs_truncate
> /usr/devel/dreamplug/sys/ufs/ffs/ffs_inode.c:321

can you give me the contents of the line?  and a few lines of context
around it?   In HEAD's source, this is DOINGASYNC, and there is no call
to memset, nor a variable assignment that would result in memset being
called...

> >>The sad thing is, that with fsck broken for the dreamplug, I have to
> >>re-format the disk, reinstall everything and recreate the config files
> >>which I didn't manage to copy to a safe place beforehand :-(
> Didn't get around yet on checking whether the fsck issue persists if 
> everything is compiled with gcc. Will take a bit, as I'm going to be on 
> holidays for the next one and a half weeks. Fact is though, fsck is 
> broken on the Dreamplug (ARMv5TE), at least when run on EVERY device 
> that is attached via USB and if compiled with CLANG.
> >>
> >>Before I do that I'll leave the system in debugging mode for a few days,
> >>in case someone can help and needs some more information.
> Currently I've run fsck on all the disks/cards that had a working world 
> for the dreamplug on it, so they're all gone. Can't do eny debugging 
> atm. I'll let you know hoe the gcc build goes once I'm back from 
> holidays though.

Hmmm. this is also worth investigating as it could be that clang is
producing bad code somewhere...

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."