Random Kernel Panic on Dreamplug (FS related)

[Ian Lepore]

On Tue, 2014-09-30 at 04:29 -0700, John-Mark Gurney wrote:
> Mattia Rossi wrote this message on Mon, Sep 29, 2014 at 10:42 +0200:
> > 
> > Am 29.09.2014 06:01, schrieb John-Mark Gurney:
> > >Mattia Rossi wrote this message on Fri, Sep 26, 2014 at 14:19 +0200:
> > >>This might be part of the weird FFS issues the Dreamplug has and no-one
> > >>knows why they're happening.
> > >Are you running w/ FFS journaling?  If so, try turning it off, but
> > >keeping softupdates on..
> > No journaling, no softupdates. I'll try enabling softupdates next time. 
> > don't know if it will panic though
> > >
> > >>data_abort_handler() at data_abort_handler+0x5c0
> > >>          pc = 0xc0de7a28  lr = 0xc0dd711c (exception_exit)
> > >>          sp = 0xde019898  fp = 0xde019a20
> > >>          r4 = 0xffffffff  r5 = 0xffff1004
> > >>          r6 = 0xc3f3f6c0  r7 = 0x00001000
> > >>          r8 = 0xc443e880  r9 = 0x00000000
> > >>         r10 = 0xc3d69000
> > >>exception_exit() at exception_exit
> > >>          pc = 0xc0dd711c  lr = 0xc0d53828 (ffs_truncate+0xaa8)
> > >>          sp = 0xde0198e8  fp = 0xde019a20
> > >>          r0 = 0xd0238120  r1 = 0x00000e60
> > >>          r2 = 0x00000000  r3 = 0x00000000
> > >>          r4 = 0x00000120  r5 = 0x00000000
> > >>          r6 = 0xc3f3f6c0  r7 = 0x00001000
> > >>          r8 = 0xc443e880  r9 = 0x00000000
> > >>         r10 = 0xc3d69000 r12 = 0xd0238120
> > >>memset() at memset+0x48
> > >>          pc = 0xc0de521c  lr = 0xc0d53828 (ffs_truncate+0xaa8)
> > >>          sp = 0xde0198e8  fp = 0xde019a20
> > >>Unwind failure (no registers changed)
> > >No more beyond this?   If you could run addr2line on 0xc0d53828 so
> > >that we know where in ffs_truncate it's failing, that'd be very
> > >nice...
> > So I was trying to save the coredump in  order to reboot and run 
> > addr2line, but that failed:
> > 
> > Physical memory: 504 MB
> > Dumping 67 MB:(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 d5 1f 20 
> > 00 00 01 00
> > (da0:umass-sim0:0:0:0): CAM status: Resource Unavailable
> > (da0:umass-sim0:0:0:0): Error 5, Retries exhausted
> > Aborting dump due to I/O error.
> > 
> > ** DUMP FAILED (ERROR 5) **
> > 
> > So I guess this error is related to the CAM errors I'm getting from time 
> > to time. I was hoping that those errors were related to the INVARIANTS 
> > option that slowed down the system and thus might have triggered CAM 
> > errors, but obviously the SD Card seems to be the real issue here.
> > So no crashdump for further analysis.
> 
> That's fine.. w/ the addr2line we have some lines to explore...
> 
> > Interestingly the CAM errors didn't show up on the terminal as other 
> > times, the kernel just panicked straight away.
> 
> Hmm.. that is odd.. someone who knows the SD card layer should look
> at this part...  It could be that the SD card driver doesn't handle
> dumping (there is this global flag that gets set) properly and the driver
> needs to behave differently when it's set...
> 

On this model of dreamplug, an sdcard is just a usb drive; internally
there is a usb<->sd adapter chip, the same one used in many usb
multi-format card reader/writers.  There's no part of the
sdhci/mmc/mmcsd driver stack involved.

> > But I've got the addr2line output, even though I'm not sure it makes any 
> > difference:
> > 
> > addr2line -f -e /mnt/kernel.debug 0xc0d53828
> > 
> > ffs_truncate
> > /usr/devel/dreamplug/sys/ufs/ffs/ffs_inode.c:321
> 
> can you give me the contents of the line?  and a few lines of context
> around it?   In HEAD's source, this is DOINGASYNC, and there is no call
> to memset, nor a variable assignment that would result in memset being
> called...
> 
> > >>The sad thing is, that with fsck broken for the dreamplug, I have to
> > >>re-format the disk, reinstall everything and recreate the config files
> > >>which I didn't manage to copy to a safe place beforehand :-(
> > Didn't get around yet on checking whether the fsck issue persists if 
> > everything is compiled with gcc. Will take a bit, as I'm going to be on 
> > holidays for the next one and a half weeks. Fact is though, fsck is 
> > broken on the Dreamplug (ARMv5TE), at least when run on EVERY device 
> > that is attached via USB and if compiled with CLANG.
> > >>
> > >>Before I do that I'll leave the system in debugging mode for a few days,
> > >>in case someone can help and needs some more information.
> > Currently I've run fsck on all the disks/cards that had a working world 
> > for the dreamplug on it, so they're all gone. Can't do eny debugging 
> > atm. I'll let you know hoe the gcc build goes once I'm back from 
> > holidays though.
> 
> Hmmm. this is also worth investigating as it could be that clang is
> producing bad code somewhere...
> 

I tested with a gcc build of -current yesterday and got the same results
as with clang -- fsck on a freshly-formatted filesystem says it's
corrupted, even when the formatting was done on a non-armv4 system.

Also, this is not related to usb or sdcards per se.  You can get the
same results with a sata drive, or even md(4) (but interestingly, some
sizes of md disk have problems and some don't).

-- Ian