Random Kernel Panic on Dreamplug (FS related)

[Mattia Rossi]

Am 30.09.2014 13:29, schrieb John-Mark Gurney:
> Mattia Rossi wrote this message on Mon, Sep 29, 2014 at 10:42 +0200:
>> Am 29.09.2014 06:01, schrieb John-Mark Gurney:
>>> Mattia Rossi wrote this message on Fri, Sep 26, 2014 at 14:19 +0200:
>>>> This might be part of the weird FFS issues the Dreamplug has and no-one
>>>> knows why they're happening.
>>> Are you running w/ FFS journaling?  If so, try turning it off, but
>>> keeping softupdates on..
>> No journaling, no softupdates. I'll try enabling softupdates next time.
>> don't know if it will panic though
>>>> data_abort_handler() at data_abort_handler+0x5c0
>>>>           pc = 0xc0de7a28  lr = 0xc0dd711c (exception_exit)
>>>>           sp = 0xde019898  fp = 0xde019a20
>>>>           r4 = 0xffffffff  r5 = 0xffff1004
>>>>           r6 = 0xc3f3f6c0  r7 = 0x00001000
>>>>           r8 = 0xc443e880  r9 = 0x00000000
>>>>          r10 = 0xc3d69000
>>>> exception_exit() at exception_exit
>>>>           pc = 0xc0dd711c  lr = 0xc0d53828 (ffs_truncate+0xaa8)
>>>>           sp = 0xde0198e8  fp = 0xde019a20
>>>>           r0 = 0xd0238120  r1 = 0x00000e60
>>>>           r2 = 0x00000000  r3 = 0x00000000
>>>>           r4 = 0x00000120  r5 = 0x00000000
>>>>           r6 = 0xc3f3f6c0  r7 = 0x00001000
>>>>           r8 = 0xc443e880  r9 = 0x00000000
>>>>          r10 = 0xc3d69000 r12 = 0xd0238120
>>>> memset() at memset+0x48
>>>>           pc = 0xc0de521c  lr = 0xc0d53828 (ffs_truncate+0xaa8)
>>>>           sp = 0xde0198e8  fp = 0xde019a20
>>>> Unwind failure (no registers changed)
>>> No more beyond this?   If you could run addr2line on 0xc0d53828 so
>>> that we know where in ffs_truncate it's failing, that'd be very
>>> nice...
>> So I was trying to save the coredump in  order to reboot and run
>> addr2line, but that failed:
>>
>> Physical memory: 504 MB
>> Dumping 67 MB:(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 01 d5 1f20
>> 00 00 01 00  <sip:2000000100>
>> (da0:umass-sim0:0:0:0): CAM status: Resource Unavailable
>> (da0:umass-sim0:0:0:0): Error 5, Retries exhausted
>> Aborting dump due to I/O error.
>>
>> ** DUMP FAILED (ERROR 5) **
>>
>> So I guess this error is related to the CAM errors I'm getting from time
>> to time. I was hoping that those errors were related to the INVARIANTS
>> option that slowed down the system and thus might have triggered CAM
>> errors, but obviously the SD Card seems to be the real issue here.
>> So no crashdump for further analysis.
> That's fine.. w/ the addr2line we have some lines to explore...
>
>> Interestingly the CAM errors didn't show up on the terminal as other
>> times, the kernel just panicked straight away.
> Hmm.. that is odd.. someone who knows the SD card layer should look
> at this part...  It could be that the SD card driver doesn't handle
> dumping (there is this global flag that gets set) properly and the driver
> needs to behave differently when it's set...

I also need to grab a new SD card, just to make sure it's really not the 
card.

>
>> But I've got the addr2line output, even though I'm not sure it makes any
>> difference:
>>
>> addr2line -f -e /mnt/kernel.debug 0xc0d53828
>>
>> ffs_truncate
>> /usr/devel/dreamplug/sys/ufs/ffs/ffs_inode.c:321
> can you give me the contents of the line?  and a few lines of context
> around it?   In HEAD's source, this is DOINGASYNC, and there is no call
> to memset, nor a variable assignment that would result in memset being
> called...

Same here.. The file hasn't been changed in a while (Fri, 31 May 2013):

                 ip->i_size = length;
                 DIP_SET(ip, i_size, length);
                 if (bp->b_bufsize == fs->fs_bsize)
                         bp->b_flags |= B_CLUSTEROK;
                 if (flags & IO_SYNC)
                         bwrite(bp);
321:        else if (DOINGASYNC(vp))
                         bdwrite(bp);
                 else
                         bawrite(bp);
                 ip->i_flag |= IN_CHANGE | IN_UPDATE;
                 return (ffs_update(vp, !DOINGASYNC(vp)));

No idea what's going on.

>>>> The sad thing is, that with fsck broken for the dreamplug, I have to
>>>> re-format the disk, reinstall everything and recreate the config files
>>>> which I didn't manage to copy to a safe place beforehand :-(
>> Didn't get around yet on checking whether the fsck issue persists if
>> everything is compiled with gcc. Will take a bit, as I'm going to be on
>> holidays for the next one and a half weeks. Fact is though, fsck is
>> broken on the Dreamplug (ARMv5TE), at least when run on EVERY device
>> that is attached via USB and if compiled with CLANG.
>>>> Before I do that I'll leave the system in debugging mode for a few days,
>>>> in case someone can help and needs some more information.
>> Currently I've run fsck on all the disks/cards that had a working world
>> for the dreamplug on it, so they're all gone. Can't do eny debugging
>> atm. I'll let you know hoe the gcc build goes once I'm back from
>> holidays though.
> Hmmm. this is also worth investigating as it could be that clang is
> producing bad code somewhere...

I'm trying to get kernel and world compiled with gcc atm. It fails 
though, as somehow the armv7 code is currently broken... and the 
toolchain build is not happy.

In file included from 
/usr/devel/dreamplug/sys/arm/arm/cpufunc_asm_armv7.S:36:
./machine/sysreg.h:97:5: error: "__ARM_ARCH" is not defined

(followed by a few more of those)

My build command:

  make TARGET=arm TARGET_ARCH=arm DESTDIR=/usr/devel/dreamplug-dist 
KERNCONF=DREAMPLUG-1001 -DWITHOUT_SHAREDOCS -DWITHOUT_EXAMPLES 
-DWITHOUT_HTML -DWITHOUT_MAN -DWITHOUT_GAMES toolchain buildworld kernel 
installworld