FFS writes to read-only mount

[David Cecil]

Hi guys,

> On Thu, Mar 15, 2007 at 11:00:31AM +0200, Kostik Belousov wrote:
>   
>> On Thu, Mar 15, 2007 at 04:56:39PM +1000, David Cecil wrote:
>>     
>>> Hi Pawel,
>>>
>>> here is what I've found.  Note that this problem doesn't occur 
>>> frequently, but does happen often enough to be a problem.  I am still 
>>> debugging the live system.
>>>
>>> I put breakpoints in g_print_bio and g_io_request.  I then continued 
>>> until the breakpoint at g_print_bio breakpoint was hit, but each time 
>>> g_io_request was hit, I printed the backtrace.  Then, at g_print_bio I 
>>> could identify the struct bio * and look for the corresponding 
>>> g_io_request trace with a matching struct bio *.
>>>
>>> That trace looks like this:
>>> db> bt
>>> Tracing pid 47 tid 100050 td 0x860d0640
>>> g_io_request(864258c4,860ec440,a0020024,98b92188,a5281bf0,...) at 
>>> g_io_request+0x107
>>> g_vfs_strategy(86104ce4,98b92188,a0020024,98b92188,86104c30,...) at 
>>> g_vfs_strategy+0x63
>>> ffs_geom_strategy(86104ce4,98b92188) at ffs_geom_strategy+0x129
>>> bufwrite(98b92188,4000,98b92188,a5281c54,805b20c1,...) at bufwrite+0x146
>>> ffs_bufwrite(98b92188) at ffs_bufwrite+0x282
>>> vfs_bio_awrite(98b92188) at vfs_bio_awrite+0x221
>>> vop_stdfsync(a5281cbc,a5281cbc,a5281c98,806eff3e,a5281cbc,...) at 
>>> vop_stdfsync+0x121
>>>       
>> It would be helpful, at this frame, to print the actual vnode, as well as
>> the dirty buffer header together with dump of buffer data. [...]
>>     
>
> This comes from devfs_fsync(), so I suspect it's devvp, and we are
> synchronizing entire file system here.
>   

I think my interpretation corresponds with yours.  The loop in 
vop_stdfsync is syncing the dirty buffers associated with the vnode that 
maps to /dev/mirror/gm0s1a

>> [...] It seems that
>> the easiest way to do that would be to obtain coredump and then use kgdb.
>>     
>
> On successfull remount to read-only we could walk (under DIAGNOSTIC) the
> list of vnodes and assert there are no dirty buffers, but I don't think
> this is the case...
>   

I am still debugging the system in ddb, so I assume I could do that now.

> The thing that worries me is that it doesn't happen from what I
> understand just after remount to read-only. Syncer wakeups every 30
> seconds by default, right? 

Yes.

> So it should generates errors on first time
> he synchronizes file system. Which bascially means, that something tries
> to write to read-only file system later. Will be good to assert it
> somehow earlier. One thing that comes to my mind is to assert:
>
> 	KASSERT(((struct g_consumer *)buf->b_bufobj.bo_private)->acw > 0);
>
> in each b*write() function. This is probably not the only way to write
> to a file system...
>   

Okay, that's something for me to bear in mind.  I as trying to get as 
much from the ddb session as I can, before rebooting and trying kernel 
mods.  This problem occurs frequently enough to be annoying, but not so 
frequently that it's easy to reproduce.

> I still don't feel too familiar with the code, so not sure if my
> comments are usable:) how-to-reproduce procedure would be the best of
> course... David, do you use something which can be considered
> non-standard, like snapshots?
>   

It may be that snapshots are used, but not explicitly.  The startup 
scripts attempt to run fsck in the background, which would normally 
require a snapshot, but shouldn't for a read-only mount, right?

I can't think of anything else that's different to standard in this area 
(FFS/GEOM)...

Regards,
Dave

-- 
Software Engineer
Secure and Mobile Connectivity
Nokia Enterprise Solutions
+61 7 5553 8307 (office)
+61 412 728 222 (cell)