BigDisk project: du(1) 64bit clean.

[Peter Jeremy]

On Tue, 2005-Jan-04 15:47:48 -0800, Julian Elischer wrote:
>One thing that needs to be done is an 2ndary storage fsck.
>that doesn't try put everything in RAM.

Agreed but VM is different to physical RAM.

>Basically this will mean extracting all the metadata from filesystems into
>files and running sort operations of various kinds on them
>to order the data in ways that allows consistencies to be checked.

How do you determine where the disk space comes from?  You can't
safely use the filesystem being checked because you can't safely write
to it until the fsck has completed.  _Any_ filesystem picked
automatically may not have sufficient free space to do an fsck.
(Older Unices prompted for a temporary directory for work files).

Overall, the most likely location for free space is swap.  I'd therefore
suggest that rather than moving back to using temporary files, we look
at how to better use swap space:
- Look into reducing VM requirements (600MB per TB seems high)
- Tweak data structures to maximise locality of reference (traversing
  linked lists is extremely slow when your list won't fit into RAM)
- Give fsck the ability to split into multiple processes to avoid process
  size limits - usr socketpair()s to allow a "master" process to delegate
  work ala dump(8).

The downside of using swap is the possibility of over-writing a crash dump.

-- 
Peter Jeremy