Painted into a corner with i386 system and big FS ...

[Jeremy Chadwick]

On Thu, Dec 29, 2011 at 02:19:49PM -0800, UFS User wrote:
> > It appears that this is impossible to fsck - I have nudged up maxdsiz all the way to:
> > kern.maxdsiz="3072000000"
> > 
> > and it's not enough. ?Any larger values cause the system to become unbootable.
> > So, my plan is to boot 8.x amd64 FIXIT, and run fsck from the fixit environment.
> > The issues I foresee are:
> > 
> > 1. Do I need to specify a maxdsiz in the 8.x fixit environment, or does this all just autoscale
> > these days ?
> 
> > It does not autoscale.? You will need to set it manually at the loader
> > prompt by entering 'set kern.maxdsiz="3072000000"' then 'boot'.
> 
> Ok, great. ?And it's the same old process:
> 
> Type '?' for a list of commands, 'help' for more detailed help.
> OK set?kern.maxdsiz="3572000000"
> OK boot
> 
> Yes ?

Correct.

> > 2. There is only 4 GB of physical in the system ... if I could grow to 3.072 GB process size in
> > i386, then all else being equal, I should be able to grow to 3.5 in 8.x FIXIT, right ?
> 
> >The 3.5GB value you came up with is equally as amusing.? How much you
> >actually get greatly varies on hardware details and so on.? Your
> >motherboard manufacturer should be able to tell you how much you'll get,
> >as long as you provide them a full list of any PCI/PCI-X/PCIe devices
> >you have installed in slots, and any system features you have enabled
> >(AHCI, on-board SCSI, etc.).
> 
> Oh, sorry for the confusion - the installed OS is freebsd i386, BUT the underlying system is in fact a 64bit system.
> It is the physical inaccessibility, and other logistical issues that prevent us from just upgrading the entire OS, etc.

In this specific case it doesn't matter if the actual hardware supports
64-bit or not.  The OS is i386, which means you're limited to 32-bit
(4GB) addressing space.  As such, devices which want memory-mapped
regions are taking up some addressing space which could be used for
physical RAM, but this would only work with a 64-bit OS.  You know this
already.  My point is that the underlying hardware supporting 64-bit
doesn't matter.

Using i386 with PAE is your only other option, and I don't advise going
that route.  There are device drivers and other "catches" with PAE;
amd64 is supported, while I'm not so sure PAE is.

> So I will boot amd64 FreeBSD 8.x FIXIT, and will be running in a proper amd64 architecture, but the limitation is I
> only have 4 GB physical ram.

I understand.

> So I am hoping that, all else being equal, if i386 can only see (from dmesg):
> 
> real memory ?= 3489071104 (3327 MB)
> avail memory = 3414245376 (3256 MB)
> 
> then an amd64 system would see (roughly) 500 megabytes more memory than that, and the 3.072 GB process size I get on i386
> will then be (roughly) 3.572 on amd64.
> 
> A reasonable guess ?

I'll repeat myself: not exactly.  This "500MByte" value you came up with
is purely random.  The amount that will become available will be decided
based upon what sorts of devices are on your system, whether or not they
actually need physical RAM as part of their addressing space (or if they
just want addressing space for internal use), etc..  There are too many
factors to consider.

Read the following manual for a Supermicro X7SBA motherboard,
specifically section 2-4 (see chart labelled "Possible System Memory
Allocation & Availability"):

http://www.supermicro.com/manuals/motherboard/3210/MNL-0976.pdf

Some of what's shown there applies to i386 (32-bit) only, but ANY of
those devices or features can require actual physical RAM (for example,
a video card that has no on-board RAM and thus uses system RAM).  AHCI,
option ROMs, and other things all "take up" physical RAM regardless if
they're on amd64 or i386.  Understand?  So, the "500MByte" value you're
assuming is in no way accurate.

The easiest way to find out what you'll get?  Boot amd64!  :-)

For "tuning kern.maxdsiz", keep reading.

> I hope I don't have to swap, but if I do, presumably this will only grow to 3.8 or 4.0 or 4.2 GB in process size, so that's just the last 1/8 of the job
> that will be slow...

Again, alternately, I would recommend you take the disk out and stick it
in an amd64 system that has more memory (16-32GB would be good) and has
a dedicated swap partition, and fsck from there.  It's better to be safe
than sorry.

Here's a better way of looking at it:

Do you really want to try and "wing it" with no swap, only to have the
system panic, resulting in a potentially even more corrupt filesystem
(fsck does make changes to the filesystem, so what happens if the kernel
panics in the middle of that?  :-), or would you rather fsck the
filesystem on a system where you know you're not going to encounter
memory exhaustion and risk making things worse?

If you *really* must do this on the same system, then add a physical
disk to the system which you can use temporarily for a swap partition,
boot an 8.2-STABLE amd64 memstick / fixit environment, set up the disk
for swap, then swapon it.  After that, fsck the other disk.  As I
stated, I/O will be slow if swap is used, but at least you won't risk
fsck segfaulting/dying off, or a kernel panic.

Also, you might be wondering if kern.maxdsiz would include swap use or
not; I imagine that it would.  So in your case, you may need to set the
loader variable to an extremely large value (many gigabytes) in the case
that the fsck process ultimately requires lots of memory (physical +
swap combined).  There's no real way to calculate this value ahead of
time, so I really can't tell you what to pick.  The situation you are in
isn't easily resolved.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |