Default inode number too low in FFS nowadays?

[Jeremy Chadwick]

On Wed, Nov 02, 2011 at 12:57:33PM +0100, Borja Marcos wrote:
> Today I?ve come across an issue long ago forgotten :) Running out of i-nodes.

The last time I dealt with this was about 3 months ago, on a Netapp
filer; hardly forgotten.  :-)

> The condition was triggered on a 8 GB /usr filesystem which includes the ports tree, on which I have compiled several ports and their dependencies.  I know, maybe not the best practices, it's a machine being used to test a couple of thingies, but I wonder how many newbies can run into such a problem. I guess many.
> 
> Filesystem     1K-blocks    Used        Avail           Capacity iused     ifree      %iused  Mounted on
> /dev/ad1s1e   8245660     5308960 2277048    70%        507099  61219   89%       /usr
> 
> Now i-nodes aren't completely full, as I have deleted /usr/obj (turns out I had made a make world as well).
> 
> Looking at the number of i-nodes per /usr subdirectory, I have noticed that, wow! /usr/ports consumes A LOT of them. 
> 
> freebsd9-borja#	find . -print | wc -l
>   405481

"find -x /usr/ports | wc -l" on my system returns 146420.

"find -x /usr | wc -l" on my system returns 264901 (and this includes a
populated /usr/obj buildworld/buildkernel tree.  /usr/obj returns
37126, just for the record).

$ df -k -i /usr
Filesystem   1024-blocks    Used    Avail Capacity iused   ifree %iused Mounted on
/dev/ada0s1f    38110116 3425114 31636194    10%  258980 4686938    5%  /usr

Your 8GB filesystem is only ~21% the capacity of mine.  So if we do some
simple math, your /usr should really be showing (assuming yours is as
"bare bones" as mine) around 26% inodes used.  Yet what you have is 89%.

You probably just have a lot of other software on /usr that "abuses"
lots of individual files.  You list off what ports you built from
source, which means you have lots of "build cruft" laying around en
masse (work/ directories).

> I know the /usr isn't that large, and I know that I have compiled a
> lot on this battered /usr filesystem, but with the increasing
> complexity of many programs and the increased number of dependences in
> ports, I would:
> 
> 1) At least double the default number of i-nodes per filesystem
> 
> 2) Suggest strongly in sysinstall to create a /usr/ports filesystem
> for the ports. But probably this won't be popular and many people will
> tend to create a single large root to rule them all....

Really?  That isn't what the default installer does (re: single
filesystem), and hasn't since...  well, as long as I can remember
(2.2.8).

If someone chooses to create a single / for all of their stuff, so be
it.  It's their choice.  This is very much a "Linux-ism"; I used to do
the same thing back when I used Linux in the early 90s and was quickly
taught why it's both useful *and* bad.

> What do you think?

Here's what I think:

I wouldn't recommend a /usr/ports filesystem.  It solves nothing.
Imagine: /usr/ports can now run out of inodes while /usr won't.  What
would this address?  It would decrease impact to applications creating
new files on /usr.  How often do you have that happening on your system?
For me it's extremely rare.  For comparison, /var gets a lot more
activity.

Adding a /usr/ports filesystem also provides zero scalability.  Meaning:
say someone decides "8GB for /usr/ports is enough" during the installer
and so on.  Then down the road, we run out of space, or inodes, or
whatever else.  Oops, what now?  Symlinks all over the place?  Yeah, not
going to happen.

We were already in this boat regarding the root filesystem size, which
for quite some time defaulted to 512MBytes, which was not enough.
Today, the default size is 1GBytes, which is enough for a kernel built
with debugging symbols and so on, but tends to reach ~55-58% used.

There's a reason /usr on FreeBSD defaults to "all the remaining space on
the disk" if you pick the defaults/auto.  Surprise.  :-)

The summarised version is:

1. You have control over this yourself: newfs(8) -i flag.  You can even
input this flag during sysinstall when building a new system.

2. When you built the system, you chose to assume /usr should be 8GBytes.
Your usage habits may have changed, software gotten larger, etc..  It
may be time for you to re-evaluate how you create your filesystems and
what sizes you choose.  You chose something that didn't scale.  :-)

3. Get in the habit of cleaning up after yourself.  Don't leave work/
directories laying around inside your ports ("make install clean" solves
this problem, but the clean phase only happens at the end of whatever
is being built, including dependencies).

4. For a limited-space (not just size but inodes!) filesystem, consider
using packages (pkg_add -r) for anything that does not require
customisation during the build/configure phase.  For example, for quite
some time now on my systems, I've always installed the package versions
of perl, python27, cmake, gettext, pcre, etc. solely because they are
"key softwares" that I hate building from source.

5. Most FreeBSD Ports folks are aware and respectful as much as possible
of limited-space environments, especially due to those on embedded
platforms.  A very hot topic/debate many years ago was whether or not
there should be a new file introduced into each ports' directory to
define some dependency bits.  Obviously it did not happen.

I'm not sure all committers are aware of this though.  For example, a
recent (new) port I submitted contained 2 files in files/ (two patches).
The committer chose to split this up into 14 files (which not only
wastes more inodes, but also makes my life as port maintainer a bigger
PITA given where the patches come from).  You get the idea I'm sure.

> I can't imagine what it might have been if I had compiled Gnome or KDE... :D

I imagine you would have experienced that your filesystem ran out of
inodes.  Oh wait, you said it did...  :-)  "What" was compiled has no
bearing on the situation; failure is failure, whether you were building
Office.org or compiling a 4-file C program.

> For the record, the ports I had compiled are:
> {snip}

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |