Returning User With Filesystem/Memory Tuning Questions

[Giorgos Keramidas]

On Wed, 3 Dec 2008 22:44:29 -0600 (CST), Kevin Monceaux <Kevin at RawFedDogs.net> wrote:
> On Thu, 4 Dec 2008, Giorgos Keramidas wrote:
>
>> For what it's worth, I've been reading newsgroups with more than 5000
>> messages in Gnus, a newsreader that runs inside GNU Emacs, and its
>> memory usage has *never* reached 512 MB,
>
> I doubt I'd have a problem with newsgroups that small with either tin
> or pan.  How well does Gnus handle groups with 1,000,000 to 2,000,000+
> messages?  My ISP dropped it's NNTP service a while back and I ended
> up signing up with GigaNews.  They have 240 day retention for binary
> groups and 1,990 day retention for text groups.  So, many of the group
> archives on their servers are huge.  Granted I don't need to retrieve
> all the headers for a particular group, but it's not unusual for me to
> be browsing a group with a header count in the five to six digit
> range.

Ouch.  That's a really big group size.  I have only opened groups with
Gnus that contain upwards of 120,000 messages (my full email archive
since around 2001).  When I try to open the particular message archive,
Gnus takes about 7-8 seconds to generate the full group summary, and the
memory usage of Emacs jumps to about 150 MB.

I haven't tried opening a *remote* group with more than a million
messages, but I suspect this is going to take a while :-)

> I finally gave up on tin a while back and switched to pan.  It seems
> to be less of a memory hog with larger groups than tin was.  I prefer
> TUI based programs over GUI based programs, but I think pan is worth
> putting up with the GUI interface.

Heh, I know what you mean.  Most of the time the only GUI programs that
run around here are Firefox for Javascript-heavy web pages and Evince or
XPDF for reading PDF documents.

> Well, I guess I'll probably need at least a few if I go with ZFS this
> time around.

It may be worth running an AMD64 version of FreeBSD if you plan to use
ZFS for heavy-duty tasks.  I am currently without a network connection,
but I will seek the FreeBSD Wiki page about ZFS tuning, and will post a
followup to this post.

I installed FreeBSD/amd64 on my laptop to test ZFS a few weeks ago, and
with the tuning mentioned in the Wiki it was rock-stable for every day
work.  The test installations I used were two:

  * One with GELI encryption for the full disk and ZFS running on top of
    the encrypted device.

  * One with ZFS only, using most of the disk (and a small 2 GB /boot
    partition for bootstrapping into a ZFS-root filesystem).

Rink Spinger has posted an excellent blog post about running with a
small /boot partition and ZFS on the root filesystem (and everything
else).  His blog post is what I used to guide me through the initial
steps of the GELI+ZFS root setup.

Look at http://planet.freebsdish.org/ for Rink's blog post, or if it has
fallen off the top of the blog planet queue, Google for it.  It may be a
very handy guide to have around in printed dead-tree format when you
attempt to install and tune ZFS.

Cheers, and have fun with FreeBSD,

Giorgos