Thinking of using ZFS/FBSD for a backup system

[CZUCZY Gergely]

Thank you for your thought, Jeremy. Yes, I was trying to refer to these things.

Putting more memory into the system then ZFS's need doesn't prove anything.
There's no _proven_ _garantee_ that it won't panic, it just makes it more
difficult (lowers the possibilty) to panic the system.

As you said tuning ARC to a lower and kmem_size{,_max} to a higher value makes
it less likely to panic, but this won't garantee anything, just makes panicing
bigger.

"Stable ZFS" would mean, that these circumstances are cleared, and there's a
proven garantee (either mathematically) that it's _unable_ to panic due to this
memory allocation issue.

It's still there, but with a bigger amount of memory it's less likely to happen.

And I haven't tried prefetch_disable back then. So i've got no experiences on
the effects of prefetch_disable.

On Tue, 8 Jul 2008 22:56:45 -0700
Jeremy Chadwick <koitsu at FreeBSD.org> wrote:

> On Wed, Jul 09, 2008 at 07:44:20AM +0200, CZUCZY Gergely wrote:
> > On Wed, 09 Jul 2008 01:14:49 +0200
> > Kris Kennaway <kris at FreeBSD.org> wrote:
> > 
> > > CZUCZY Gergely wrote:
> > > I don't know; empirically my setup is an upper bound.  How large was
> > > "as large as it was allowed" for you?
> > Well, we cannot buy "upper bounds" all over, just because some
> > developer is unable to figure out things. I think you can't expect
> > FreeBSD users to spend as much money as possible, just because the devs
> > can't tell how much is enough...
> > It seems more like a twilight zone then a stable feature now ;)
> > 
> > It was exactly as much as an amd64 installation would allow with 2GB of
> > physical memory. We've dismissed the setup around february, and I don't
> > have the configs anymore. It was an amd64 setup with 2GB of physical
> > memory.
> 
> The bottom line here is that i386 and amd64 both have a kmem_size limit
> of 2GB.  You can throw 32GB of RAM into an amd64 box, but FreeBSD will
> only utilise up to 2GB of that for kmem.  That is purely a FreeBSD
> limitation, and is being dealt with in HEAD by Alan Cox.  I believe he
> has a patch, or it may have been committed -- I don't follow HEAD.  I
> can point people to a mailing list URL, if needed.
> 
> This is one of the limitations Gergely is referring to.
> 
> Since ZFS is incredibly memory-hungry, you're forced to tune ZFS to try
> and get it to "play nice" with that 2GB limit on STABLE/RELEASE systems.
> You also need to keep in mind that you can't just set kmem_size and
> kmem_size_max to 2048M, because the kernel needs memory for other
> things.
> 
> The tuning parameters I use on my 2GB amd64 and 4GB amd64 boxes are:
> 
> vm.kmem_size="1536M"
> vm.kmem_size_max="1536M"
> vfs.zfs.arc_min="16M"
> vfs.zfs.arc_max="64M"
> 
> If you set kmem_size and kmem_size_max any higher than that, the machine
> will panic on boot, stating (indirectly) that there isn't enough memory
> available for the kernel to allocate for other things.
> 
> Until I added the arc_min and arc_max setting, I could occasionally
> panic the machines under very heavy load (heavy zpool I/O), caused by
> kmem exhaustion.  Since adding the arc_* tunings, I've tried very hard
> to crash the machines, and I cannot.
> 
> But there's absolutely no guarantee those tuning parameters above will
> ensure FreeBSD won't panic due to kmem exhaustion.  I believe this is
> the point Gergely is making about the "stability" of the whole thing.
> 
> Now, with regards to prefetch_disable, folks can disable that if they
> want.  I disable it on my above systems because for what they do, the
> overall performance appears better with prefetching disabled.
> 
> I hope this helps shed some light here...
> 


-- 
Üdvölettel,

Czuczy Gergely
Harmless Digital Bt
mailto: gergely.czuczy at harmless.hu
Tel: +36-30-9702963
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20080709/469167cb/signature.pgp