this 48-core box...
schut at sarvision.nl
Fri Sep 20 09:14:45 UTC 2013
On Thu, 19 Sep 2013 12:05:14 -0700 (PDT)
Dennis Glatting <dg at pki2.com> wrote:
> On Thu, 19 Sep 2013, Vincent Schut wrote:
> > On Tue, 17 Sep 2013 12:08:43 -0500
> > Michael Chen <michael at foxbatcapital.com> wrote:
> >> I'm considering bidding on this 48-core box:
> >> http://www.ebay.com/itm/Supermicro-A-Server-1042G-TF-1U-H8QG6-4-CPUS-48-cores-2-2Ghz-128GB-RAM-/151119828428?pt=COMP_EN_Servers&hash=item232f7195cc
> >> Does anyone have experience with it and can I use all the cores?
> >> Thanks!
> >> _______________________________________________
> >> freebsd-questions at freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> >> To unsubscribe, send any mail to
> >> "freebsd-questions-unsubscribe at freebsd.org"
> > I recently bought one like that (48 cores but 'only' 96 Gb ram). It
> > was meant to play a double role as both zfs file server and data
> > processing server (we do lots of satellite image processing),
> > running FreeBSD 9.1. It connects with a SAN and we'll use it to
> > process about 36TB of satellite data in the next months. (In a
> > couple of weeks we will probably have budget to split those roles,
> > and buy a dedicated file server.) After several weeks of tweaking
> > and testing, I can say that:
> > - the zfs/file server part runs without problems
> > - the satellite data processing had problems scaling to all 48
> > cores, I got max performance when running about 18 processes in
> > parallel, scaling up more would lower the overall performance.
> > However, this (sorry guys) appeared to be a FreeBSD problem, and
> > not a hardware problem. As a test I switched to linux with ZoL (ZFS
> > on Linux), and, though zfs performance is less compared to freebsd,
> > data processing is much much better, like a factor 12 or so.
> I've noticed this same scaling problem on 32+ core servers but
> haven't had a chance to look into the detail. From the performance
> graphs I am confused whether my problems are processing problems or a
> data I/O problem.
I have done some (light) investigation as I did need the processing
power. In my case the bottleneck was definitely not data I/O. Bonnie+
rates from and to the dataset were as expected, top and atop and other
utils did not show any stress on the I/O system, and the algorithm which
did not scale should not be IO bound, rather cpu or memory (or
both). I've heard/read rumors (when I was investigating the extreme
long compile time of openblas on freebsd compared to linux) about bsd
being less well optimized in e.g. using the processor's L2 cache.
Things like this can play an important role in the processing we do
(many numerical calculations on lots of data in memory). Most of the
calculations were done by quite optimized software for numerical
processing (numpy/scipy using openblas (yes I did make sure openblas
used only 1 thread when scaling up)). The fact that the problems
disappeared when running the same under linux also point in the
cpu/memory direction rather that I/O, as the ZFS on Linux performance
is still behind that of ZFS on BSD.
> > Conclusion: the hardware is alright, however when needed to do lots
> > of heavy calculations on terabytes of data, the combination with
> > FreeBSD appears not ideal.
> > Of course it is you get what you pay for. Decent, OK working
> > hardware, but none of the special handy-dandy features expensive
> > brands will give you. If you don't need them, in my experience it
> > is decent hardware for a good price.
> > regards,
> > Vincent.
> > _______________________________________________
> > freebsd-questions at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to
> > "freebsd-questions-unsubscribe at freebsd.org"
> freebsd-questions at freebsd.org mailing list
> To unsubscribe, send any mail to
> "freebsd-questions-unsubscribe at freebsd.org"
More information about the freebsd-questions