math/py-numpy vs. math/atlas-devel

Maho NAKATA chat95 at mac.com
Tue Nov 10 07:13:21 UTC 2009


Hi all,

I'm willing to apply patches to atlas ports if available.
I talked with Goto Kazushige (author of GotoBLAS), he told me
that L2 cache handling on FreeBSD is very bad. It may be a reason
why ATLAS build is so fragile. On Linux machines, it takes only 20 min or so.

Thanks for good discussions!

Best,
 Nakata Maho

From: Scott Bennett <bennett at cs.niu.edu>
Subject: Re: math/py-numpy vs. math/atlas-devel
Date: Mon, 09 Nov 2009 15:07:55 -0600 (CST)

>      On Mon, 9 Nov 2009 20:26:15 +0000 "b. f." <bf1783 at googlemail.com>
> wrote:
>>On 11/9/09, Scott Bennett <bennett at cs.niu.edu> wrote:
>>>      On Sun, 08 Nov 2009 23:59:29 -0800 Doug Barton <dougb at FreeBSD.org>
>>> wrote:
>>
>>...
>>
>>>      Anyway, the math/py-numpy port now proceeds to build without bothering
>>> with math/atlas.  It quickly goes astray when it doesn't recognize that any
>>> of gfortran4[345] has been installed--I guess there no longer is a FORTRAN
>>> compiler included in the base system--and instead tries to use lang/g95,
>>> which has also been installed.  Of course, this won't work because the ATLAS
>>> library needs to have been compiled with the same compiler as the programs
>>> that use it.
>>>
>>> ===>  Configuring for py26-numpy-1.3.0_2,1
>>> Running from numpy source directory.
>>>  [39mF2PY Version 2 [0m
>>>  [39mblas_opt_info: [0m
>>>  [39mblas_mkl_info: [0m
>>>  [39m  libraries mkl,vml,guide not found in /usr/lib [0m
>>>  [39m  libraries mkl,vml,guide not found in /usr/local/lib [0m
>>>  [39m  libraries mkl,vml,guide not found in
>>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../ [0m
>>>  [39m  NOT AVAILABLE [0m
>>>  [39m [0m
>>>  [39matlas_blas_threads_info: [0m
>>>  [39mSetting PTATLAS=ATLAS [0m
>>>  [39mSetting PTATLAS=ATLAS [0m
>>>  [39mSetting PTATLAS=ATLAS [0m
>>>  [39m  FOUND: [0m
>>>  [39m    libraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r'] [0m
>>>  [39m    library_dirs = ['/usr/local/lib'] [0m
>>>  [39m    language = c [0m
>>>  [39m    include_dirs = ['/usr/local/include'] [0m
>>>  [39m [0m
>>> /usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/command/config.py:361:
>>> DeprecationWarning:
>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>> Usage of get_output is deprecated: please do not
>>> use it anymore, and avoid configuration checks
>>> involving running executable on the target machine.
>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>   DeprecationWarning)
>>>  [39mcustomize GnuFCompiler [0m
>>>  [32mFound executable /usr/local/bin/gfortran44 [0m
>>>  [31mgnu: no Fortran 90 compiler found [0m
>>>  [31mgnu: no Fortran 90 compiler found [0m
>>>  [39mcustomize Gnu95FCompiler [0m
>>>  [39mcustomize Gnu95FCompiler [0m
>>>  [39mcustomize Gnu95FCompiler using config [0m
>>> compiling '_configtest.c':
>>>
>>> /* This file is generated from numpy/distutils/system_info.py */
>>> void ATL_buildinfo(void);
>>> int main(void) {
>>>   ATL_buildinfo();
>>>   return 0;
>>> }
>>>  [39mC compiler: gcc44 -DNDEBUG -O2 -fno-strict-aliasing -pipe
>>> -march=prescott -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x100000 -O2
>>> -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44
>>> -fPIC
>>>  [0m
>>>  [39mcompile options: '-c' [0m
>>>  [39mgcc44: _configtest.c [0m
>>>  [39mgcc44 _configtest.o -L/usr/local/lib -lalapack_r -lf77blas_r -lcblas_r
>>> -latlas_r -o _configtest [0m
>>> /usr/bin/ld: _configtest: hidden symbol `__powisf2' in
>>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o)
>>> is referenced by DSO
>>> collect2: ld returned 1 exit status
>>> /usr/bin/ld: _configtest: hidden symbol `__powisf2' in
>>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o)
>>> is referenced by DSO
>>> collect2: ld returned 1 exit status
>>>  [39mfailure. [0m
>>>  [39mremoving: _configtest.c _configtest.o [0m
>>>  [39mStatus: 255 [0m
>>>  [39mOutput:  [0m
>>>  [39m  FOUND: [0m
>>>  [39m    libraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r'] [0m
>>>  [39m    library_dirs = ['/usr/local/lib'] [0m
>>>  [39m    language = c [0m
>>>  [39m    define_macros = [('NO_ATLAS_INFO', 2)] [0m
>>>  [39m    include_dirs = ['/usr/local/include'] [0m
>>>  [39m [0m
>>>  [39mlapack_opt_info: [0m
>>>  [39mlapack_mkl_info: [0m
>>>  [39mmkl_info: [0m
>>>  [39m  libraries mkl,vml,guide not found in /usr/lib [0m
>>>  [39m  libraries mkl,vml,guide not found in /usr/local/lib [0m
>>>  [39m  libraries mkl,vml,guide not found in
>>> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../ [0m
>>>  [39m  NOT AVAILABLE [0m
>>>  [39m [0m
>>>  [39m  NOT AVAILABLE [0m
>>>  [39m [0m
>>>  [39matlas_threads_info: [0m
>>>  [39mSetting PTATLAS=ATLAS [0m
>>>  [39m  libraries lapack_atlas not found in /usr/local/lib [0m
>>>  [39mnumpy.distutils.system_info.atlas_threads_info [0m
>>>  [39mSetting PTATLAS=ATLAS [0m
>>> /usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/system_info.py:999:
>>> UserWarning:
>>> *********************************************************************
>>>     Lapack library (from ATLAS) is probably incomplete:
>>>       size of /usr/local/lib/libalapack_r.so is 3832k (expected >4000k)
>>>
>>>     Follow the instructions in the KNOWN PROBLEMS section of the file
>>>     numpy/INSTALL.txt.
>>> *********************************************************************
>>>
>>>   warnings.warn(message)
>>>
>>>
>>> The above sequence gets more or less repeated several more times, and
>>> after much compiling, running of python26, and other activities, the
>>> process runs aground as follows.
>>>
>>>
>>>  [39mcreating build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/fft [0m
>>>  [39mcompile options: '-Inumpy/core/include
>>> -Ibuild/src.freebsd-7.2-STABLE-i386-2.6/numpy/core/include/numpy
>>> -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.6 -c' [0m
>>>  [39mgcc44: numpy/fft/fftpack_litemodule.c [0m
>>>  [39mgcc44: numpy/fft/fftpack.c [0m
>>>  [39mcc -shared -pthread -O2 -fno-strict-aliasing -pipe -march=prescott
>>> -Wl,-rpath=/usr/local/lib/gcc44
>>> build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/fft/fftpack_litemodule.o
>>> build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/fft/fftpack.o
>>> -Lbuild/temp.freebsd-7.2-STABLE-i386-2.6 -o
>>> build/lib.freebsd-7.2-STABLE-i386-2.6/numpy/fft/fftpack_lite.so [0m
>>>  [39mbuilding 'numpy.linalg.lapack_lite' extension [0m
>>>  [39mcompiling C sources [0m
>>>  [39mC compiler: gcc44 -DNDEBUG -O2 -fno-strict-aliasing -pipe
>>> -march=prescott -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x100000 -O2
>>> -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44
>>> -fPIC
>>>  [0m
>>>  [39mcreating build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg [0m
>>>  [39mcompile options: '-DATLAS_INFO="\"3.9.11\"" -I/usr/local/include
>>> -Inumpy/core/include
>>> -Ibuild/src.freebsd-7.2-STABLE-i386-2.6/numpy/core/include/numpy
>>> -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.6 -c' [0m
>>>  [39mgcc44: numpy/linalg/lapack_litemodule.c [0m
>>>  [39mgcc44: numpy/linalg/python_xerbla.c [0m
>>>  [39mcc -shared -pthread -O2 -fno-strict-aliasing -pipe -march=prescott
>>> -Wl,-rpath=/usr/local/lib/gcc44
>>> build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/lapack_litemodule.o
>>> build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/python_xerbla.o
>>> -L/usr/local/lib -Lbuild/temp.freebsd-7.2-STABLE-i386-2.6 -lalapack_r
>>> -lalapack_r -lf77blas_r -lcblas_r -latlas_r -lgfortran -lm -lpthread -o
>>> build/lib.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/lapack_lite.so [0m
>>> /usr/bin/ld: cannot find -lgfortran
>>> /usr/bin/ld: cannot find -lgfortran
>>> error: Command "cc -shared -pthread -O2 -fno-strict-aliasing -pipe
>>> -march=prescott -Wl,-rpath=/usr/local/lib/gcc44
>>> build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/lapack_litemodule.o
>>> build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/python_xerbla.o
>>> -L/usr/local/lib -Lbuild/temp.freebsd-7.2-STABLE-i386-2.6 -lalapack_r
>>> -lalapack_r -lf77blas_r -lcblas_r -latlas_r -lgfortran -lm -lpthread -o
>>> build/lib.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/lapack_lite.so" failed
>>> with exit status 1
>>> *** Error code 1
>>>
>>> Stop in /usr/ports/math/py-numpy.
>>>
>>> ===>>> make failed for math/py-numpy
>>> ===>>> Aborting update
>>>
>>>      I looked in /usr/local/lib and didn't spot libraries by name for
>>> gfortran[345], so I assume they have either been renamed or included in
>>> with the corresponding gcc libraries.  What is the best way to proceed?
>>> Will simply deleting the instances of "-lgfortran" do the job?  Or do I
>>> need to specify the library explicitly?
>>>
>>
>>The port is broken WITH_ATLAS=yes on your system:  it reverts to using
>>the base system C compiler, cc, when it should be using gcc44.  You
>>trimmed off some relevant parts, so it is difficult to tell at a
>>glance whether this is because something is wrong with your system, or
>>with the port itself,  although the port has a history of problems
>>with this non-default option.  If I have some time later, I may play
>>with it,  but at the moment, I don't have atlas installed.  You should
> 
>      Okay, but be advised that the ATLAS library takes around six hours
> to build on a 3.4 GHz Prescott with little else active on the machine
> at the time.  (ATLAS should be built on a very quiet machine, so that
> its timing tests will be fairly accurate.  The build is likely to die
> if several runs of a timing test yield a variance greater than what the
> build procedure wants.)
> 
>>send a full build transcript to me and the math/py-numpy maintainer
>>via private email.
> 
>      Will do, but probably not till tonight sometime.
>>
>>The libraries in question should be in /usr/local/lib/gcc44.  When
>>searching for registered shared libraries, it is easier to do
>>something like:
>>
>>ldconfig -vr | grep -ie fortran
> 
>         1046:-lgfortran.3 => /usr/local/lib/gcc43/libgfortran.so.3
>         1056:-lgfortran.3 => /usr/local/lib/gcc44/libgfortran.so.3
>         1068:-lgfortran.3 => /usr/local/lib/gcc45/libgfortran.so.3
>>
>>and for static libraries or unregistered shared libraries, locate(1)
>>and find(1) are your friends.
>>
>      Well, locate(1) *ought* to be, you're right.  However, although it
> worked fine on FreeBSD 5, I've never succeeded in getting FreeBSD 6 or 7
> to build the locate database properly.  It would be nice, but I've already
> spent too many hours on too many occasions trying to track down the trouble,
> so I just use find(1).
>      find(1) turns up the following.
> 
> /usr/local/lib/gcc43/gcc/i386-portbld-freebsd7.2/4.3.5/libgfortranbegin.la
> /usr/local/lib/gcc43/gcc/i386-portbld-freebsd7.2/4.3.5/libgfortranbegin.a
> /usr/local/lib/gcc43/libgfortran.so.3
> /usr/local/lib/gcc43/libgfortran.so
> /usr/local/lib/gcc43/libgfortran.a
> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgfortranbegin.la
> /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgfortranbegin.a
> /usr/local/lib/gcc44/libgfortran.so.3
> /usr/local/lib/gcc44/libgfortran.so
> /usr/local/lib/gcc44/libgfortran.a
> /usr/local/lib/gcc45/gcc/i386-portbld-freebsd7.2/4.5.0/libgfortranbegin.la
> /usr/local/lib/gcc45/gcc/i386-portbld-freebsd7.2/4.5.0/libgfortranbegin.a
> /usr/local/lib/gcc45/libgfortran.so.3
> /usr/local/lib/gcc45/libgfortran.so
> /usr/local/lib/gcc45/libgfortran.a
> 
>      I'll get the rest of the stuff off to you later tonight.  Thank you
> very much for looking at all of this.  I wonder if I'm the only person
> trying to install science/gnudatalanguage.  There have been enough
> obstacles to getting that done that I have to wonder how the maintainer
> managed to test it.
> 
> 
>                                   Scott Bennett, Comm. ASMELG, CFIAG
> **********************************************************************
> * Internet:       bennett at cs.niu.edu                              *
> *--------------------------------------------------------------------*
> * "A well regulated and disciplined militia, is at all times a good  *
> * objection to the introduction of that bane of all free governments *
> * -- a standing army."                                               *
> *    -- Gov. John Hancock, New York Journal, 28 January 1790         *
> **********************************************************************
> 


More information about the freebsd-questions mailing list