The ABINIT Users Mailing List ( CLOSED )

[Luigi Genovese <luigi.genovese@gmail.com>]

Have you did "make clean" before?

It seems that the previously compiled routines were still without the
second underscore.

Let me know

Luigi

On Mon, Sep 21, 2009 at 12:31 PM, Jyh-Shyong <c00jsh00@nchc.org.tw> wrote:
> Hi,
>
> thanks for the information, with flag "--assume 2underscores", libGPU.a is
> recognized now, however, new
> problem occurs:
>
>
>
> ....
>
> /opt/mvapich2/intel/bin/mpif90 -I../modules -I/opt/mvapich2/intel/include
> -I. -I. -O2 -assume 2underscores -L/opt/mvapich2/intel/lib
> -L/opt/lapack/intel -L/opt/lapack/intel -L../../libXC/src -o conv_check
> conv_check.o memory.o base.o ../libbigdft.a -L/opt/cuda/lib/ -lcudart
> -lcublas libGPU.a -lxc -llapack -lblas -lmpich -libverbs -lpthread
> conv_check.o: In function `MAIN__':
> conv_check.f90:(.text+0xea7): undefined reference to `convrot_n_per__'
> conv_check.f90:(.text+0x1ed6): undefined reference to `fill_mod_arr__'
> conv_check.f90:(.text+0x2e38): undefined reference to `ana_rot_per__'
> conv_check.f90:(.text+0x44c8): undefined reference to
> `adjust_keys_for_gpu__'
> conv_check.f90:(.text+0x5365): undefined reference to
> `adjust_keys_for_gpu__'
> conv_check.f90:(.text+0x63ae): undefined reference to `syn_rot_per__'
> memory.o: In function `memory_occupation__':
> ../profiling/memory.f90:(.text+0x289): undefined reference to `mpi_abort__'
> ../profiling/memory.f90:(.text+0x549): undefined reference to `mpi_abort__'
> ../libbigdft.a(base.o): In function `module_base_mp_herk_simple_':
> base.F90:(.text+0xd33): undefined reference to `cublas_cherk_'
> ../libbigdft.a(base.o): In function `module_base_mp_syrk_double_':
> base.F90:(.text+0xd93): undefined reference to `cublas_dsyrk_'
> ../libbigdft.a(base.o): In function `module_base_mp_syrk_simple_':
> base.F90:(.text+0xdf3): undefined reference to `cublas_ssyrk_'
> ../libbigdft.a(base.o): In function `module_base_mp_c_gemm_simple_':
> base.F90:(.text+0xf05): undefined reference to `cublas_cgemm_'
> ../libbigdft.a(base.o): In function `module_base_mp_gemm_double_':
> base.F90:(.text+0xfe5): undefined reference to `cublas_dgemm_'
> ../libbigdft.a(base.o): In function `module_base_mp_gemm_simple_':
> base.F90:(.text+0x10c5): undefined reference to `cublas_sgemm_'
> ../libbigdft.a(base.o): In function `module_base_mp_nrm2_simple_':
> base.F90:(.text+0x114e): undefined reference to `cublas_snrm2_'
> ../libbigdft.a(base.o): In function `module_base_mp_dot_simple_':
> base.F90:(.text+0x117e): undefined reference to `cublas_sdot_'
> ../libbigdft.a(base.o): In function `module_base_mp_c_axpy_simple_':
> base.F90:(.text+0x11ae): undefined reference to `cublas_caxpy_'
> ../libbigdft.a(base.o): In function `module_base_mp_axpy_simple_':
> base.F90:(.text+0x11de): undefined reference to `cublas_saxpy_'
> ../libbigdft.a(base.o): In function `module_base_mp_c_trmm_simple_':
> base.F90:(.text+0x12b5): undefined reference to `cublas_ctrmm_'
> ../libbigdft.a(base.o): In function `module_base_mp_trmm_simple_':
> base.F90:(.text+0x13d5): undefined reference to `cublas_strmm_'
> ../libbigdft.a(base.o): In function `module_base_mp_c_scal_simple_':
> base.F90:(.text+0x144e): undefined reference to `cublas_cscal_'
> ../libbigdft.a(base.o): In function `module_base_mp_scal_simple_':
> base.F90:(.text+0x147e): undefined reference to `cublas_sscal_'
> make[1]: *** [conv_check] Error 1
> make[1]: Leaving directory `/work/chem/bigdft-1.3.0/src/CUDA'
> make: *** [check-am] Error 2
>
>
> Here is the configure options:
>
> export FC=/opt/mvapich2/intel/bin/mpif90
> export FCFLAGS=""
> export LDFLAGS=" -L/opt/mvapich2/intel/lib"
> export LIBS="-lmpich -libverbs -lpthread"
> export CC=icc
> export CPPFLAGS="-I/opt/mvapich2/intel/include"
> export CXX=icpc
> export MPI_INCLUDE="-I/opt/mvapich2/intel/include"
> export MPI_LDFLAGS="-L/opt/mvapich2/intel/lib"
> export MPI_LIBS="-lmpich -lfmpich"
> export MPI_NAME="mpich"
> ./configure -prefix=/package/chem/workdir/abinit-5.8.4/tmp/plugins/bigdft
> --with-blas --with-blas-path=/opt/lapack/intel --with-lapack
> --with-lapack-path=/opt/lapack/intel --enable-mpi --enable-cuda-gpu
> --with-cuda-path=/opt/cuda --with-lib-cutils=/opt/cuda/lib
>
>
> Flags --assume 2underscores was added my editing Makefile.
>
> Thanks for your help.
>
> Jyh-Shyong Ho
>
>
>
> Luigi Genovese 提到:
>>
>> Hi,
>>
>> as Matthieu Ospici pointed out, I have just realised that I did not
>> read the output of your configure... sorry for that.
>>
>> Another option which sould be added to the fortran compiler flags for
>> supporting cuda routines is the one which adds a second underscore to
>> functions, whic in ifort is "-assume 2underscores".
>>
>> Sorry for the incomplete answer
>>
>> Luigi
>>
>>
>>
>> On Sun, Sep 20, 2009 at 8:01 PM, Luigi Genovese
>> <luigi.genovese@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> to compile BigDFT with the CUDA routines you have to add
>>> --enable-cuda-gpu
>>> in the configure script options, and optionally provide the path of
>>> the cuda libraries if it is not the default value.
>>> See configure --help for more details.
>>>
>>> However, your GPU must support double precision (NVIDIA release number
>>> 1.3) to run BigDFT on. Simple precision is not supported.
>>>
>>> In the 1.3.0 version, not so much is add for what concern the
>>> documentation for GPU. Do not hesitate to communicate us the problems
>>> you may have.
>>>
>>> Regards
>>>
>>> Luigi Genovese
>>>
>>> On Sun, Sep 20, 2009 at 10:26 AM, Jyh-Shyong Ho <c00jsh00@nchc.org.tw>
>>> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> BigDFT is one of plugin programs of abinit 5.8.4, now BigDFT can be
>>>> built with
>>>> NVIDIA CUDA support. I wonder if anyone has successfully built BigDFT
>>>> with
>>>> CUDA?
>>>>
>>>> I downloaded the latest version bigdft-1.3 and tried to install it on my
>>>> computer with CUDA 2.3, but failed. If someone can provide me some hint
>>>> where
>>>> might be wrong, it will be very appreciated.
>>>>
>>>> In directory bigdft-1.3.0/src/CUDA, I was able to build libGPU.a,
>>>> however, I
>>>> failed to built check_conv which uses libGPU.a:
>>>>
>>>> ...
>>>> chem@gch:/work/chem/bigdft-1.3.0/src/CUDA> make check
>>>> make  conv_check GPUham
>>>> make[1]: Entering directory `/work/chem/bigdft-1.3.0/src/CUDA'
>>>> ifort -I../modules  -I. -I. -O2  -L/opt/lapack/intel -L/opt/lapack/intel
>>>> -L../../libXC/src -o conv_check conv_check.o memory.o base.o
>>>>  ../libbigdft.a
>>>> -L/opt/cuda/lib/ -lcudart -lcublas libGPU.a -lxc -llapack -lblas
>>>>  -lpthread
>>>> conv_check.o: In function `MAIN__':
>>>> conv_check.f90:(.text+0x4f0): undefined reference to `set_gpu_double_'
>>>> conv_check.f90:(.text+0x1417): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x144c): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x1488): undefined reference to `gpu_send_'
>>>> conv_check.f90:(.text+0x1703): undefined reference to `gpu_receive_'
>>>> conv_check.f90:(.text+0x171a): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x1731): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x230d): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x2342): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x2377): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x23ac): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x23e8): undefined reference to `gpu_send_'
>>>> conv_check.f90:(.text+0x269e): undefined reference to `gpu_receive_'
>>>> conv_check.f90:(.text+0x26b5): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x26cc): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x26e3): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x26fa): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x2fc3): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x2ff8): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x3034): undefined reference to `gpu_send_'
>>>> conv_check.f90:(.text+0x32bc): undefined reference to `gpu_receive_'
>>>> conv_check.f90:(.text+0x32d3): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x32ea): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x441b): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x4458): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x448f): undefined reference to `gpu_send_'
>>>> conv_check.f90:(.text+0x475b): undefined reference to `gpu_receive_'
>>>> conv_check.f90:(.text+0x4772): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x4789): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x47a0): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x52aa): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x52e7): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x532c): undefined reference to `gpu_send_'
>>>> conv_check.f90:(.text+0x55df): undefined reference to `gpu_receive_'
>>>> conv_check.f90:(.text+0x55f6): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x560d): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x5624): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x6539): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x656e): undefined reference to `gpu_allocate_'
>>>> conv_check.f90:(.text+0x65aa): undefined reference to `gpu_send_'
>>>> conv_check.f90:(.text+0x6832): undefined reference to `gpu_receive_'
>>>> conv_check.f90:(.text+0x6849): undefined reference to `gpu_deallocate_'
>>>> conv_check.f90:(.text+0x6860): undefined reference to `gpu_deallocate_'
>>>> memory.o: In function `memory_occupation_':
>>>> ../profiling/memory.f90:(.text+0x289): undefined reference to
>>>> `mpi_abort_'
>>>> ../profiling/memory.f90:(.text+0x549): undefined reference to
>>>> `mpi_abort_'
>>>> base.o: In function `module_base_mp_herk_simple_':
>>>> ../modules/base.F90:(.text+0xd33): undefined reference to
>>>> `cublas_cherk_'
>>>> base.o: In function `module_base_mp_syrk_double_':
>>>> ../modules/base.F90:(.text+0xd93): undefined reference to
>>>> `cublas_dsyrk_'
>>>> base.o: In function `module_base_mp_syrk_simple_':
>>>> ../modules/base.F90:(.text+0xdf3): undefined reference to
>>>> `cublas_ssyrk_'
>>>> base.o: In function `module_base_mp_c_gemm_simple_':
>>>> ../modules/base.F90:(.text+0xf05): undefined reference to
>>>> `cublas_cgemm_'
>>>> base.o: In function `module_base_mp_gemm_double_':
>>>> ../modules/base.F90:(.text+0xfe5): undefined reference to
>>>> `cublas_dgemm_'
>>>> base.o: In function `module_base_mp_gemm_simple_':
>>>> ../modules/base.F90:(.text+0x10c5): undefined reference to
>>>> `cublas_sgemm_'
>>>> base.o: In function `module_base_mp_nrm2_simple_':
>>>> ../modules/base.F90:(.text+0x114e): undefined reference to
>>>> `cublas_snrm2_'
>>>> base.o: In function `module_base_mp_dot_simple_':
>>>> ../modules/base.F90:(.text+0x117e): undefined reference to
>>>> `cublas_sdot_'
>>>> base.o: In function `module_base_mp_c_axpy_simple_':
>>>> ../modules/base.F90:(.text+0x11ae): undefined reference to
>>>> `cublas_caxpy_'
>>>> base.o: In function `module_base_mp_axpy_simple_':
>>>> ../modules/base.F90:(.text+0x11de): undefined reference to
>>>> `cublas_saxpy_'
>>>> base.o: In function `module_base_mp_c_trmm_simple_':
>>>> ../modules/base.F90:(.text+0x12b5): undefined reference to
>>>> `cublas_ctrmm_'
>>>> base.o: In function `module_base_mp_trmm_simple_':
>>>> ../modules/base.F90:(.text+0x13d5): undefined reference to
>>>> `cublas_strmm_'
>>>> base.o: In function `module_base_mp_c_scal_simple_':
>>>> ../modules/base.F90:(.text+0x144e): undefined reference to
>>>> `cublas_cscal_'
>>>> base.o: In function `module_base_mp_scal_simple_':
>>>> ../modules/base.F90:(.text+0x147e): undefined reference to
>>>> `cublas_sscal_'
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `gpu_precond_helper_stream_':
>>>> convolut_common_interface_cuda.f90:(.text+0x11b): undefined reference to
>>>> `gpuprecond_stream_'
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `gpu_locham_helper_stream_':
>>>> convolut_common_interface_cuda.f90:(.text+0x3b4): undefined reference to
>>>> `gpulocham_stream_'
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `gpu_locden_helper_stream_':
>>>> convolut_common_interface_cuda.f90:(.text+0x6ee): undefined reference to
>>>> `gpulocden_stream_'
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `local_partial_density_gpu_':
>>>> convolut_common_interface_cuda.f90:(.text+0x9ee): undefined reference to
>>>> `gpu_send_'
>>>> convolut_common_interface_cuda.f90:(.text+0xbc2): undefined reference to
>>>> `gpu_receive_'
>>>> convolut_common_interface_cuda.f90:(.text+0xbee): undefined reference to
>>>> `create_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0xc70): undefined reference to
>>>> `mem_copy_f_to_c_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0xca1): undefined reference to
>>>> `gpu_send_pi_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0xde4): undefined reference to
>>>> `gpulocden_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0xe82): undefined reference to
>>>> `gpu_receive_pi_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0xea0): undefined reference to
>>>> `mem_copy_c_to_f_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0xea7): undefined reference to
>>>> `launch_all_streams_'
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `preconditionall_gpu_':
>>>> convolut_common_interface_cuda.f90:(.text+0xfc4): undefined reference to
>>>> `gpu_send_'
>>>> convolut_common_interface_cuda.f90:(.text+0x120e): undefined reference
>>>> to
>>>> `gpu_receive_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1322): undefined reference
>>>> to
>>>> `create_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x135a): undefined reference
>>>> to
>>>> `mem_copy_f_to_c_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1387): undefined reference
>>>> to
>>>> `gpu_send_pi_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x144d): undefined reference
>>>> to
>>>> `gpuprecond_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x147e): undefined reference
>>>> to
>>>> `gpu_receive_pi_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x149c): undefined reference
>>>> to
>>>> `mem_copy_c_to_f_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x14b6): undefined reference
>>>> to
>>>> `launch_all_streams_'
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `local_hamiltonian_gpu_':
>>>> convolut_common_interface_cuda.f90:(.text+0x15d1): undefined reference
>>>> to
>>>> `gpu_send_'
>>>> convolut_common_interface_cuda.f90:(.text+0x181e): undefined reference
>>>> to
>>>> `gpu_receive_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1869): undefined reference
>>>> to
>>>> `create_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x18c0): undefined reference
>>>> to
>>>> `mem_copy_f_to_c_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x18fd): undefined reference
>>>> to
>>>> `gpu_send_pi_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1904): undefined reference
>>>> to
>>>> `launch_all_streams_'
>>>> convolut_common_interface_cuda.f90:(.text+0x19fe): undefined reference
>>>> to
>>>> `create_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1ac9): undefined reference
>>>> to
>>>> `gpulocham_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1b04): undefined reference
>>>> to
>>>> `gpu_receive_pi_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1b2b): undefined reference
>>>> to
>>>> `mem_copy_c_to_f_stream_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1b4d): undefined reference
>>>> to
>>>> `launch_all_streams_'
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `free_gpu_':
>>>> convolut_common_interface_cuda.f90:(.text+0x1b79): undefined reference
>>>> to
>>>> `gpu_deallocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1b89): undefined reference
>>>> to
>>>> `gpu_deallocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1b99): undefined reference
>>>> to
>>>> `gpu_deallocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1ba9): undefined reference
>>>> to
>>>> `gpu_deallocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x1bb9): undefined reference
>>>> to
>>>> `gpu_deallocate_'
>>>>
>>>> ../libbigdft.a(convolut_common_interface_cuda.o):convolut_common_interface_cuda.f90:(.text+0x1bc8):
>>>> more undefined references to `gpu_deallocate_' follow
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `adjust_keys_for_gpu_':
>>>> convolut_common_interface_cuda.f90:(.text+0x2348): undefined reference
>>>> to
>>>> `gpu_int_allocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2368): undefined reference
>>>> to
>>>> `gpu_int_send_'
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `prepare_gpu_for_locham_':
>>>> convolut_common_interface_cuda.f90:(.text+0x247f): undefined reference
>>>> to
>>>> `set_gpu_double_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2b47): undefined reference
>>>> to
>>>> `gpu_int_allocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2b6d): undefined reference
>>>> to
>>>> `gpu_int_send_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2dc4): undefined reference
>>>> to
>>>> `gpu_allocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2e12): undefined reference
>>>> to
>>>> `gpu_allocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2e35): undefined reference
>>>> to
>>>> `gpu_allocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2e58): undefined reference
>>>> to
>>>> `gpu_allocate_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2e92): undefined reference
>>>> to
>>>> `gpu_allocate_'
>>>>
>>>> ../libbigdft.a(convolut_common_interface_cuda.o):convolut_common_interface_cuda.f90:(.text+0x2ec6):
>>>> more undefined references to `gpu_allocate_' follow
>>>> ../libbigdft.a(convolut_common_interface_cuda.o): In function
>>>> `prepare_gpu_for_locham_':
>>>> convolut_common_interface_cuda.f90:(.text+0x2f3b): undefined reference
>>>> to
>>>> `cpu_pinned_allocation_'
>>>> convolut_common_interface_cuda.f90:(.text+0x2f5b): undefined reference
>>>> to
>>>> `cpu_pinned_allocation_'
>>>> make[1]: *** [conv_check] Error 1
>>>> make[1]: Leaving directory `/work/chem/bigdft-1.3.0/src/CUDA'
>>>> make: *** [check-am] Error 2
>>>>
>>>>
>>>> Most undefined references are included in ligGPU.a, however, for some
>>>> known
>>>> reason, ligGPU.a was not recognized.
>>>>
>>>> Thanks for any help.
>>>>
>>>> Jyh-Shyong Ho, Ph.D.
>>>> Research Scientist
>>>> National Center for High Performance Computing
>>>> Hsinchu, Taiwan, ROC
>>>>
>>>>
>>
>>
>
>