The ABINIT Users Mailing List ( CLOSED )

[Eric Roman <ESRoman@berkeley.edu>]

> We prefered to replace (spaceComm /= 0) tests by (spaceComm /= 
> abinit_comm_self). What do you think about that ?

It's tough to say.  I used MPI_COMM_SELF, because it makes sense, when
running in parallel, to skip the reduction operations over the local 
processor.

It's misleading to consider abinit_comm_self a communicator.  It isn't.
We're never going to initialize it.  (Why would we?)  MPI_COMM_SELF is there
to do this.  If we ever by mistake pass abinit_comm_self into an MPI routine
we'll get an error back, because the communicator isn't initialized.

I think abinit_comm_self shouldn't be considered a real communicator.
It shouldn't be used as one.  It should be considered a dummy constant,
used only in the serial version of the code.

If someone for some reason in the future wants to use a local communicator
in an MPI call, they shouldn't be using abinit_comm_self to do it.  We would
want them to use MPI_COMM_SELF.  That's why I had xcomm_setup return 
MPI_COMM_SELF, rather than abinit_comm_self.

Use abinit_comm_self in the serial version only.  MPI_COMM_SELF in parallel.

abinit_comm_self should be be renamed to abinit_comm_dummy, as 
abinit_comm_self
is misleading.

BTW.  I tested these changes with LAM MPI over the weekend, and I was able to
run the full suite of parallel tests successfully.  I'll be testing on our SP
later today.

 - E

> 
> 
> Regards,
> 
> Marc
> 
> 
> 
> Eric Roman a écrit :
> >Marc
> >
> >Thanks for confirming, and thanks for your help on this.
> >
> >I produced a patch last night along these lines.  I've not yet tested it,
> >but tell me if you think this is the right way to go.
> >
> >I changed the (spaceComm /= 0) tests to (spaceComm /= MPI_COMM_SELF) inside
> >the reduction routines in xfuncmpi.f.  I'm fairly certain this is the 
> >correct
> >way to do things.
> >
> >I also changed xdef_comm.f to return a non-zero constant in case it was
> >running in serial.  (This should be unnecessary, due to the way the 
> >preprocessor is used, but I did it just to clarify things, and for safety
> >in case a similar change arises in the future.)
> >
> >There's a patch attached to this e-mail with the changes.  I'd appreciate
> >your comments.
> >
> > - E
> >
> >
> >
> >------------------------------------------------------------------------
> >
> >Index: xdef_comm.f
> >===================================================================
> >--- xdef_comm.f      (revision 6)
> >+++ xdef_comm.f      (working copy)
> >@@ -18,13 +18,16 @@
> > !!
> > !! SOURCE
> > 
> >-    ! xcomm_init  definition
> >+! xcomm_init  definition
> > 
> >+!Used to distinguish serial execution when performing reduction 
> >operations.
> >+
> >  subroutine xcomm_world(spaceComm)
> > #          if defined MPI || defined MPI_FFT
> > !         use mpi    
> > #          endif
> >  implicit none
> >+ integer, parameter :: abinit_comm_self = 12345
> > #          if defined MPI || defined MPI_FFT
> >            include 'mpif.h'
> > #          endif
> >@@ -33,7 +36,7 @@
> > #          if defined MPI
> >             spaceComm = MPI_COMM_WORLD
> > #          else
> >-            spaceComm = 0           
> >+            spaceComm = abinit_comm_self
> > #          endif
> >  end subroutine xcomm_world
> > 
> >@@ -43,6 +46,7 @@
> > !          use mpi    
> > #          endif
> >  implicit none
> >+ integer, parameter :: abinit_comm_self = 12345
> > #          if defined MPI || defined MPI_FFT
> >            include 'mpif.h'
> > #          endif
> >@@ -52,18 +56,18 @@
> >             if (mpi_enreg%paral_level > 1) then
> >                 if (mpi_enreg%paral_level == 2) then
> >                         spaceComm = MPI_COMM_WORLD
> >-                        else
> >+                else
> >                         if (mpi_enreg%num_group_fft /= 0) then
> >                                 spaceComm =  
> >                                 
> > mpi_enreg%fft_comm(mpi_enreg%num_group_fft)
> >-                                else
> >-                                spaceComm = 0
> >+                        else
> >+                                spaceComm = MPI_COMM_SELF
> >                         endif
> >                 endif
> >-                else
> >+            else
> >                 spaceComm = mpi_enreg%kpt_comm_para(mpi_enreg%ipara)
> >            endif
> > #          else
> >-                spaceComm = 0
> >+                spaceComm = abinit_comm_self
> > #          endif
> >  end subroutine xcomm_init
> > 
> >Index: xfuncmpi.f
> >===================================================================
> >--- xfuncmpi.f       (revision 6)
> >+++ xfuncmpi.f       (working copy)
> >@@ -39,7 +39,7 @@
> >  integer  :: ier,spaceComm
> > #          if defined MPI || defined MPI_FFT
> >             integer , allocatable :: xsum(:)
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >           !Accumulate xval on all proc. in spaceComm
> >             n1 = size(xval)
> >             allocate(xsum(n1))
> >@@ -72,7 +72,7 @@
> >  integer  :: ier,spaceComm
> > #          if defined MPI || defined MPI_FFT
> >             integer  :: xsum
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >           !Accumulate xval on all proc. in spaceComm
> >             call MPI_ALLREDUCE(xval,xsum,1,MPI_INTEGER,&
> >           &  MPI_SUM,spaceComm,ier)
> >@@ -101,7 +101,7 @@
> >  integer :: xval,xsum
> >  integer  :: ier,spaceComm
> > #          if defined MPI || defined MPI_FFT
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >           !Accumulate xval on all proc. in spaceComm
> >             call MPI_ALLREDUCE(xval,xsum,1,MPI_INTEGER,&
> >           &  MPI_SUM,spaceComm,ier)
> >@@ -132,7 +132,7 @@
> > #          if defined MPI || defined MPI_FFT
> >             integer , allocatable :: xsum(:)
> >           !Accumulate xval on all proc. in spaceComm
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >             allocate(xsum(n1))
> >             call MPI_ALLREDUCE(xval,xsum,n1,MPI_INTEGER,&
> >           &  MPI_SUM,spaceComm,ier)
> >@@ -164,7 +164,7 @@
> >  integer  :: ier,spaceComm
> > #          if defined MPI || defined MPI_FFT
> >             integer , allocatable :: xsum(:,:)
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >           !Accumulate xval on all proc. in spaceComm
> >             n1 =size(xval,dim=1)
> >             n2 =size(xval,dim=2)
> >@@ -200,7 +200,7 @@
> > #          if defined MPI || defined MPI_FFT
> >             integer , allocatable :: xsum(:,:,:)
> >           !Accumulate xval on all proc. in spaceComm
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >             n1 =size(xval,dim=1)
> >             n2 =size(xval,dim=2)
> >             n3 =size(xval,dim=3)
> >@@ -236,7 +236,7 @@
> > 
> > #          if defined MPI || defined MPI_FFT
> >             real(dp) , allocatable :: xsum(:)
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >        !Accumulate xval on all proc. in spaceComm
> >             n1 = size(xval)
> >             allocate(xsum(n1))
> >@@ -272,7 +272,7 @@
> > #          if defined MPI || defined MPI_FFT
> >             real(dp)  :: xsum
> >        !Accumulate xval on all proc. in spaceComm
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >             call MPI_ALLREDUCE(xval,xsum,1,MPI_DOUBLE_PRECISION,&
> >           &  MPI_SUM,spaceComm,ier)
> >             xval  = xsum
> >@@ -303,7 +303,7 @@
> > 
> > #          if defined MPI || defined MPI_FFT
> >             real(dp) , allocatable :: xsum(:)
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >        !Accumulate xval on all proc. in spaceComm
> >             allocate(xsum(n1))
> >             call MPI_ALLREDUCE(xval,xsum,n1,MPI_DOUBLE_PRECISION,&
> >@@ -337,7 +337,7 @@
> > 
> > #          if defined MPI || defined MPI_FFT
> >             real(dp) , allocatable :: xsum(:,:)
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >             n1 = size(xval,dim=1)
> >             n2 = size(xval,dim=2)
> >        !Accumulate xval on all proc. in spaceComm
> >@@ -373,7 +373,7 @@
> > 
> > #          if defined MPI || defined MPI_FFT
> >             real(dp) , allocatable :: xsum(:,:,:)
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >             n1 = size(xval,dim=1)
> >             n2 = size(xval,dim=2)
> >             n3 = size(xval,dim=3)
> >@@ -409,7 +409,7 @@
> >  integer  :: ier,spaceComm
> >  
> > #          if defined MPI || defined MPI_FFT
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >             !Accumulate xval on all proc. in spaceComm
> >             call MPI_ALLREDUCE(xval,xsum,n1,MPI_DOUBLE_PRECISION,&
> >           &  MPI_SUM,spaceComm,ier)
> >@@ -439,7 +439,7 @@
> >  integer  :: ier,spaceComm
> > 
> > #          if defined MPI || defined MPI_FFT
> >-            if (spaceComm /= 0) then
> >+            if (spaceComm /= MPI_COMM_SELF) then
> >             !Accumulate xval on all proc. in spaceComm
> >             call MPI_ALLREDUCE(xval,xsum,n1,MPI_DOUBLE_PRECISION,&
> >           &  MPI_SUM,spaceComm,ier)

-- 
Eric Roman                       Department of Physics
510-642-7302                     UC Berkeley