User Tools

Site Tools


correlator:difx

Installing DiFX for ATNF

We have access to the latest version of the correlator code via the svn repository at ATNF. Contact Cormac if you need the details.

DiFX comes with a script to make the installation easier. Some instructions are available in the developer's wiki.

Magnus-specific instructions

These instructions are specific to DiFX-2.4, but other versions should be similar. I use the install-difx script distributed with DiFX. Edit $DIFXROOT/setup.bash to amend the path to the IPP libraries, etc.

# change to GNU compilers, etc.
module swap PrgEnv-cray PrgEnv-gnu
module load python/2.7.10
module load doxygen


# use the correct C++ compiler by editing setup.bash:
export MPICXX=CC

Also Need to add expat libs to make files for mpifxcorr (autoconf does not pick them up). I hacked this by adding -lexpat to the 'LIBS=' line of the configure.ac file.

espresso

needs mx.datetime to be built from source. Download from http://www.egenix.com/products/python/mxBase

python setup.py --home=/home/cormac --prefix=''

To run espresso, you will need to set a number of environment variables and load some modules:

. /home/cormac/corr.setup

As well as setting a truckload of environment variables, the command above will load the following modules:

module load python/2.7.10
module load numpy
module load psutil

calcserver

Needs to be installed somewhere else, but accessible from the Pawsey network.

Pypeline

The Pypeline runs on Zeus, not Magnus. Zeus shares a home area with Magnus and can see all the Magnus data disks (/scratch, /group).

The Pypeline needs obit/parseltongue

# for obit, need gsl, fftw and the correct version of python
module load PrgEnv-gnu
module load gsl fftw python/2.7.10
# also tell where gsl lives as configure does not pick it up on its own:
configure --with-gsl=/pawsey/cle52up04/devel/PrgEnv-gnu/5.2.82/gcc/4.8.2/haswell/gsl/1.16 (see $LD_LIBRARY_PATH)

To use, you will need to set some environment variables and load some modules:

. /home/cormac/pipe.setup

The above command will set some environment variables and load the following modules:

module load python
module load gsl
module load numpy
module load matplotlib

Cuppa-specific instructions

There follows a full list of what was required to install DiFX-1.5.1 on cuppa01. Many of these steps will not need to be repeated for future installations.

Check out the version of the code you want (preferably one of the tagged versions from master_tags). In the top level of the version edit 'setup.bash' to have the correct paths for cuppa.

  • Note that we run the em64t versions of the ipp libraries. The provided 'setup.bash' mentions a -liomp5 which we don't have and so it should be removed.
apt-get install gfortran
apt-get install build-essential pkg-config openmpi-bin libopenmpi-dev
install-difx --perl
  • We don't have infiniband so I disabled the annoying messages from openmpi by editing /etc/openmpi/openmpi-mca-params.conf to include:
    btl = ^openib
  • Note that cssh and parallel-ssh are installed on cuppa01 allowing you to easily run commands on multiple nodes
  • Remember that iptables must be turned off between the cluster nodes. MPI uses all the high numbered ports randomly.
  • Another note on 'setup.bash': if this is sourced from .bashrc, the line which echoes DIFX_VERSION to stdout screws up the shell on interactive login - can avoid this problem by commenting out the last 'echo …' line in 'setup.bash'

Hayley's (dummies) guide to DiFX2.0 (beta)

correlator/difx.txt · Last modified: 2016/08/26 12:56 by cormac