Generic instructions for getting and compiling the code are available in the developer's wiki. We normally use the install-difx script described there.

These instructions are specific to DiFX-2.4, but other versions should be similar. I use the install-difx script distributed with DiFX. Edit $DIFXROOT/setup.bash to amend the path to the IPP libraries, etc.

# change to GNU compilers, etc.
module swap PrgEnv-cray PrgEnv-gnu
module load gsl

# use the correct C++ compiler by editing setup.bash:
export MPICXX=CC

# use own version of expat (see below)
export LIBRARY_PATH=${HOME}/lib64/magnus/lib

libexpat.a is no longer available on Magnus, so I have installed my own:

cd ~/src
wget https://github.com/libexpat/libexpat/archive/R_2_2_5.tar.gz
tar xvfz R_2_2_5.tar.gz
cd libexpat-R_2_2_5/expat
./buildconf.sh
./configure --prefix=${HOME}/lib64/magnus --without-xmlwf
make install

Also need to add expat libs to make files for mpifxcorr (autoconf does not pick them up). I hacked this by adding -lexpat to the 'LIBS=' line of the configure.ac file.

To run espresso, you will need to set a number of environment variables and load some modules:

. /home/cormac/corr.setup

As well as setting a truckload of environment variables, the command above will load the following modules:

module unload PrgEnv-cray
module load PrgEnv-gnu
module load gsl
module load requests
module load numpy
module load psutil
module load matplotlib
module load astropy

Needs to be installed somewhere else, but accessible from the Pawsey network. Currently have installation on a Nectar cloud VM.

Ocean loading coefficients are determined by the 2-letter station. Add non-standard codes to $DIFX_ROOT/share/difxcalc/ocean_pole_tide.coef

The Pypeline runs on Zeus, not Magnus. Zeus shares a home area with Magnus and can see all the Magnus data disks (/scratch, /group).

The Pypeline needs obit/parseltongue

# for obit, need gsl, fftw and the correct version of python
module load PrgEnv-gnu
module load gsl fftw python/2.7.14 swig
# also tell where gsl lives as configure does not pick it up on its own:
# Magnus:
configure --with-gsl=/pawsey/cle52up04/devel/PrgEnv-gnu/5.2.82/gcc/4.9.2/haswell/gsl/1.16 (see $LD_LIBRARY_PATH for gsl module)
# zeus:
./configure --with-gsl=/pawsey/sles12sp3/apps/sandybridge/gcc/4.8.5/gsl/2.4/ (see $LD_LIBRARY_PATH for gsl module)

To use, you will need to set some environment variables and load some modules:

. /home/cormac/pipe.setup

The above command will set some environment variables and load the following modules:

module load python
module load gsl
module load numpy
module load matplotlib

difx2fits is also most conveniently run on the pipeline machine, zeus. It may be necessary to compile difx2fits on zeus specifically for this purpose (due to library incompatibilities with magnus). Rather than have a full DiFX installation on zeus I simply checkout and compile the necessary components. Note that svn does not seem to work properly on zeus, so checkout the code on magnus, then switch to zeus to compile.

cd ~/DiFX/zeus/
svn checkout --depth empty https://svn.atnf.csiro.au/difx/master_tags/DiFX-2.5.2/
cd DiFX-2.5.2
svn update --depth empty  libraries
svn update --depth infinity  libraries/difxio
svn update --depth empty  applications
svn update --depth infinity applications/difx2fits
svn update --depth infinity applications/difx2mark4

run difx2fits as a slurm job with the fitswrap alias.

There follows a full list of what was required to install DiFX-1.5.1 on cuppa01. Many of these steps will not need to be repeated for future installations.

Check out the version of the code you want (preferably one of the tagged versions from master_tags). In the top level of the version edit 'setup.bash' to have the correct paths for cuppa.

• Note that we run the em64t versions of the ipp libraries. The provided 'setup.bash' mentions a -liomp5 which we don't have and so it should be removed.

apt-get install gfortran
apt-get install build-essential pkg-config openmpi-bin libopenmpi-dev
install-difx --perl

• We don't have infiniband so I disabled the annoying messages from openmpi by editing /etc/openmpi/openmpi-mca-params.conf to include:
  btl = ^openib
• Note that cssh and parallel-ssh are installed on cuppa01 allowing you to easily run commands on multiple nodes
• Remember that iptables must be turned off between the cluster nodes. MPI uses all the high numbered ports randomly.
• Another note on 'setup.bash': if this is sourced from .bashrc, the line which echoes DIFX_VERSION to stdout screws up the shell on interactive login - can avoid this problem by commenting out the last 'echo …' line in 'setup.bash'

Hayley's (dummies) guide to DiFX2.0 (beta)