User Tools

Site Tools


difx:difx_run

Running DiFX

Setting up MPI and mpifxcorr

Setting up mpifxcorr is quite a bit easier than it used to be - just add setup.bash or setup.csh to your login script. These setup files can be found at the top level of your installation. Assuming you're running bash, and running DiFX-2.2 (now rather old) the file you want is master_tags/DiFX-2.2/setup.bash. You should have sorted this out when installing DiFX, anyway.

So that defines a bunch of variables such as $DIFXROOT, and adds (among other things) $DIFXROOT/bin/ to your path. So all of the stuff you compiled should be ready to run. Type 'which mpifxcorr' - and you should see the version of mpifxcorr that you will be running.

To make sure MPI is ready to run, type mpirun. It should give you the list of options that it takes. If it complains about the environment not being ready, you may need to run 'mpd'. Other than that - consult the specific MPI documentation for your distribution, there are too many different MPI distributions to go into detailed troubleshooting here.

Running mpifxcorr

If you have downloaded some example data and just want to test it, you can get started right away, since you don't need to generate the auxiliary files. However, its probably better to follow the normal path for setting up a correlation right from the start, where you just use the vex file to generate all the mpifxcorr configuration files. A flowchart showing the way correlations are typically set up is shown below:

As you can see, there are two steps that require input from the user. The first is generating the .v2d file. See the vex2difx doco for the layout of this file, but it can be (and often is) very simple. The other is the machine file and .threads file. These describe which computers mpifxcorr will run on, and how many compute threads it will use. An example is shown here. If you are just running on one machine (e.g. your desktop) then simply use its name over and over again. You need at least N+2 entries, where N is the number of telescopes.

Before you start up the correlator, you should start up errormon2 to log the messages which come out of the correlator. Just type errormon2, and it will write a file called log (you can rename it later if you want).

Once you have your machine file, and errormon2 is running, you should be able to start the correlator with:

mpirun -machinefile machines.list -np X mpifxcorr example.input

replacing machines.list with your machine file name, X with the number of processes you want to use, and example.input with your input file name.

There has been some progress to providing useful configurable scripts for auto-generating machines files and threads files, and for generating the mpirun command so long as you provide some information in a file describing the cluster being used. Currently the primary options for utilities which will do this are genmachines/startdifx, espresso and startcorr.pl. genmachines is designed for correlation from Mark5 units, espresso for file-based correlation, and startcorr.pl for eVLBI. Some day these might be unified.

Auxilliary programs

As described above, DiFX comes with a number of auxiliary programs to assist in setting up the correlator jobs and converting the output to export formats. A full list of the programs, linking to a brief explanation, is given below.

vex2difx

vex2difx is already well documented here.

calcif2

Use calcif2 to generate the model files for the correlation. It works on the .calc file produced by vex2difx, and produces the .delay, .uvw, .rate and .im files. You should run calcif2 after vex2difx and before mpifxcorr.

difx2fits

difx2fits is used to build FITS files. You need to have set the output format to SWIN in the .input file in order to use difx2fits once the correlation has finished. difx2fits is also already well documented here.

difx2mark4

difx2mark4 is used to build geodetic-style mark4 visibility datasets. It runs in an analogous manner to difx2fits. difx2mark4 is already well-documented here.

Reference Guide

A document containing information about mpifxcorr and much of its support software is available here.

difx/difx_run.txt · Last modified: 2021/08/12 18:16 by cormac