User Tools

Site Tools


Verification/pipeline analysis of correlated data in AIPS

The purposes of data verification are:

  1. to make sure the correlated data are OK, and no recorrelation is required.
  2. to provide feedback to PIs, and telescopes, on how their observations went, and any problems with the data.

We also want to create off-source flagging (UVFLG) and calibration (ANTAB) files for each experiment, to provide to the PIs.

Pipeline verification is now done immediately after correlation as described in the Espresso documentation.

Location of files

Pipelining is done on Zeus (not magnus). Output data is written to $CORR_DATA and the pipeline area is $PIPE. Pipeline directories can be setup for a new experiment with mkpipe <expcode>.

Normally we use AIPS user number 1000 for standard pipelining. To see which AIPS ID numbers are currently in use: .

The correlator output data is stored in $CORR_DATA/<expcode>, where <expcode> should be replaced with the desired experiment code. Everything is also archived to the Pawsey data store.
Correlator input files are kept in $CORR_HOME/<year>/<session>/<expcode>.

After pipelining, a wiki page is created using archpipe will post this to the ATNF wiki. <expname>.wikilog > <expname>.txt
archpipe <expname>

Descriptions of the output files should be filled in in the table under “Correlation Notes”; comments can also be added to the “Analysis Notes” section.

Setting up off-source flagging and amplitude calibration files

Note: an empty .uvflg file may be used for the purposes of pipeline verification. The mkpipe command will create this for you.

The UVFLG file

For each station, a file formatted for task UVFLG should be extracted from the telescope observing log.

For Hobart, Ceduna and Tidbinbilla, the files should have been placed in the experiment ftp area at ATNF, which is linked from the experiment setup page on the VLBI Observers wiki. Usually the files are named <expcode><st>.uvflg where <expcode> is the experiment code and <st> is the 2-letter station code.

Off-source flagging files for ATNF stations are generated by scripts also linked from experiment page on the Observers wiki.

In case the flag file for a particular station is not available, the a priori flag file generated by SCHED (which can also be found in the experiment ftp area, <expcode>.flag) may be used for that station, but it is always better to use the off-source data extracted from the observing logs if possible, as this reflects when the telescopes were actually off-source.

The UVFLG files from each station can be concatenated into one file <expcode>.uvflg

Some editing may be required - see example below.

Example for v212a

Get UVFLG file for Tid from the ftp area:


Get UVFLG files for ATNF stations using Chris Phillip's script linked from (under “Observing logs” at the bottom of the page). I do this by right-clicking on the link, select “Copy Link Location” and paste into the terminal where I'm running wget. But then I have to edit the line to escape '&' characters in the query (by putting a '\' before them).

For Parkes:

wget -O v212apa.uvflg\&start=159/09:00:00\&end=159/20:58:00

For ATCA (need to add year for August 2008 and earlier experiments):

wget -O v212aat.uvflg\&end=159/20:58:00\&year=2008

For Mopra:

wget -O v212amp.uvflg


cat v212ati.flag v212a??.uvflg » v212a.uvflg

Edit station names in the uvflg file if necessary. Run prtan to see list of antenna names in AIPS. For v212a, it was necessary to replace 'TID' with 'DSS43' and replace all 'ATCA' with 'CATW109'. Also edited to flag Tid data before start/after stop time.

The ANTAB file

Currently, producing an ANTAB file is the responsibility of the PI. Tips are provided on the calibration page.

Automated processing with the LBA pipeline

See below for a description of how to load data and produce diagnostic plots interactively in AIPS.

An alternative, quicker and efficient way of processing the LBA data and producing some useful plots is to use the LBA pipeline.

This is a python/ParselTongue based pipeline.

How it Works

The LBA pipeline takes a text file as an input, which contains the relevant details of an experiment you wish to process the data from.

An example of the LBA pipeline input file is supplied with the pipeline in a file called lba_template.inp. The input file template can be copied from there, then edited for a certain experiment (mkpipe does this much of this automatically, but its output may be further tuned by hand).

Pipeline Setup:

The pipeline is run on

mkpipe <expcode> will do the following: create a <expcode> directory in the standard area ($PIPE). Create two sub-directories 'in' and 'out' in the <expcode> directory. The 'in' should contain the pipeline input file and the .uvflg file and 'out' is the directory which the plot files will be written to. Copy the lba_template.inp file to the 'in' directory and substitute standard FITS file names.

The following parameters can be set in the input file:

tmask = 1                         # tmask is a range of steps to be carried out. Generally it is advisable to set tmask=1 as it will run the whole pipeline
experiment = <expcode>            # the ident of the experiment e.g. v252j
userno = aipsid                   # the AIPS ID you wish to use for the processing (1000 by default)
refant = Mp, Ho                   # a priotised list of reference antennas. Generally the compact array can be specified as the first in the list
plotref = MP, Ho                  # plotref is a list of antennas, baselines to which will be plotted. 
# the phase calibrator(s) and the respective source(s) can be specified if phase-referencing
phaseref = <calibrator>           
target =   <source>

solint = -1                       # solution interval. '-1' set the interval to the typical scan length
fits_file =                       # path to the data file. Multiple files, separated with a comma, can be specified.
indir = /data/pipe/<expcode>/in   # the directory containing the input file and the .uvflg file
outdir = /data/pipe/<expcode>/out # the directory to write out the plot files to

The optional parameters can be left as they are

Running the Pipeline

The pipeline program is called, but this must be run through the slurm batch scheduler. After completing the pipeline setup, cd to the directory $PIPE/<expcode>/ and run the pipeline as follows:

lbawrap in/<expcode>.inp

Where <expcode>.inp is the input file created previously. lbawrap is a slurm wrapper for the pipeline which will launch it on the Zeus batch scheduler. It is an alias for sbatch -J $(basename $1) –time 12:00:00 –wrap $1.

The status of running pipelines can be checked with squeue and/or tail -f out/<expcode>.pypelog.

The pipeline will automatically run various AIPS tasks such as FITLD, DTSUM, LISTR, POSSM, VPLOT, FRING etc and will produce useful diagnosis and verification plots in pdf files written in the directory /data/pipe/<expcode>/out.

The plots can be viewed in any pdf viewer.

In complicated cases, e.g. when there is a clock jump, it may be useful to visually inspect the FRING delays. The pipeline includes a script to help you do that: -p "AIPSNO NAME CLASS DISK SEQ"

Checking properties of calibration sources

Look in the Radio Fundamental Catalog compiled by Leonid Petrov.

Interactive method for producing plots in AIPS

The following is somewhat out of date.

Set an environment variable to point to the data area, e.g. export DATA=/data/corr/corrdat/<expcode>. If you start aips from e.g. /data/corr/pipe/<expcode> you can use PWD to write plots out to that area.

Change to the directory that is to contain the output plots etc. for the experiment to be verified. Otherwise set an environment variable with the path to this directory, e.g. export V212A=/data/pipe/v212a.

Start AIPS with

aips tv=local:0

Enter AIPS ID number at the prompt.

Three windows should appear on your local desktop: the message server, the TV screen server, and TEK server (generally not used). If the message server dies, AIPS messages will appear in the main AIPS terminal instead.

Note that the adverbs for tasks listed below are just a guide, not a complete list, i.e. for a specific task you may need to reset some other input parameters in order to get sensible output. Feel free to add other specific adverb settings to these instructions if you think this will help you and others in the future! Use help or explain (task) to get help on AIPS tasks and their adverbs.

Loading the data

Since January 2010, output data are FITS-IDI format created with difx2fits. These can be loaded with task FITLD. Specify DATAIN.

Prior to this, the output data was normally in RPFITS format, which must be loaded with task ATLOD.

task 'atlod'
infile 'PWD:<expcode>.rpf $ if required, replace PWD with previously set environment variable for data file directory
outname '<expcode>        $ TIP: open quote in AIPS can be used to read/write lower case, closed quotes ALWAYS refer to upper case
aparm -1,0,1           $ aparm(3) retains autocorrelations. aparm(1)=-1 "kludge" polarizations. 
inp                    $ shows inputs to task
go                     $ executes task

It seems aparm(1) has to be set to -1 to properly handle single polarization LBA data. For dual polarizations aparm(1) can be set to zero. Defaults should be OK for all other parameters.

Check on progress in the message window. Example messages:

CUPPA0> ATLOD1: Task ATLOD  (release of 31DEC08) begins
CUPPA0> ATLOD1: Using compressed uv data format.
CUPPA0> ATLOD1: Dropping all flagged visibilities.
CUPPA0> ATLOD1: Retaining autocorrelations.
CUPPA0> ATLOD1: Vis weights set to integration time in units of 15 sec
CUPPA0> ATLOD1: Averaging all pulsar bins
CUPPA0> ATLOD1: Processing file:  1
CUPPA0> ATLOD1: Input polarizations: RR,LL
CUPPA0> ATLOD1: Create v212a       .UVDATA.   1 (UV)  on disk  1  cno    1
CUPPA0> ATLOD1: Recomputing UVWs for pre-20/02/1998 LBA data
CUPPA0> ATLOD1: File:  1,  Scan:   1
CUPPA0> ATLOD1: Date: 2008-06-07    UT start:  0 09:00:01
CUPPA0> ATLOD1: Visibilities accepted/rejected for each source/frequency:
CUPPA0> ATLOD1:    Sideband              +1          +1          +1          +1
CUPPA0> ATLOD1:    AIPS IF axis           1           2           3           4
CUPPA0> ATLOD1:    Source/Freq(GHz)   2.226000   2.242000   2.258000   2.274000
CUPPA0> ATLOD1:    1424-418           6000/0      6000/0      6000/0      6000/0
CUPPA0> ATLOD1:    SN2004IP          97200/0     97200/0     97200/0     97200/0
CUPPA0> ATLOD1:    J1826-3650        97200/0     97200/0     97200/0     97200/0
CUPPA0> ATLOD1:    1921-293          15000/0     15000/0     15000/0     15000/0
CUPPA0> ATLOD1: Category summaries of visibilities for this scan:
CUPPA0> ATLOD1: Finished reading file number  1.
CUPPA0> ATLOD1: The average visibility weight was  0.133
CUPPA0> ATLOD1: Appears to have ended successfully
CUPPA0> ATLOD1: cuppa01 31DEC08 TST: Cpu=     13.3  Real=     24  IO=       443

To view the new catalog entry and check the header information of the loaded file:

getn 1
imh     $ imheader (all AIPS commands use minimum match)


AIPS 1: Image=1424-418  (UV)         Filename=v212a       .UVDATA.   1
AIPS 1: Telescope=ATLBA              Receiver=
AIPS 1: Observer=                    User #= 1975
AIPS 1: Observ. date=07-JUN-2008     Map date=26-MAR-2009
AIPS 1: # visibilities    215400     Sort order  TB
AIPS 1:            WEIGHT  SCALE
AIPS 1: ----------------------------------------------------------------
AIPS 1: Type    Pixels   Coord value     at Pixel     Coord incr   Rotat
AIPS 1: COMPLEX      1   1.0000000E+00       1.00  1.0000000E+00    0.00
AIPS 1: STOKES       2  -1.0000000E+00       1.00 -1.0000000E+00    0.00
AIPS 1: FREQ        65   2.2260000E+09       1.00  2.5000000E+05    0.00
AIPS 1: IF           4   1.0000000E+00       1.00  1.0000000E+00    0.00
AIPS 1: RA           1    14 27 56.297       1.00       3600.000    0.00
AIPS 1: DEC          1   -42 06 19.427       1.00       3600.000    0.00
AIPS 1: ----------------------------------------------------------------
AIPS 1: Coordinate equinox 2000.00
AIPS 1: Maximum version number of extension files of type HI is   1
AIPS 1: Maximum version number of extension files of type FQ is   1
AIPS 1: Maximum version number of extension files of type SU is   1
AIPS 1: Maximum version number of extension files of type AN is   1
AIPS 1: Maximum version number of extension files of type NX is   1
AIPS 1: Maximum version number of extension files of type CL is   1

In general the data should appear with sort order 'TB'. (In case not, it may be necessary to run UVSRT or MSORT, followed by INDXR, but why are the data out of order?)

Sometimes there will be more than one output data file for a given experiment. It may make sense to load them into a single file (e.g. for different time-ranges correlated with the same parameters), or if different correlation parameters were used (e.g. a spectral line and a continuum pass), load each pass separately.

Flagging of off-source data

task 'uvflg'
getn 1
intext 'PWD:<expcode>.uvflg   $file with flagging commands
inp                           $check inputs

Delete edge channels?

Summary of (u,v) data points per baseline for entire experiment:

task 'dtsum'
docrt -1                          $ print to file (prints to screen if docrt >1)
aparm 0                           $ set all aparm values to 0
outprint 'PWD:<expcode>.DTSUM.txt $ specify filename
inp                               $ check inputs

This can be useful for checking antenna numbers.

Note that if flag table has not been applied, visibilities will be present for all stations for the whole experiment, whether the station is off-source/recording or not.

Scan listing:

task 'listr'
optype 'scan'
outprint 'PWD:<expcode>.SCAN.txt  $ specify filename
inp                               $ check inputs

Produce some plots of uncalibrated data

Plot autocorrelations

These will indicate e.g. RFI at particular stations.

task 'possm'
stokes 'half' $ =RR,LL pols
aparm 0
aparm(7) 1    $ Label x-axis with frequency
aparm(8) 1    $ to plot total power
aparm(9) 1    $ plot IFs together
solint -1     $ average over each scan
nplots 9      $ 9 plots per page (the maximum)
dotv -1       $ plot to file
inp           $ check inputs

Note: this can result in a lot of output, especially for experiments with a large number of scans!

imh will show the number of 'PL' (plot) type extension files produced.

Print the plots to a postscript file:

task 'lwpla'
plver 1
inver 9999 $ >= number of plots
outfile 'PWD:<expcode>
inp        $ check inputs

Check the output file: in a terminal window outside of AIPS, 'gv <expcode>'

You can then use the verb EXTDEST (inext 'pl';inver -1;extdest) to delete the PL extension files.

Alternatively, you will have to keep track of the maximum PL version number after each run, in order to set 'plver' each time you run LWPLA to produce output plot files (otherwise you will end up with one really huge postscript file containing all plots). Also, if outfile already exists, LWPLA will append to it, so use a unique file name if you don't want this.

Plot uncalibrated amplitude and phase against time

task 'vplot'
optype ''         $ cross-correlation
bchan 10;echan 55 $ discard edge channels (example, for 64-channel data - CHECK how many channels in your data (FREQ in command 'imh'))
avgchan 1         $ average channels
stokes 'half'     $ for dual polarization (RR,LL)
solint 0.5        $ averaging time (mins)
aparm 0
bparm 0,-1        $ plot amp & phase together
nplots 6
inp               $ check inputs
task 'lwpla'
plver 1
inver 9999 $ >= number of plots (Can check again after vplot completes, using imh to check PL number - should be small)
outfile 'PWD:<expcode>
inp        $ check inputs

Note: If flagging has not been applied, this plot will contain off-source data! Also, where a station has NO data, the amplitudes appear as zero but the phase does not - i.e. the phases at such times may look meaningful, but are not! Be warned…

Plot uncalibrated amplitude and phase against frequency

Preferably choose calibrator sources (adverb sources); these plots will not be very useful for faint targets. Picking which sources are calibrators and which are targets might require some experience. The target source may be obvious from the experiment title. Otherwise it may help to check the scan listing produced with LISTR (observed sources are listed near the end). A typical phase-referencing observation alternates between target and phase-reference calibrator scans. To check which sources are likely to be calibrators you can try looking for them in the ATCA calibrators catalogue.

Sometimes the target sources will be many and bright (e.g. for experiments such as v252); in that case just leave the source selection blank.

You may wish to limit the output by setting antennas to plot baselines to a chosen reference antenna (or two).

The output of DTSUM shows the antenna numbers (or use task PRTAN).

task 'possm'
sources '1424-418','J1826-3650','1921-293' $ example source selection
stokes 'half' $ May wish to plot cross-hand pols 'RL,LR' (if present) to a separate file.
bchan 0;echan 0
aparm 1,1,0,0,-180,180,1,0,1 $ plot phase with fixed range -180,180 (see help)
solint -1     $ average over each scan
nplots 9      $ 9 plots per page (the maximum)
dotv -1       $ plot to file
inp           $ check inputs
task 'lwpla'
plver 1
inver 9999 $ >= number of plots
outfile 'PWD:<expcode>

Fringe-fit the data

Example inputs. These will need to be specified carefully for each experiment.

Note: Old LBA data in RPFits format, read in with ATLOD, requires the integration time to be set (dparm(4)). The “correlation notes” on the wiki should include what integration time was used. This does NOT need to be set for data in FITS-IDI format read in with FITLD (default for experiments correlated since January 2010).

task 'fring'
calsour '1424-418','J1826-3650','1921-293' $ select calibrators
bchan 10;echan 55                          $ discard edge channels (in this case there are 64 channels per IF)
refant 1                                   $ reference antenna should be sensitive, and on source most of the time
search 3,2,4                               $ prioritised list for exhaustive search (not usually needed) 
solint 2                                   $ solution interval (minutes) - typical length of the phase-reference calibrator scan.
aparm 0
aparm(6)=2                                 $ prints a lot of information including antenna S/N ratio
aparm(9)=1                                 $ do exhaustive baseline search
dparm(4)=2                                 $ the integration time in seconds - MUST BE SET for LBA data

Plot FRING solutions and fringe-fitted data

Plot FRING delay solutions

Note: If you realise you've messed something up, and you need to run abort, be careful - it doesn't always abort nicely. If as a consequence you get errors and your task starts dying of “unnatural causes”, see this thread. Moral of the story: Run getn 1 to go back to your original file and clrs to clear any problem statuses. If like Claire, you aborted because you'd forgotten to reset the pl or sn files, best option is to let it run, it seems, but DO remember to clear them before you run your task again :)

Note 2: ucat will show the READ/WRITE status flag. If you see this and there's no task running, then it's probably a spurious flag from an aborted task (do clrstat)

task 'snplt'
inex 'sn'            $ plot the 'SN' (solution) table
inver 0
sources ''
nplots 8
optype 'dela'        
$pixrange -5e-8,5e-8  $ example to limit displayed y-axis range to +/-50 nanosec (time in seconds)
dotv -1

Delay solutions should be typically within tens of nanoseconds. No unexpected gaps?

Note there may be a few outlying points (due to e.g. 'bad' solution with marginal S/N)

Print delay solutions to a file:

tget lwpla
outfile 'PWD:<expcode>
inext 'pl';inver -1;extdest

Plot FRING rate solutions

tget snplt
optype 'rate'        
pixrange 0

Typically less than a few milliHz?

tget lwpla
outfile 'PWD:<expcode>
inext 'pl';inver -1;extdest

Plot FRING phase corrections

tget snplt
optype 'phas'

Nicely connected on phase-reference calibrator? (May not appear well-connected for observations of many sources spread over the sky)

tget lwpla
outfile 'PWD:<expcode>
inext 'pl';inver -1;extdest

Apply FRING solutions

Apply fringe-fit solutions from calibrator sources used in FRING (calsour) to all sources (sources), creating a new 'CL' table.

task 'clcal'
interpol 'ambg'     $ uses rates to resolve phase ambiguities (where phase changes by large amount between cal scans)
bparm 0             $ for no smoothing
inver 1             $ specify SN table to apply
gainver 0;gainuse 0 $ apply to highest numbered CL table, write new CL table
$ NB - keep account of SN & CL tables - be careful not to apply solutions twice. 

Plot data with FRING solutions applied

Amplitude and phase against time
tget 'vplot'
docal 1      $ apply the CL table (gainuse 0 = highest)
tget lwpla
outfile '
inext 'pl';inver -1;extdest
Amplitude and phase against frequency
tget possm
sources '1424-418','J1826-3650','1921-293' $ example source selection
stokes 'half'
bchan 0;echan 0
aparm 1,0     $ autoscale phase plots this time to see small deviations
aparm(7)=1    $ label x-axis with frequency
aparm(9)=1    $ plots IFs together in a row
solint -1     $ average over each scan
nplots 9      $ 9 plots per page (the maximum)
dotv -1       $ plot to file
inp           $ check inputs
docal 1       $ apply FRING solutions
tget lwpla
outfile 'PWD:<expcode>
inext 'pl';inver -1;extdest

For the fringe-fitted data, the phases should be close to zero and flat across the band (in the parallel-hand polarizations RR,LL, and where there is sufficient S/N).

Plot (u,v) coverage for the target source(s)


task 'uvplt'
sour 'SN2004IP''    $ 1 source at a time
bchan 32;echan 32   $ plot only 1 channel from each IF to speed things up
aparm 0
bparm 6,7,0         $ plot u,v coverage
inp                 $ check inputs
tget lwpla
outfile 'PWD:v212a_<srcnam>
inext 'pl';inver -1;extdest

To do: bandpass calibration, amplitude calibration (ANTAB, APCAL), imaging…

Exiting AIPS

When you've finished with AIPS, exit with the command kleenex to close all servers.

===== Put plots on the CIRA ftp server ===== (If needed, convert plots to PDF with ps2pdf. NB the pipeline output plots are already in PDF) Ftp to as astronomy (password required). passive to use passive mode
cd to anonymous/<expcode>
mput <expcode>*.pdf
mput <expcode>*.txt

Edit the experiment page on the wiki

Edit the “Analysis notes and links to plots” section on<expcode>

Insert links to the plots in the ftp area (replace all “expcode” with the actual experiment code) and comments as required.

Template for analysis notes

There is now a script ~corrtest/progs/Pypeline/ to produce the wiki page after running the pipeline.

Cleaning up AIPS area

The verb ALLDEST can be used to delete all AIPS files for a user number, once the data have been finished with, e.g.

inname ''
inclass ''
inseq 0
intype ''

When all data have been deleted, ALLDEST will also ask to destroy all SAVE/GET and TGET files.

FREESPAC tells how much space is available on the AIPS disk.

correlator/verification.txt · Last modified: 2018/05/18 14:13 by cormac