The purposes of data verification are:
We also want to create off-source flagging (UVFLG) and calibration (ANTAB) files for each experiment, to provide to the PIs.
Pipeline verification is now done immediately after correlation as described in the Espresso documentation.
Pipelining is done on Zeus (not magnus).
Output data is written to
$CORR_DATA and the pipeline area is
Pipeline directories can be setup for a new experiment with
Normally we use AIPS user number 1000 for standard pipelining. To see which AIPS ID numbers are currently in use:
The correlator output data is stored in
$CORR_DATA/<expcode>, where <expcode> should be replaced with the desired experiment code.
Everything is also archived to the Pawsey data store.
Correlator input files are kept in
After pipelining, a wiki page is created using
archpipe will post this to the ATNF wiki.
lba_feedback.py <expname>.wikilog > <expname>.txt archpipe <expname>
Descriptions of the output files should be filled in in the table under “Correlation Notes”; comments can also be added to the “Analysis Notes” section.
Note: an empty .uvflg file may be used for the purposes of pipeline verification. The
mkpipe command will create this for you.
For each station, a file formatted for task UVFLG should be extracted from the telescope observing log.
For Hobart, Ceduna and Tidbinbilla, the files should have been placed in the experiment ftp area at ATNF, which is linked from the experiment setup page on the VLBI Observers wiki. Usually the files are named <expcode><st>.uvflg where <expcode> is the experiment code and <st> is the 2-letter station code.
Off-source flagging files for ATNF stations are generated by scripts also linked from experiment page on the Observers wiki.
In case the flag file for a particular station is not available, the a priori flag file generated by SCHED (which can also be found in the experiment ftp area, <expcode>.flag) may be used for that station, but it is always better to use the off-source data extracted from the observing logs if possible, as this reflects when the telescopes were actually off-source.
The UVFLG files from each station can be concatenated into one file <expcode>.uvflg
Some editing may be required - see example below.
Get UVFLG file for Tid from the ftp area:
Get UVFLG files for ATNF stations using Chris Phillip's script linked from http://www.atnf.csiro.au/vlbi/wiki/index.php?n=LBAJun2008.V212A (under “Observing logs” at the bottom of the page). I do this by right-clicking on the link, select “Copy Link Location” and paste into the terminal where I'm running wget. But then I have to edit the line to escape '&' characters in the query (by putting a '\' before them).
wget -O v212apa.uvflg http://www.atnf.csiro.au/cgi-bin/vlbi/parkes-onsource.pl?session=Jun2008\&start=159/09:00:00\&end=159/20:58:00
For ATCA (need to add year for August 2008 and earlier experiments):
wget -O v212aat.uvflg http://www.atnf.csiro.au/cgi-bin/vlbi/atca-onsource.pl?start=159/09:00:00\&end=159/20:58:00\&year=2008
wget -O v212amp.uvflg http://www.atnf.csiro.au/cgi-bin/vlbi/mopra_onsource.pl?exper=v212a
cat v212ati.flag v212a??.uvflg » v212a.uvflg
Edit station names in the uvflg file if necessary. Run
prtan to see list of antenna names in AIPS. For v212a, it was necessary to replace 'TID' with 'DSS43' and replace all 'ATCA' with 'CATW109'. Also edited to flag Tid data before start/after stop time.
Currently, producing an ANTAB file is the responsibility of the PI. Tips are provided on the calibration page.
See below for a description of how to load data and produce diagnostic plots interactively in AIPS.
An alternative, quicker and efficient way of processing the LBA data and producing some useful plots is to use the LBA pipeline.
This is a python/ParselTongue based pipeline.
How it Works
The LBA pipeline takes a text file as an input, which contains the relevant details of an experiment you wish to process the data from.
An example of the LBA pipeline input file is supplied with the pipeline in a file called
The input file template can be copied from there, then edited for a certain experiment (
mkpipe does this much of this automatically, but its output may be further tuned by hand).
The pipeline is run on zeus.pawsey.org.au
mkpipe <expcode> will do the following:
create a <expcode> directory in the standard area ($PIPE). Create two sub-directories 'in' and 'out' in the <expcode> directory. The 'in' should contain the pipeline input file and the .uvflg file and 'out' is the directory which the plot files will be written to. Copy the lba_template.inp file to the 'in' directory and substitute standard FITS file names.
The following parameters can be set in the input file:
tmask = 1 # tmask is a range of steps to be carried out. Generally it is advisable to set tmask=1 as it will run the whole pipeline experiment = <expcode> # the ident of the experiment e.g. v252j userno = aipsid # the AIPS ID you wish to use for the processing (1000 by default) refant = Mp, Ho # a priotised list of reference antennas. Generally the compact array can be specified as the first in the list plotref = MP, Ho # plotref is a list of antennas, baselines to which will be plotted. # the phase calibrator(s) and the respective source(s) can be specified if phase-referencing phaseref = <calibrator> target = <source> solint = -1 # solution interval. '-1' set the interval to the typical scan length fits_file = # path to the data file. Multiple files, separated with a comma, can be specified. indir = /data/pipe/<expcode>/in # the directory containing the input file and the .uvflg file outdir = /data/pipe/<expcode>/out # the directory to write out the plot files to
The optional parameters can be left as they are
Running the Pipeline
The pipeline program is called
LBA.py, but this must be run through the slurm batch scheduler.
After completing the pipeline setup, cd to the directory
$PIPE/<expcode>/ and run the pipeline as follows:
Where <expcode>.inp is the input file created previously.
lbawrap is a slurm wrapper for the pipeline which will launch it on the Zeus batch scheduler. It is an alias for
sbatch -J $(basename $1) –time 12:00:00 –wrap LBA.py $1.
The status of running pipelines can be checked with
tail -f out/<expcode>.pypelog.
The pipeline will automatically run various AIPS tasks such as FITLD, DTSUM, LISTR, POSSM, VPLOT, FRING etc and will produce useful diagnosis and verification plots in pdf files written in the directory /data/pipe/<expcode>/out.
The plots can be viewed in any pdf viewer.
In complicated cases, e.g. when there is a clock jump, it may be useful to visually inspect the FRING delays. The pipeline includes a script to help you do that:
clocks.py -p "AIPSNO NAME CLASS DISK SEQ"
Look in the Radio Fundamental Catalog compiled by Leonid Petrov.
The following is somewhat out of date.
Set an environment variable to point to the data area, e.g.
export DATA=/data/corr/corrdat/<expcode>. If you start aips from e.g.
/data/corr/pipe/<expcode> you can use PWD to write plots out to that area.
Change to the directory that is to contain the output plots etc. for the experiment to be verified. Otherwise set an environment variable with the path to this directory, e.g.
Start AIPS with
Enter AIPS ID number at the prompt.
Three windows should appear on your local desktop: the message server, the TV screen server, and TEK server (generally not used). If the message server dies, AIPS messages will appear in the main AIPS terminal instead.
Note that the adverbs for tasks listed below are just a guide, not a complete list, i.e. for a specific task you may need to reset some other input parameters in order to get sensible output. Feel free to add other specific adverb settings to these instructions if you think this will help you and others in the future!
explain (task) to get help on AIPS tasks and their adverbs.
Since January 2010, output data are FITS-IDI format created with
difx2fits. These can be loaded with task FITLD. Specify DATAIN.
Prior to this, the output data was normally in RPFITS format, which must be loaded with task ATLOD.
task 'atlod' infile 'PWD:<expcode>.rpf $ if required, replace PWD with previously set environment variable for data file directory outname '<expcode> $ TIP: open quote in AIPS can be used to read/write lower case, closed quotes ALWAYS refer to upper case aparm -1,0,1 $ aparm(3) retains autocorrelations. aparm(1)=-1 "kludge" polarizations. inp $ shows inputs to task go $ executes task
It seems aparm(1) has to be set to -1 to properly handle single polarization LBA data. For dual polarizations aparm(1) can be set to zero. Defaults should be OK for all other parameters.
Check on progress in the message window. Example messages:
CUPPA0> ATLOD1: Task ATLOD (release of 31DEC08) begins CUPPA0> ATLOD1: Using compressed uv data format. CUPPA0> ATLOD1: Dropping all flagged visibilities. CUPPA0> ATLOD1: Retaining autocorrelations. CUPPA0> ATLOD1: Vis weights set to integration time in units of 15 sec CUPPA0> ATLOD1: Averaging all pulsar bins CUPPA0> ATLOD1: CUPPA0> ATLOD1: Processing file: 1 CUPPA0> ATLOD1: Input polarizations: RR,LL CUPPA0> ATLOD1: CUPPA0> ATLOD1: Create v212a .UVDATA. 1 (UV) on disk 1 cno 1 CUPPA0> ATLOD1: Recomputing UVWs for pre-20/02/1998 LBA data CUPPA0> ATLOD1: CUPPA0> ATLOD1: File: 1, Scan: 1 CUPPA0> ATLOD1: Date: 2008-06-07 UT start: 0 09:00:01 CUPPA0> ATLOD1: CUPPA0> ATLOD1: Visibilities accepted/rejected for each source/frequency: CUPPA0> ATLOD1: Sideband +1 +1 +1 +1 CUPPA0> ATLOD1: AIPS IF axis 1 2 3 4 CUPPA0> ATLOD1: Source/Freq(GHz) 2.226000 2.242000 2.258000 2.274000 CUPPA0> ATLOD1: 1424-418 6000/0 6000/0 6000/0 6000/0 CUPPA0> ATLOD1: SN2004IP 97200/0 97200/0 97200/0 97200/0 CUPPA0> ATLOD1: J1826-3650 97200/0 97200/0 97200/0 97200/0 CUPPA0> ATLOD1: 1921-293 15000/0 15000/0 15000/0 15000/0 CUPPA0> ATLOD1: Category summaries of visibilities for this scan: CUPPA0> ATLOD1: CUPPA0> ATLOD1: Finished reading file number 1. CUPPA0> ATLOD1: CUPPA0> ATLOD1: The average visibility weight was 0.133 CUPPA0> ATLOD1: Appears to have ended successfully CUPPA0> ATLOD1: cuppa01 31DEC08 TST: Cpu= 13.3 Real= 24 IO= 443
To view the new catalog entry and check the header information of the loaded file:
ucat getn 1 imh $ imheader (all AIPS commands use minimum match)
>imh AIPS 1: Image=1424-418 (UV) Filename=v212a .UVDATA. 1 AIPS 1: Telescope=ATLBA Receiver= AIPS 1: Observer= User #= 1975 AIPS 1: Observ. date=07-JUN-2008 Map date=26-MAR-2009 AIPS 1: # visibilities 215400 Sort order TB AIPS 1: Rand axes: UU-L VV-L WW-L BASELINE TIME1 SOURCE FREQSEL AIPS 1: WEIGHT SCALE AIPS 1: ---------------------------------------------------------------- AIPS 1: Type Pixels Coord value at Pixel Coord incr Rotat AIPS 1: COMPLEX 1 1.0000000E+00 1.00 1.0000000E+00 0.00 AIPS 1: STOKES 2 -1.0000000E+00 1.00 -1.0000000E+00 0.00 AIPS 1: FREQ 65 2.2260000E+09 1.00 2.5000000E+05 0.00 AIPS 1: IF 4 1.0000000E+00 1.00 1.0000000E+00 0.00 AIPS 1: RA 1 14 27 56.297 1.00 3600.000 0.00 AIPS 1: DEC 1 -42 06 19.427 1.00 3600.000 0.00 AIPS 1: ---------------------------------------------------------------- AIPS 1: Coordinate equinox 2000.00 AIPS 1: Maximum version number of extension files of type HI is 1 AIPS 1: Maximum version number of extension files of type FQ is 1 AIPS 1: Maximum version number of extension files of type SU is 1 AIPS 1: Maximum version number of extension files of type AN is 1 AIPS 1: Maximum version number of extension files of type NX is 1 AIPS 1: Maximum version number of extension files of type CL is 1
In general the data should appear with sort order 'TB'. (In case not, it may be necessary to run UVSRT or MSORT, followed by INDXR, but why are the data out of order?)
Sometimes there will be more than one output data file for a given experiment. It may make sense to load them into a single file (e.g. for different time-ranges correlated with the same parameters), or if different correlation parameters were used (e.g. a spectral line and a continuum pass), load each pass separately.
task 'uvflg' getn 1 intext 'PWD:<expcode>.uvflg $file with flagging commands inp $check inputs go
Delete edge channels?
Summary of (u,v) data points per baseline for entire experiment:
task 'dtsum' docrt -1 $ print to file (prints to screen if docrt >1) aparm 0 $ set all aparm values to 0 outprint 'PWD:<expcode>.DTSUM.txt $ specify filename inp $ check inputs go
This can be useful for checking antenna numbers.
Note that if flag table has not been applied, visibilities will be present for all stations for the whole experiment, whether the station is off-source/recording or not.
task 'listr' optype 'scan' outprint 'PWD:<expcode>.SCAN.txt $ specify filename inp $ check inputs go
These will indicate e.g. RFI at particular stations.
task 'possm' stokes 'half' $ =RR,LL pols aparm 0 aparm(7) 1 $ Label x-axis with frequency aparm(8) 1 $ to plot total power aparm(9) 1 $ plot IFs together solint -1 $ average over each scan nplots 9 $ 9 plots per page (the maximum) dotv -1 $ plot to file inp $ check inputs go
Note: this can result in a lot of output, especially for experiments with a large number of scans!
imh will show the number of 'PL' (plot) type extension files produced.
Print the plots to a postscript file:
task 'lwpla' plver 1 inver 9999 $ >= number of plots outfile 'PWD:<expcode>_POSSM_AUTOCORR.ps inp $ check inputs go
Check the output file: in a terminal window outside of AIPS, 'gv <expcode>_POSSM_AUTOCORR.ps'
You can then use the verb EXTDEST (
inext 'pl';inver -1;extdest) to delete the PL extension files.
Alternatively, you will have to keep track of the maximum PL version number after each run, in order to set 'plver' each time you run LWPLA to produce output plot files (otherwise you will end up with one really huge postscript file containing all plots).
outfile already exists, LWPLA will append to it, so use a unique file name if you don't want this.
task 'vplot' optype '' $ cross-correlation bchan 10;echan 55 $ discard edge channels (example, for 64-channel data - CHECK how many channels in your data (FREQ in command 'imh')) avgchan 1 $ average channels stokes 'half' $ for dual polarization (RR,LL) solint 0.5 $ averaging time (mins) aparm 0 bparm 0,-1 $ plot amp & phase together nplots 6 inp $ check inputs go
task 'lwpla' plver 1 inver 9999 $ >= number of plots (Can check again after vplot completes, using imh to check PL number - should be small) outfile 'PWD:<expcode>_VPLOT_UNCAL.ps inp $ check inputs go
Note: If flagging has not been applied, this plot will contain off-source data! Also, where a station has NO data, the amplitudes appear as zero but the phase does not - i.e. the phases at such times may look meaningful, but are not! Be warned…
Preferably choose calibrator sources (adverb
sources); these plots will not be very useful for faint targets. Picking which sources are calibrators and which are targets might require some experience. The target source may be obvious from the experiment title. Otherwise it may help to check the scan listing produced with LISTR (observed sources are listed near the end). A typical phase-referencing observation alternates between target and phase-reference calibrator scans.
To check which sources are likely to be calibrators you can try looking for them in the ATCA calibrators catalogue.
Sometimes the target sources will be many and bright (e.g. for experiments such as v252); in that case just leave the source selection blank.
You may wish to limit the output by setting
antennas to plot baselines to a chosen reference antenna (or two).
The output of DTSUM shows the antenna numbers (or use task PRTAN).
task 'possm' sources '1424-418','J1826-3650','1921-293' $ example source selection stokes 'half' $ May wish to plot cross-hand pols 'RL,LR' (if present) to a separate file. bchan 0;echan 0 aparm 1,1,0,0,-180,180,1,0,1 $ plot phase with fixed range -180,180 (see help) solint -1 $ average over each scan nplots 9 $ 9 plots per page (the maximum) dotv -1 $ plot to file inp $ check inputs go
task 'lwpla' plver 1 inver 9999 $ >= number of plots outfile 'PWD:<expcode>_POSSM_UNCAL.ps go
Example inputs. These will need to be specified carefully for each experiment.
Note: Old LBA data in RPFits format, read in with ATLOD, requires the integration time to be set (
dparm(4)). The “correlation notes” on the wiki should include what integration time was used. This does NOT need to be set for data in FITS-IDI format read in with FITLD (default for experiments correlated since January 2010).
task 'fring' calsour '1424-418','J1826-3650','1921-293' $ select calibrators bchan 10;echan 55 $ discard edge channels (in this case there are 64 channels per IF) refant 1 $ reference antenna should be sensitive, and on source most of the time search 3,2,4 $ prioritised list for exhaustive search (not usually needed) solint 2 $ solution interval (minutes) - typical length of the phase-reference calibrator scan. aparm 0 aparm(6)=2 $ prints a lot of information including antenna S/N ratio aparm(9)=1 $ do exhaustive baseline search dparm(4)=2 $ the integration time in seconds - MUST BE SET for LBA data inp go
Note: If you realise you've messed something up, and you need to run
abort, be careful - it doesn't always abort nicely.
If as a consequence you get errors and your task starts dying of “unnatural causes”, see this thread.
Moral of the story: Run
getn 1 to go back to your original file and
clrs to clear any problem statuses.
If like Claire, you aborted because you'd forgotten to reset the pl or sn files, best option is to let it run, it seems, but DO remember to clear them before you run your task again :)
ucat will show the READ/WRITE status flag. If you see this and there's no task running, then it's probably a spurious flag from an aborted task (do
task 'snplt' inex 'sn' $ plot the 'SN' (solution) table inver 0 sources '' nplots 8 optype 'dela' $pixrange -5e-8,5e-8 $ example to limit displayed y-axis range to +/-50 nanosec (time in seconds) dotv -1 inp go
Delay solutions should be typically within tens of nanoseconds. No unexpected gaps?
Note there may be a few outlying points (due to e.g. 'bad' solution with marginal S/N)
Print delay solutions to a file:
tget lwpla outfile 'PWD:<expcode>_FRING_DELAY.ps inp go
inext 'pl';inver -1;extdest
tget snplt optype 'rate' pixrange 0 inp go
Typically less than a few milliHz?
tget lwpla outfile 'PWD:<expcode>_FRING_RATE.ps inp go
inext 'pl';inver -1;extdest
tget snplt optype 'phas' inp go
Nicely connected on phase-reference calibrator? (May not appear well-connected for observations of many sources spread over the sky)
tget lwpla outfile 'PWD:<expcode>_FRING_PHASE.ps inp go
inext 'pl';inver -1;extdest
Apply fringe-fit solutions from calibrator sources used in FRING (
calsour) to all sources (
sources), creating a new 'CL' table.
task 'clcal' interpol 'ambg' $ uses rates to resolve phase ambiguities (where phase changes by large amount between cal scans) bparm 0 $ for no smoothing inver 1 $ specify SN table to apply gainver 0;gainuse 0 $ apply to highest numbered CL table, write new CL table $ NB - keep account of SN & CL tables - be careful not to apply solutions twice. inp go
tget 'vplot' docal 1 $ apply the CL table (gainuse 0 = highest) inp go
tget lwpla outfile 'PWD:v212a_VPLOT_FRING.ps inp go
inext 'pl';inver -1;extdest
tget possm sources '1424-418','J1826-3650','1921-293' $ example source selection stokes 'half' bchan 0;echan 0 aparm 1,0 $ autoscale phase plots this time to see small deviations aparm(7)=1 $ label x-axis with frequency aparm(9)=1 $ plots IFs together in a row solint -1 $ average over each scan nplots 9 $ 9 plots per page (the maximum) dotv -1 $ plot to file inp $ check inputs docal 1 $ apply FRING solutions inp go
tget lwpla outfile 'PWD:<expcode>_POSSM_FRING.ps inp go
inext 'pl';inver -1;extdest
For the fringe-fitted data, the phases should be close to zero and flat across the band (in the parallel-hand polarizations RR,LL, and where there is sufficient S/N).
task 'uvplt' sour 'SN2004IP'' $ 1 source at a time bchan 32;echan 32 $ plot only 1 channel from each IF to speed things up aparm 0 bparm 6,7,0 $ plot u,v coverage inp $ check inputs go
tget lwpla outfile 'PWD:v212a_<srcnam>_UV.ps inp go
inext 'pl';inver -1;extdest
To do: bandpass calibration, amplitude calibration (ANTAB, APCAL), imaging…
When you've finished with AIPS, exit with the command
kleenex to close all servers.
===== Put plots on the CIRA ftp server =====
(If needed, convert plots to PDF with ps2pdf. NB the pipeline output plots are already in PDF)
Ftp to cira.ivec.org as astronomy (password required).
passive to use passive mode
cd to anonymous/<expcode>
Edit the “Analysis notes and links to plots” section on http://cira.ivec.org/dokuwiki/doku.php/correlator/records/<expcode>
Insert links to the plots in the ftp area (replace all “expcode” with the actual experiment code) and comments as required.
There is now a script
~corrtest/progs/Pypeline/lba_feedback.py to produce the wiki page after running the pipeline.
The verb ALLDEST can be used to delete all AIPS files for a user number, once the data have been finished with, e.g.
inname '' inclass '' inseq 0 intype '' alldest
When all data have been deleted, ALLDEST will also ask to destroy all SAVE/GET and TGET files.
FREESPAC tells how much space is available on the AIPS disk.