User Tools

Site Tools


correlator:correlation:cira-difx-1.0

Setting up for correlation

  • The contact author should have been emailed regarding their desired correlation parameters (ask Hayley).
  • From the VLBI Observers wiki at ATNF, find out which shipped disks were recorded and make sure they are loaded. For data transferred electronically, find out which local disks/Xraids they live on, or transfer from iVEC's PB store.
  • Also need to know which ATCA station pad was used as the reference. This should be recorded in the ATCA experiment log at the beginning of the session and shouldn't change during the session - but check the wiki. Also can be checked in MoniCA (still to be installed) - which logs ATCA monitoring data.
  • Log in to cuppa as corr and make a working area for the experiment: /home/corr/LBA/<year>/<session>/<expcode>
  • Keep notes of setup and any problems encountered, e.g. in a file called <expcode>_notes.txt. Lost time or other problems should also be recorded here on the wiki.
  • Get the vex file <expcode.skd> from the ATNF ftp area (linked from the observers wiki)
  • We take source coordinates from this vex file. Occasionally the PI may supply updated coordinates for correlation - edit in vex file if necessary.
  • Run vex2model_updated.pl <expcode.skd> to produce the delay and uvw files ('Y' to take sources from the vex file).
  • The vex file may need editing to reflect some stations not recording all channels. For experiments at 512 Mbps, typically Ceduna and Hobart record half of the channels. Sometimes stations record with swapped polarizations, this can also be edited in the vex file. (If preferred, it is also possible to edit the correlator input file datastream and baseline tables. See the explanation of the correlator input file format.)
  • Also noted on the ATNF wiki is what to do in case of band inversion. This is an issue for 64 MHz recording modes.
  • Generate a list of data files. To list all files for an experiment on each Xraid, ls -1 /exports/xraid0?/?_?/<expcode>/<expcode>_* » files.list and if they are on a local disk, append this with e.g. ls -1 /arch/corr/<expcode>/<expcode>_* » files.list
  • There is an IDL script in /home/corr/LBA/scripts/filelist.pro which will take the list of data files and produce output in the correlator input file format (however note: filelist.pro will require editing and may not always be useful - assumes some specific naming conventions). Copy filelist.pro to the working directory for the current experiment and edit the stations list and start and stop times. In my limited experience so far, the correlator (version 1.0) sequentially opens all files listed in the input file, regardless of whether they are inside or outside the time range specified for correlation. So if you have a long list of files and only want to correlate a short time range (say, for fringe-finding or clock checking) it is much more efficient to specify only the files within that time range. Similarly some stations may keep recording beyond the end of the experiment, so the correlator will encounter this and spit out a whole lot of messages at the end, when it could be finished already! Because of this I usually specify the timerange in filelist.pro. This works for LBA data where the file names contain the start time - note that Mk5 data e.g. from Hart is labelled scan-by-scan, not currently implemented in filelist.pro.
  • Run filelist.pro using GDL (the “Gnu data language”, a free IDL interpreter):
$ gdl
GDL> .r filelist
GDL> exit

filelist.pro creates an output file called datafiles.list (or whatever you prefer to call it) in input file format.

  • vex2config.pl generates the correlator control file. Run with no inputs it will show a list of options. e.g. -files datafiles.list gives it the list of files to correlate. See experiment-specific notes for examples. We generally output all 4 polarization products (-crosspol option)
  • Edit the output file name & location, if necessary
  • Edit the TELESCOPE TABLE to reflect current best clock estimates (e.g. from another experiment close in time). Different IFs can have different clock offsets as well - this is specified in the DATASTREAM TABLE. (However, different polarizations cannot have offset clocks, in the current version) When I have good clock estimates I create a file with only the telescope and datastream tables, called e.g. CLOCKS.input, and each time vex2config.pl is run, I replace these sections of the new input file with the updated CLOCKS.input. Not very clever, but hopefully we don't need to do it very many times for each experiment.
  • For spectral line correlation, often the BASELINE TABLE will need editing as it is only desirable to correlate one IF with high spectral resolution. In this case, 'NUM FREQS X: N' should be changed to 'NUM FREQS X: 1' (where X refers to the baseline and N is the original number of frequency products assuming correlation of all data). Then delete the unwanted 'D/STREAM ..' and 'POL PRODUCTS ..' lines, being careful that the remaining 'D/STREAM ..' lines correspond to the correct IF. For an example, compare /home/corr/LBA/2008/November/v276b/v276b.line.input with v276b.cont.input in the same directory.
  • Create the threads file (usually called <expcode>.threads), machines file, and run file. Alternative to shell script run file: use startcorr.pl
  • Run the correlator! Pipe standard output and error messages to a log file, e.g. run >& <expcode>.input.log. For running long jobs use the screen program (man screen for details - or use VNC), then e.g. you can detach & reattach to check on progress later.
correlator/correlation/cira-difx-1.0.txt · Last modified: 2010/01/20 21:01 by hbignall