ATLOD: Reading RPFITS Files

ATCA data will be initially in RPFITS format. This needs to be converted to Miriad's format, by task atlod, before any further processing can be done. We will discuss the various input parameters to Miriad atlod.

• The in parameter gives the name of the input UNIX file, or files, to load. Many files can be given, and wildcards are supported.

  Almost all ATCA RPFITS files can now be downloaded directly from the AT Online Archive (ATOA). Any requested RPFITS files can be saved onto your local disk, and atlod can be directed to load this data from there.

  RPFITS files from the old correlator are commonly archived on CDROM or DVD: a CDROM/DVD appears much like a disk to the operating system, and atlod can load RPFITS files directly off these. To use a CDROM/DVD drive, first check that the drive is not in use (an orange light flashing). To load a disc, press the Open/Close on the front of the drive, place the disc in and again press the Open/Close button. After about 10 seconds, the operating system will mount the CDROM/DVD, and it can be accessed. CDROM/DVDs written at the ATCA will have the RPFITS data in the directory

  /media/cdrom/DATA

  You can do a directory listing to see what you have - its like a normal file system!

  You can set the in parameter to the RPFITS files in this directory. For example, to select all files belonging to project C561, use

    in=/media/cdrom/DATA/*.C561
  

  When you are done with a CDROM/DVD drive you eject it with the command

     eject cdrom
  

  (the Open/Close button will not work).

  RPFITS files are (or at least were once) commonly written to exabyte tapes. Unlike other Miriad tasks, atlod can read exabytes directly. In this case you should use the physical device name as the input name. This is usually displayed on the exabyte drive. On the Epping Solaris systems, the device name is usually /dev/rmt/0lbn or similar (this is the so-called non-rewinding, raw, BSD compatibility interface). On older Solaris systems, it may be /dev/nrst4 or something similar (non-rewinding, raw interface).

  An alternative to directly reading from exabyte is to load the data to disk with the Unix commands ansiread or ansitape, which are available on a number of the ATNF systems. See the UNIX man pages for more information.

• The out parameter gives the name of the output Miriad visibility dataset. There is no default.

• The ifsel parameter makes atlod extract only the IFs that are selected here from the RPFITS files. The IFs to extract should be specified as a comma-separated list. This can be very useful if you are interested only in a specific frequency, or if the data has two or more IFs with different numbers of channels that would normally need to be read in with options=noif.

• The options parameter gives miscellaneous processing options. Several values can be given, separated by commas.

  Some of the options described below control on-the-fly corrections that atlod makes to the data, while others can reduce the amount of disk space that the dataset will consume. It is very important therefore to pass the correct options to atlod, to save yourself time later doing corrections that could have been done more easily right at the start.

  At frequencies above 10 GHz, the opacity of the atmosphere can be significant: in mediocre conditions, the signal loss can be 20% at low elevations. With high-frequency data, an atmospheric opacity correction should be applied to the data; atlod will perform this correction if it is passed the opcorr option. This option can be safely included for data of any wavelength, as it does very little (and certainly does not harm the data) at low frequencies, and is completely ignored at 3mm wavelengths, where it is not appropriate.

  For high frequency data taken with the old ATCA correlator, it is also generally desirable to load both observing bands together, to allow joint calibration. A shortcoming of Miriad is that it is poor at handling multiple observing bands when the polarisation products differ between the bands. However the old ATCA correlator configurations that combined narrowband and wideband observations generally did measure different polarisation products in the two bands. As polarimetry with the ATCA 3-mm system was not possible until recently, the simplest approach to ensuring the polarisation products of the two bands are the same is by discarding polarisation products that are not needed. The atlod option nopol causes atlod to discard any XY and YX products.

  A few sets of permutations of the options will be appropriate with most observations. These include:

  • For CABB data: options=birdie,rfiflag,xycorr,opcorr
  • For old-correlator continuum: options=birdie,xycorr or options=birdie,reweight,xycorr. See below for a discussion on whether to use reweight or not.
  • For old-correlator non-polarimetric line: options=birdie or options=birdie,compress or options=birdie,hanning or options=birdie,compress,hanning. See Section 16.3 for more on using atlod with spectral line data and Section 24.2 for high time resolution bin-mode data.
  • For old-correlator 3mm data: options=birdie,nopol,reweight

  Possible options are:

  birdie:

  The old ATCA correlator suffered from self-interference at frequencies that were a multiple of 128 MHz, while the CABB correlator displays self-interference spikes at frequencies related to the 640 MHz sampler clock. The birdie option flags out the channels affected by this self-interference. This option is strongly recommended.

  Additionally, in the 33 channel/128 MHz mode, the birdie option also discards some edge channels and every other channel. This operation does not incur a sensitivity penalty, as correlator channels are not independent in this mode. The net result is that the output consists of either 13 or 14 good channels.

  For CABB data, this option also flags 100 channels on each edge of the band, and any part of the 20cm or 13cm band that is not within the recognised frequency range of the receiver.

  xycorr:

  In polarimetric correlator configurations, the ATCA makes an on-line measurement of the phase difference between the X and Y channels. Although it is generally only a few degrees (assuming the observation was correctly set-up), this phase difference should be corrected when doing polarimetric work. Since a hardware upgrade in November 1992, the recommendation has been that the on-line measurements should be applied, without averaging, using the xycorr option in atlod. For data prior to November 1992 - seek expert advise.

  reweight:

  In the 33 channel/128 MHz mode, a Gibbs phenomena (see Albert Bos in the Indirect Imaging proceedings) affects the ATCA. This introduces a non-closing error into the data. The effect is moderately subtle, and is not significant for dynamic ranges of less than about 500. The `reweight' option reweights the visibility spectrum in the lag domain to eliminate this problem.

  This option is recommended for high dynamic range work (more than about 500). However, the option also reduces the effectiveness of the birdie option to reject self-interference. This can be a significant effect for 20cm continuum observations, where the usual observing band straddles the 1408 MHz birdie. Particularly at 20cm, the reweight option is not recommended where dynamic ranges of less than 500 are expected.

  This option is not required for CABB data.

  compress:

  Normally the correlation data are stored as 32-bit floating point numbers. Alternatively, the compress switch can be used to instruct atlod to store the correlations as 16-bit integers, with a scale factor associated with each spectrum. This approximately halves the disk space occupied by a dataset, and so may be very advantageous for large spectral-line observations.

  hanning:

  Hanning smooth the data, and discard every second channel. This can be useful for spectral line experiments. Task atlod will refuse to Hanning smooth data with 33 channels or less.

  This is unlikely to be beneficial for CABB data, as each CABB channel is independent, and boxcar smoothing (or channel binning) is preferred.

  bary:

  Use the barycentre (often called heliocentre and usually used for extra-galactic work) as the rest frame when calculating velocity information. The default is to compute velocity information relative to the LSR frame (which is usually used for Galactic work).

  hires

  This option may be appropriate for observations using the ATCA's high time resolution bin mode. See Chapter 24 for more information.

  opcorr:

  This option corrects the visibility data for atmospheric opacity. It is since October 2003 that there is sufficient information in the visibility dataset to allow this. This options should not be used for 3mm data. Generally it is relevant for 12mm observations only. See Sections 22.1 and 23.2.2 for more information.

  samcorr:

  Correct the measured visibilities for bad sampler statistics. Normally the samplers adjust their various threshold levels to ensure optimal operation. However various transients can cause the sampler levels to be wrong. All is not lost - you can correct the data for the imperfect sampler levels after the fact. Use the samcorr (sampler-correct) option for this, and is the recommended approach for data observed before December 1993. Since December 1993, sampler correction has been done automatically on-line. For this data, the samcorr option is (quietly) ignored.

  relax:

  This option is generally not recommended. Normally atlod flags data if it has not previously read the appropriate calibration record or if the information in the calibration record suggest the data are bad. Although calibration records generally precede the data, there are some instances when this is not the case, and the data are still good. In this case, the relax option causes atlod to be more lenient, and not flag the data.

  unflag:

  Normally atlod will discard a visibility record (i.e. not write it to disk) if all the data in it are flagged bad. Flagging usually indicates that there was significant reason to suspect the data - for example the telescope was not on source or was off-line, etc. The unflag option causes all data to be saved to disk, although flagged data is still marked as such. This option is generally not recommended, although the disk-space penalty for using it is often modest.

  noauto

  and nocross cause atlod to discard any autocorrelation or cross-correlation data respectively. It's fairly unlikely that you will want to discard the cross-correlation data!

  noif:

  When there are multiple frequency bands being observed simultaneously, Miriad normally maps the frequency bands to its spectral windows (IF axis in AIPS terminology), and writes them out in a single record. Alternatively, using the noif switch causes atlod to write the simultaneous frequencies as sequential records, making it appear somewhat like a frequency-switch. You will need to use the noif option if you have two spectral windows which sample different polarization parameters, or that have different numbers of channels.

  nopflag:

  If at least one polarization of a set of 2 or 4 polarimetric spectra are bad, atlod normally flags all of the polarizations. With this option only the nominally bad spectrum is flagged.

  caldata:

  atlod will usually mark as bad any scans that are labelled as a reference pointing calibration or a paddle measurement. Normally, this data is of little use as corrections from these scans are applied on-line. However, occasionally, the data in these scans may be of use, and including this option causes atlod to keep this data instead of discarding it, although it will still be flagged as bad.

  rfiflag:

  At low frequencies, the large bandwidth of the CABB correlator makes it impossible to avoid most of the known RFI. Including this option causes atlod to automatically flag regions of known interference as bad. It is also possible to specify flagging regions manually using the file rfiflag.txt in the current directory, or the default version in the MIRCAT directory. Each line in this file should have two numbers representing the lower and upper frequencies to flag in MHz.

• nscans gives two numbers which are the number of scans to skip over (before processing), and the number of scans to save. The default is to save all scans.

• nfiles, like nscans, gives two numbers, which are the number of files to skip and process. This parameter is only really useful when reading from exabyte. The default is to process the first file only. Note, however, that the mechanism used to skip files is rather inefficient. If you wish to skip a large number of files, you should do this with the Unix command mt. A complication is that every RPFITS file appears as three files to mt - so you will want to skip three times as many tape files as RPFITS files. For example, to skip 10 RPFITS files, you would use the Unix command

     mt -f /dev/nrst4 fsf 30
  

  See the Unix man page on mt for more information.

Typical inputs for atlod are given below.

ATLOD
in=/dev/nrst0	Input is either exabyte name
in=2010-02-04_1844.C007	or RPFITS file.
out=0823.uv	Output visibility data set.
options=birdie,rfiflag,xycorr,opcorr	Normal CABB data options, or
options=birdie,reweight,xycorr	Normal old-correlator continuum mode options, or
options=birdie,hanning,compress	possible old-correlator spectral line options.
nfiles=0,3	Skip 0, then read 3 files (use only for tape reading).
nscans	Unset to save all scans

Miriad's atlod saves a number of on-line measurements as visibility variables. These measurements may by helpful in analysing and flagging the data. They can be plotted and listed with task varplt (see Section 10.7). These on-line parameter, and its Miriad variable name, are described below.

xyphase:

This is the on-line measurement of the XY phase. It is important to examine this measurement if you are doing polarimetry.

xyamp:

This is the amplitude of the correlation between the X and Y polarization channel of a given antenna.

xsampler, ysampler:

These variables give the sampler statistics for the X and Y polarization channels. There are three numbers per antenna per IF, which reflect the sampler levels. They should have values of 17.3, 50 and 17.3. Thus for six antennas, the xsampler variable will consist of 18 numbers per IF. These values are not available for CABB data.

xtsys, ytsys:

These give the system temperature, in Kelvin, for the X and Y polarization channels. Unfortunately Miriad tasks were originally developed around a single polarization model of a telescope, and so most tasks that concern themselves with system temperature do not handle dual polarization systems. The partial workaround used in Miriad has been to store the geometric mean of the X and Y system temperatures in the variable systemp. It is this variable that Miriad tasks use when they need to use the system temperature value.

axisrms, axismax:

These give the antenna rms and maximum tracking errors in a particular cycle. The units are arcseconds.

airtemp,pressmb,relhumid,wind,winddir:

These give various meteorological measurements at the observatory as a function of time.

smonrms:

This gives a measure indicative of the seeing conditions at mm wavelengths, and is obtained by a two-element interferometer observing a communications satellite at 30 GHz. This number is the RMS of the difference between the expected path length difference and that observed, and is due to the atmospheric conditions above the Narrabri site. It may not however directly represent the atmosphere along the line-of-sight to the object currently being observed. A smaller number here means a more stable atmosphere, and better conditions for mm observing.