Mirflag/Plotvis

 
 

Before diving into the gritty details, my lawyers have asked me to remind everyone that Mirflag and Plotvis are still in the “beta” development stage and have not undergone thorough testing with a large number of data sets or on a large number of different platforms. As such, it is suggested that you make a backup of your data set before applying Mirflag to it. At the very least, keep a backup of the flags file (which can be found in the MIRIAD data set folder). After applying Mirflag, please ensure that you check that the appropriate data has been flagged using tools such as Plotvis and other MIRIAD data visualisation tools before proceeding with your data reduction. Note that while it is also experimental, Plotvis is a visualisation-only tool and is thus fairly benign and should NOT affect your data set.


Let’s face it - flagging CABB data is a pain! Wouldn’t it be nice if it could somehow be automated? Well, now it is :-) Mirflag is a MIRIAD-like task that allows automated flagging of a UV data set. It is closely based on Enno Middelberg’s Pieflag but has been extensively re-written to optimise for speed and the larger bandwidths available with CABB. The task scans UV data on a baseline-dependent and channel-dependent basis to find bad data and flag them accordingly. Four flagging modes may be specified via options amp, rms, short and long.


Amplitude-based flagging is performed when the amp option is supplied. On a per-baseline basis the median amplitude for a designated "good" channel is determined over the course of the observation, this is the channel-median. An effective noise measure for the "good" channel is made by taking the median of the absolute difference between channel-median and the channel amplitude throughout the course of the observation, this is the channel-diff-median. The "badness" of an amplitude measure in another channel is determined by how far it deviates from the channel-median with respect to the channel-diff-median and a user defined cutoff scaling. This provides a number of threshold points:


     -2*thresh    -thresh    Median    +thresh   +2*thresh

<---------+----------+----------+----------+----------+--------->

Probably | Possibly | Probably | Probably | Possibly | Probably

   Bad    |   Bad    |    OK    |    OK    |   Bad    |   Bad


Where the threshold is defined as cutoff * channel-diff-median. Visibilities marked as "Probably Bad" will be flagged immediately. NOTE: To apply amplitude-based flagging on small clusters visibilities marked as "Possibly Bad" the "short" option must also be provided.


RMS-based flagging is performed when the rms option is supplied. In this task the amplitude RMS is measured on a per-baseline and per-scan basis for a designated "good" channel (Note: a scan will be split up if the length of the scan exceeds the value specified by the gap parameter). The median of these RMS values is taken as a reference. If the RMS of another channel (for a particular scan/baseline) exceeds the reference RMS by more than a factor of rmscut then the entire scan is flagged.


Short cluster flagging, as specified with the short option, flags short (±60 s) sections around bad points.


Large cluster flagging, as specified with the long option, searches for and flags larger clusters of bad points using a running sum window with length specified by the int parameter. The flagging threshold is specified with the thres parameter.


Key: vis

  1. The name of the input UV data set.  A visibility file name must be supplied.  Only one file may be edited at a time.


Key: stokes

  1. Specifies which stokes parameter/polarisation to read and analyse (first value) and which polarisations to flag (subsequent values). For example, one can read and analyse rr and use the results to flag both rr and ll by specifying stokes rr,rr,ll. Defaults to read Stokes I and to flag all available polarisations.


Key: gap

  1. Used in conjunction with the rms option. Time in seconds which defines a scan gap (source changes also define scans). Defaults to 150 s.


Key: rmscut

  1. Used in conjunction with the rms option. Factor by which the RFI-free channel's scan-by-scan rms is to be multiplied and above which data is considered bad. Defaults to 3.


Key: goodch

  1. Number of a good (relatively RFI-free) channel to which the other channels are to be compared. By default, Mirflag will attempt to find the channel with the lowest median noise.


Key: cutoff

  1. Used in conjunction with the amp option. Multiples of median of difference to median in data, above which data are suspicious. Will be multiplied by two to obtain a threshold above which data are certainly bad. Defaults to 7.


Key: int

  1. Used in conjunction with the long option. Width of running sum window in seconds in which to calculate fraction of flagged visibilities. 1200 is reasonable for weak sources, 600 for calibrator observations. Defaults to 1200.


Key: thres

  1. Used in conjunction with the long option. Fraction of maximum of running sum, defines threshold above which data are deemed bad in search for large clusters of bad points. Defaults to 0.15.


Key: options

  1. Extra processing options. Several can be given, separated by commas. Possible values are:


  1. ignore

  2. Ignore existing flags in data set, default is to read them.

  3. replace

  4. Replace existing flags in the data set with new ones, default is to add to existing flags.

  5. noapply

  6. Run only the analysis and provide a summary of the flagging but do not flag the data.

  7. amp

  8. Switch on amplitude-based flagging (used in combination with the short option).

  9. rms

  10. Switch on rms-based flagging.

  11. short

  12. Flag short (±60 s) sections around bad points. Without this option amplitude-based flagging is effectively turned off because amplitude-based flagging does not generate flags directly.

  13. long

  14. Search for larger clusters of points.

  15. medsed

  16. Subtract channel median to remove effects of a residual SED or poor bandpass calibration.

  17. medtcal

  18. Subtract bandpass median at each time-stamp to remove effects of poor secondary calibration.




Plotvis is a tool to graphically visualise a MIRIAD data set in three dimensions: amplitude/phase versus channel and time. The tool utilises the s2plot library developed by the Centre for Astrophysics & Supercomputing at Swinburne University.


Key: vis

  1. The name of the input UV data set.  A visibility file name must be supplied.  Only one file may be viewed at a time.


Key: stokes

  1. select the stokes parameter to visualise e.g. stokes=xx to select stokes xx.


Key: device

  1. select the s2 device to use e.g. device=/S2MONOF for full screen viewing. By default the /S2MONO device is used i.e. visualisation is performed within a window.




Binaries for Mirflag and Plotvis are provided in the download section to the right. Note: to run Plotvis an opengl capable machine is required. For LINUX machines, the libglut library must be installed before Plotvis will work.


Source code for both Mirflag and Plotvis are provided, however, building these tools is a rather messy affair and is not recommended for the faint-hearted. The tools require an up-to-date installation of MIRIAD and s2plot version 2.4. For 64-bit Linux compilations, some modification to the Makefile may be required depending on the flavour of Linux being used.

 

Mirflag/Plotvis

DOWNLOADS

Last updated 15 Jul 2012

Version 0.1.3



Mirflag/Plotvis Source Code:


mirflag_0.1.3.tgz



MAC Binary:


mirflag_darwin_x86.tgz



LINUX Binary:


mirflag_linux.tgz



Windows Binary:


ROFL! You’re joking right?