Next: How Pieflag works Up: Automated Editing of Radio Previous: Automated Editing of Radio

Introduction

Data editing, to rid data sets from measurements which are obviously wrong (for whatever reason) or affected by external factors, is a widely accepted practice in all branches of science. Radio astronomical measurements, in particular at frequencies below 5GHz, can be severely affected by emission from man-made transmitters, which is then known as radio frequency interference (RFI). Only a few attempts have been made as yet to automate the editing of radio astronomical observations, an example includes the ``WSRT flagger'' used at the Westerbork telescope¹. However, it is recognized that powerful RFI mitigation techniques need to be developed for future instruments such as the Square Kilometre Array (SKA) (Ellingson, 2004).

In radio interferometry, data editing is commonly known as ``flagging''. RFI-affected data are not discarded but an entry (a ``flag'') is made in a computer file, to indicate that the data are not to be used in further processing. When large amounts of data have to be handled, data flagging can be very laborious and dull. This can lead to situations in which one either flags too many or too few data, sacrificing good data in the first case, or accepting a degradation of the quality of the observations in the latter.

The automation of flagging requires the creation of a measure that indicates what is ``good'' data, and then the observed data needs to be compared to this measure. Both of these requirements have to work in a variety of circumstances, and assumptions which may work well in one case can be invalid in another.

In Pieflag², two algorithms are implemented to detect RFI-affected data made under a variety of circumstances. Two steps of postprocessing extrapolate the flags which were created by Pieflag to points close in time. This yields a safety margin around RFI-affected data and picks up those data which are likely to be affected as well, but have not been found by the finding algorithms.

Pieflag has been designed to read data processed with Miriad (Sault et al., 1995), which is the standard data reduction package for observations made with the Australia Telescope Compact Array (ATCA). However, it will work on data from other interferomters as well, when these are converted into the Miriad format. I note that the algorithms presented here are not specific to interferometric observations. Time series of single-dish radio telescope data could be analysed in the same fashion.

I will first describe the details of the algorithms used to detect RFI and to generate flags (Section 2), then the display of the results and how the flags are applied to data sets (Section 3 and 4), followed by an assessment of the limitations (Section 5), and a description of usage (Section 8) and some examples (Section 6).

Next: How Pieflag works Up: Automated Editing of Radio Previous: Automated Editing of Radio