Averaged continuum (channel 0) data

                                               

Consider now the editing of a continuum data set in which you have averaged each band to one channel (channel 0) as discussed in the previous section. Inspection of the data is now fairly straightforward.

1. First, it is good to get an overall feel for the data; do they appear to be of high quality, or are there sections of bad data etc. The best way to do this is to make some plots. There are two main plotting tasks that you might use: UVPLT and VBPLT. Unlike all other AIPS tasks, UVPLT can plot all freqids together. VBPLT has the capability to put multiple plots on the one page, one for each baseline.

   With UVPLT, the sorts of plots that are initially useful are plots of all the visibility amplitudes and/or phases from all baselines as a function of both time and uv-distance (). Use the bparm adverbs to control these quantities. When you plot amplitude versus time, include all sources, as the time axis distinguishes sources. When you plot amplitude or phase versus uv-distance, do them one source at a time, or you will get all the sources jumbled up on top of each other. When plotting time with UVPLT, I suggest you use time in hours (bparm(1) =11) rather than the aesthetically displeasing decimal days. The adverb xinc is also very useful; it lets you plot every xincth point. This is very handy when you have large data bases. If the data are sorted in time order, make sure xinc does not divide exactly into the number of baselines. Otherwise, you will see the same baselines over and over. This is not a problem with data sorted in spacing order.

   sources=' ' Plot all sources

   calcode=' ' Plot all sources

   qual=-1 Plot all sources

   stokes='I' Plot Stokes I or plot

   stokes='RR' RR or LL if polarization

   stokes='LL' trickery (really XX,YY) invoked

   freqid=2 Select frequency id

   uvrange=0 Select full uv range

   docalib=-1 No calibration

   flagver=0 Read highest version

   xinc=1 Plot every xinc'th visibility (if you have a lot you may want to make this bigger)

   bparm=11,1,0 Plot time (x-axis) v amplitude (y-axis) and self scale plot

   bparm=0 Plot uv-distance v amplitude and self scale remember to select one source only in this case

   For calibrators, you should see a roughly constant amplitude with time or uv-distance and some scatter determined by the noise. You might see a gain jump, where suddenly a source gets brighter or fainter by tens of per cent. However, this is not necessarily a problem and should not be marked as ready for the grim reaper just yet, because you can probably calibrate it out. It is really no different from a slow gain change except that the time scale is a bit shorter. You will have to be a little careful in the calibration procedure though, to deal with this correctly, and I will discuss what to do in the appropriate section. Such a gain jump is most likely to occur because you reran the on-line calibration program (CACAL or DELCOR) for some reason (not that this is generally reccommended). It is extremely unlikely that the receivers themselves underwent a gain jump.

2. You may notice from your plots that data are often corrupt at the beginnings of scans on all baselines (this problem is most common in ATCA data taken in the first couple of years of operation). These problems usually follow a change of source or frequency, or both, as sometimes the system takes some (short) time to settle down when reconfigured. There is a specialized task to deal with this. It has the unlikely name of QUACK; I presume you are `doctoring' the data. It enables you to flag a specified amount of time at the beginning of all scans in a specified time range in a simple fashion. Look out for the confusing definition of opcode='end'.

   sources='2512-92',' ' Select source to flag

   timerang=1,23,10,0, 2,0,10,0 Select desired time range to QUACK over In this case, examine data from day 1 23 hours, 10 min and 0 sec to day 2, 0 hour 10 min and 0 sec

   flagver=3 0 means 1 here. Specify if you want something other than one.

   opcode='beg' Flag first aparm(2) minutes of scan

   opcode='end' Flag up to aparm(1) minutes from end of scan

   reason='bad AGC' Write reason into flag table too. May be used by UVFLG to easily unflag data

   aparm=0.5,0.5 Specify times to flag in minutes as specified by opcode

3. Now, having plotted and QUACKed the data, it is time to clean it up in earnest, if required. The task best suited to this is TVFLG, which displays the data on a TV device as baseline in the x-direction and time in the y-direction. The intensity of the image displayed might be amplitude, or phase, or amplitude difference from the running mean and so on. Thus all your data at one frequency can be clearly displayed at one time. You can read the EXPLAIN file to TVFLG to find out exactly what it can do. However, TVFLG is a little daunting at first, so feel free to seek help straight away with it. It is easier to learn how to drive it by demonstration than by reading. But it is worth learning. Note also that TVFLG has good on-line help for every command. Usually, you can start it up in default mode, taking care to specify the averaging time equal to the integration time of your observation (15 seconds or the like). TVFLG\ will now display the visibility amplitudes, and it may do some averaging in time to fit it all on the TV.

   Now, with the menu and the mouse select (two steps - first click on desired box with the mouse and then hit button A at the top of the workstation keyboard) to display amplitude differences from a running mean. Then select the number of integrations over which to form the running mean (say 20 or so) and select to reload the image. TVFLG\ will then display the new quantity. Now comes the real power of such a task. Sporadic bad data will stand out very obviously as a large amplitude difference. You can then flag these bad data out with the cursor, selecting by pixel, by area, by time or by baseline and so on. This is a very fast way to clean up obvious garbage.

   Experiment with the various options of TVFLG to become comfortable with it. For example, the clip option is also very useful. This option can be used to clip say, on an amplitude or an amplitude difference image. You specify the range outside of which data are clipped and flagged as bad. In this way, TVFLG replaces the functionality of the CLIP task which is not useful for multi-source tasks. Make sure you do not iteratively clip your data, at each iteration apparently reducing the rms. Eventually, after you have done this enough times you will have clipped away all of your data. Just do it once to get rid of the really bad outliers.

   When examining your program source, you might find the option to order the baselines by increasing length helpful, because for sources that are extended, you expect the visibility amplitude to increase with decreasing spacing (the unmeasured zero-spacing is the total flux of the source), and this helps distinguish genuine source structure from errors. However, again, I suggest that you concentrate largely on the calibrators at this time.

   TVFLG doesn't actually write to the FG table until you exit, and then you have the option to write to the FG table. In addition, it writes (and optionally catalogues) a master image where the data have been gridded (TVFLG\ uses this image to keep track of source names, frequencies, times, baseline numbers, and if samples have been flagged already). You can restart TVFLG and have it read this file in, rather than have it recompute it, which may take a significant amount of time. This option is useful only if you are starting up TVFLG to do some more editing with exactly the same inputs. Otherwise, TVFLG will need to make a new grid.

   Note that it is meaningless to display Stokes I from a data base that you have loaded as polarizations (i.e., no Stokes conversion), and which has not had an on-line (done with observing program delcor) or off-line calibration made or has not had the XY phase difference applied to the Y gains (see § 4.1). This is because XX and YY (called RR and LL with the polarization kludge) must be summed to make I, and they must be added up in phase.

   If your source is polarized, it is important that when you flag one polarization as bad (e.g., RR=XX) you flag them all. This is because you will form Stokes I=XX+YY, and it is not true that XX=I or that YY=I when the source is polarized. You can interactively set the Stokes flagging mask with TVFLG by selecting the `SET STOKES FLAG' option and typing in the mask in the AIPS window. Each bit in the mask corresponds to a polarization type. Thus, mask=1011 means flag XX, XY and YX, but not YY. Mask=1111 means flag all polarizations. This is what I recommend you use if the source is polarized. You might use this mask even if the source is unpolarized, working on the assumption that data which are bad in XX are likely to be bad in YY too. This is often a good assumption and may save you some time in the editing procedure.

   docat=1 Catalogue master work file

   in2seq=0 Create new master work file or

   in2seq=2 Select old master work file with sequence number 2

   sources=' ' Select all sources

   timerang=0 Select all times

   stokes='I' Display Stokes I or

   stokes='RR' RR or LL if polarization

   stokes= 'LL' trickery (really XX,YY) invoked

   freqid=1 Select desired frequency id

   antennas=0 Select all baselines baseline=0

   docalib=-1 No calibration

   flagver=0 Write to highest version of FG table already present or make a new one

   doband=-1 No bandpass to apply dparm=0

   dparm(6)=15 Set minimum integration time in seconds

4. Either solely, or in conjunction with TVFLG, you should consider the use of IBLED (Interactive BaseLine Editor). This is a task that allows much finer control over the editing procedure than TVFLG. It provides interactive editing of graphical displays (on a TV device) of either amplitude or phase as a function of time, one baseline and polarization at a time. Since the ATCA has, at most, 15 baselines, most people do have the patience to edit 12 hours of data with IBLED. It allows you to do an extremely careful job of editing. For strong program sources, you can allow your eye and brain to do mean and r.m.s. calculations, so that you can edit them as well as you might edit your calibrators. But remember the warnings about being over zealous with your editing. A point that is just a bit wrong may be more useful retained than discarded.

   IBLED breaks up each full observation into a number of shorter time frames, and plots one at a time, allowing you to step through them sequentially. It also plots all the data points from the full time range at the top of the screen so that you can select a frame at random. This is useful if you have already cleaned a lot of the data up with TVFLG, and just need to do some filagree work. You can step through the baselines sequentially or at random also.

   The actual editing is achieved by placing a vertical cursor on the offending point or by embracing a collection of points with a cursor designated box. Like TVFLG, the editing commands are written into the FG table on exiting from the program.

   The comments about Stokes flagging masks in the discussion on TVFLG above applies equally to IBLED and any other editing task.

   docat=1 Catalogue master work file

   in2seq=0 Create new master work file or

   in2seq=2 Select old master work file with sequence number 2

   sources=' ' Select all sources

   timerang=0 Select all times

   stokes='I' Display Stokes I or

   stokes='RR' RR or LL if polarization

   stokes='LL' p trickery (really XX,YY) invoked

   freqid=1 Select desired frequency id

   antennas=0 Select all baselines baseline=0

   docalib=-1 No calibration

   flagver=0 Write to highest version of FG table already present or make a new one

   doband=-1 No bandpass to apply dparm=0

   dparm(5)=15 Set minimum integration time in seconds

5. There are other more basic tools available for data editing. If you suspect that a particular section of data has some miscreant points in it, you might like to just plot it quickly, rather than wait for IBLED to do so. You can use UVPLT again for this; fine control is given by the timerang (specify time range), antennas (specify antennas only) and baseline (specify actual baselines with the combination of antennas and baseline), and uvrange (specify a uv-range of data) adverbs.

   Another very handy task is UVFND. It allows you to set clip levels, above and below which data are deemed bad and reported to the user. The example shows how to look for bad data on one source for which you expect the flux density to be about 2 Jy on an uncalibrated channel 0 data base. It searches for data outside the range 2.5 and 1.5 Jy. UVFND has a full complement of calibration adverbs.

   channel=0 One channel only

   niter=500 Print 500 lines maximum of output

   uvrange=0 No uv range restriction

   stokes='I' Examine Stokes I or

   stokes='RR' RR or LL if polarization

   stokes='LL' trickery (really XX,YY) invoked

   sources='1223-23',' ' Select desired source

   timer=0 No time restrictions

   freqid=3 Select frequency id.

   docalib=-1 Optional calibration

   gainuse=2 CL table to apply if docal true

   flagver=0 Read highest version of FG table

   opcode='CLIP' Search for points outside of range

   aparm=2.5,100,1.5,0 100 and 0 ensures no cross hand point will be found erroneously

   You can also print out some of your data, just to confirm your suspicions about its quality, with PRTUV. Although PRTUV is not enormously well endowed with adverbs (and especially it does not read FG tables) you can select by source and time range, so it's sufficient. Note that the adverb ncount in PRTUV gives the maximum number of visibilities to print. Its default value of 0, just gives one page of listing; set it to 100 or 1000 or so. You can always quit when you have seen enough, as all these tasks, when listing to the terminal, give you the option to stop every page of output.

   sources='1333-33', ' ' Select source

   channel=0 For continuum data base

   channel=10 Print channel 10 from spectral data base

   ncount=1000 Print at most 1000 visibilities (0 means 1 page)

   xinc=1 Print every visibility

   cparm=0,10,0,0, 0,10,20,0 Select time range to print 10 hours 0 min to 10 hours 20 min on first day

   You might also use LISTR to list your data in a variety of ways. In particular, LISTR has a matrix listing option well known to VLA users. LISTR, however, can also apply any calibrations you desire, so that you can print calibrated data later on if you wish.

6. If you have tracked down some more bad data with any of UVPLT, UVFND, PRTUV, or LISTR, now you need to actually flag it. Do this with the task UVFLG. Note that the freqid adverb is not included in UVFLG, so that you must specify completely the bad data with the other adverbs, notably, sources, timerang, antennas, and baseline. UVFLG will write commands into the FG table, and you specify which version with the flagver adverb. You may wish to keep multiple FG tables when you're, say, testing the effect of flagging a certain section of data. A very useful adverb in UVFLG (and QUACK) is reason. This string, specifying why you flagged the data, is written into the FG table. It can then be used simply to unflag that data if you make a mistake. You do this by setting opcode = 'reas' and specifying the desired reason. Note that all other selection criteria are ignored when unflagging by reason. TVFLG and IBLED fill in the reason location in the FG table with their names and date. If you wish to unflag with all selection criteria (including reason) then put opcode = 'uflg'. These must be identical to the way the FG table entry was written to unflag an FG table entry. I suggest you read the UVFLG EXPLAIN file fairly carefully before getting stuck into it.

   sources='0233-27',' ' Select source

   timerang=1,12,0,0,1,12,30,0 Select time range carefully

   bchan=0 Flag all channels echan=0

   antennas=2,0 Flag antenna 2 with all others baseline=0

   stokes ' ' Flag all polarizations

   flagver=0 Write to FG table 1

   opcode='flag' Flag data

   opcode='uflg' Unflag data by all selection criterial

   opcode='reas' Unflag data by reason

   reason='antenna 2 died' Reason so that it's easy to unflag

   You may come across a task called CLIP in which you set a clipping range, above and below which the data are bad and are flagged. However, this task too has not been adapted for multi-source files, so that it is short on adverbs. CLIP works well for single-source files, but is best ignored for multi-source files.

7. If you need to look at the contents of the FG table, use the task PRTAB with inext='fg'. Note however, that the order of the table may not reflect the order in which you entered flagging commands. This is because a number of the calibration tasks will sort some of the tables into time order for speed of access, and rewrite them. You can copy the FG table (to a new file or the same one) with TACOP. If you get desperate, you can edit your FG table with TABED. You can also flag rows of it (flagging the FG table !!) with TAFLG and plot it with TAPLT.
8. Occasionally you may find you want to concatenate FG tables. This can also be done with TABED using the `COPY' option. If the output table already exists, the input table is just joined to it. The most instance of needing to do this is as follows. Imagine that you do your initial editing on a spectral-line data base (see § 7.2 below). You then form a channel 0 data base with AVSPC and decide to edit further with it. You then wish to combine the FG tables from the channel 0 and spectral-line data bases. You can do it as in the example below. Make sure that when you edited the channel 0 data base that you set bchan=1 and echan=0 so that all the spectral-line data base will be flagged whenever a channel 0 FG table entry is found.

   inname='1934' Input file inclass='ch0' inseq=1 indisk=4

   outname='1934' Output file outclass='uvtb' outseq=1 outdisk=4

   inext='fg' Append FG table

   invers=1 Input FG version

   outvers=1 Output FG version

   optype='copy' Copy mode