There are several steps involved in running the spectral-line imaging, with several optional pre-processing steps:
Following this pre-processing, the resulting MS is imaged by either the imager task (the default), or the old simager, creating a set of spectral cubes. Imager provides the ability to image in the barycentric reference frame, and allows (for efficiency purposes) the option of writing out multiple sub-cubes (each having a subset of the full range of channels).
Following imaging, the cube statistics are calculated, using a distributed task within the same slurm job as the spectral imaging. This produces a file listing a series of statistics for each channel, as well as a plot of the statistics. See Validation and Diagnostics for examples. These statistics are then used to identify problematic channels (for example, due to divergence in the imaging caused by RFI) that are then masked. Blank channels (those not imaged, e.g. due to barycentric correction or missing data) are also masked. The statistics file is then regenerated.
A final task can perform image-based continuum subtraction. There are two choices for this step, made via the SPECTRAL_IMSUB_SCRIPT parameter. The first uses the robust_contsub_mpi.py script in the ACES directory to fit and subtract a low-order polynomial to each spectrum in the cube separately. The second uses the contsub_im.py script which uses a Savitzky-Golay filter to find and remove the spectral baseline, again in each spectrum of the cube separately. These tasks are intended as demonstrations of this capability - there will be an ASKAPsoft equivalent to this in a future release. The task assumes that the requested python script is in $ACES/tools - if it is not found, the task will not run. The former script is MPI-enabled, allowing it to run on multiple cores (defined by NCORES_IMCONTSUB). The latter is still a serial task, so will take considerably longer.
The variables presented below work in the same manner as those for the continuum imaging, albeit with names that clearly refer to the spectral-imaging.
Whether or not the spectral processing is done is governed by the DO_SPECTRAL_PROCESSING parameter - the default approach of the pipeline is to not do any of the processing above, but if this parameter is set to true then it falls to the individual switches for each task. Each of these default to true, so if DO_SPECTRAL_PROCESSING is turned on then everything will be done.
A note on the imagers and the output formats. The default approach is to use the new imager imager (imager (under test)) to produce the spectral-line cubes. The legacy spectral imager application simager can be used by setting DO_ALT_IMAGER_SPECTRAL or DO_ALT_IMAGER to false. The latter is the switch controlling all types of imaging, but can be overridden by the former, if provided.
The default output format is CASA images, although FITS files can be written directly by setting IMAGETYPE_SPECTRAL to fits (rather than casa). This will only work with the new imager, as simager does not have this functionality. This mode is still in development, so may not be completely reliable. The recommended method for getting images into FITS format is still to use the DO_CONVERT_TO_FITS flag, which makes use of the FITS conversion application. A single FITS file can be produced by setting ALT_IMAGER_SINGLE_FILE=true.
| Variable | Default | Parset equivalent | Description |
|---|---|---|---|
| DO_SPECTRAL_PROCESSING | false | none | Whether to do the spectral-line processing. |
| JOB_TIME_SPECTRAL_IMAGE | JOB_TIME_DEFAULT (24:00:00) | none | Time request for imaging the spectral-line data |
| IMAGETYPE_SPECTRAL | fits | imagetype (imager (under test)) | Image format to use - can be either ‘casa’ or ‘fits’, although ‘fits’ can only be given in conjunction with DO_ALT_IMAGER_SPECTRAL=true. |
| Preparation of spectral dataset | |||
| DO_COPY_SL | false | none | Whether to copy a channel range of the original full-spectral-resolution measurement set into a new MS. If the original MS is original.ms, this will create original_SL.ms. |
| JOB_TIME_SPECTRAL_SPLIT | JOB_TIME_DEFAULT (24:00:00) | none | Time request for splitting out a subset of the spectral data |
| CHAN_RANGE_SL_SCIENCE | “1-NUM_CHAN_SCIENCE“ | channel (mssplit (Measurement Splitting/Averaging Utility)) | The range of channels to copy from the original dataset (1-based). |
| TILENCHAN_SL | 1 | stman.tilenchan (mssplit (Measurement Splitting/Averaging Utility)) | The number of channels in the tile size used for the new MS. The tile size defines the minimum amount read at a time. |
| DO_APPLY_CAL_SL | true | none | Whether to apply the gains calibration determined from the continuum self-calibration (see GAINS_CAL_TABLE in User Parameters - Continuum imaging). |
| JOB_TIME_SPECTRAL_APPLYCAL | JOB_TIME_DEFAULT (24:00:00) | none | Time request for applying the gains calibration to the spectral data |
| DO_CONT_SUB_SL | true | none | Whether to subtract a continuum model from the spectral-line dataset. If true, the clean model from the continuum imaging will be used to represent the continuum, and this will be subtracted from the spectral-line dataset (either the original full-spectral-resolution one, or the reduced-channel-range copy), which gets overwritten. |
| JOB_TIME_SPECTRAL_CONTSUB | JOB_TIME_DEFAULT (24:00:00) | none | Time request for subtracting the continuum from the spectral data |
| Continuum subtraction | |||
| CONTSUB_METHOD | Cmodel | none | This defines which method is used to determine the continuum that is to be subtracted. It can take one of three values: Cmodel (the default), which uses a model image constructed by Cmodel (cmodel (Model Image Generator)) from a continuum components catalogue generated by Selavy (Selavy Basics); Components, which uses the Selavy catalogue directly by in the form of components; or CleanModel, in which case the clean model from the continuum imaging will be used. |
| CONTSUB_SELAVY_NSUBX | 6 | nsubx (Selavy Basics) | Division of image in x-direction for source-finding |
| CONTSUB_SELAVY_NSUBY | 3 | nsuby (Selavy Basics) | Division of image in y-direction for source-finding |
| CONTSUB_SELAVY_THRESHOLD | 6 | snrCut (Selavy Basics) | SNR threshold for detection with Selavy in determining components to go into the continuum model. |
| CONTSUB_MODEL_FLUX_LIMIT | 10uJy | flux_limit (cmodel (Model Image Generator)) | Flux limit applied to component catalogue - only components brighter than this will be included in the model image. Parameter takes the form of a number+units string. |
| CONTSUB_SELAVY_FLAG_ADJACENT | true | flagAdjacent (Selavy Basics) | Whether to enforce pixels in islands to be contiguous. |
| CONTSUB_SELAVY_SPATIAL_THRESHOLD | 5 | threshSpatial (Selavy Basics) | If CONTSUB_SELAVY_FLAG_ADJACENT=false, this is the threshold in pixels within which islands are joined. |
| Basic variables for imaging | |||
| DO_SPECTRAL_IMAGING | true | none | Whether to do the spectral imaging |
| NUM_CPUS_SPECIMG_SCI | 200 | none | The total number of cores allocated to the spectral-imaging job. One will be the master, while the rest will be devoted to imaging individual channels. This is not used when DO_ALT_IMAGER_SPECTRAL=true - instead, it is set to NCHAN / NCHAN_PER_CORE_SL + 1. If NCHAN_PER_CORE_SL does not evenly divide into NCHAN, then an error is raised and no jobs are submitted. |
| CPUS_PER_CORE_SPEC_IMAGING | 20 | none | The number of cores per node to use (max 20). |
| IMAGE_BASE_SPECTRAL | i.SB%s.cube | Helps form Images.name (simager) | The base name for image cubes: if IMAGE_BASE_SPECTRAL=i.blah then we’ll get image.i.blah, image.i.blah.restored, psf.i.blah etc. The %s wildcard will be resolved into the scheduling block ID. |
| DIRECTION_SCI | none | Images.direction (simager) | The direction parameter for the image cubes, i.e. the central position. Can be left out, in which case it will be determined from the measurement set by mslist. This is the same input parameter as that used for the continuum imaging. |
| NUM_PIXELS_SPECTRAL | 1024 | Images.shape (simager) | The number of spatial pixels along the side for the image cubes. Needs to be specified (unlike the continuum imaging case). |
| CELLSIZE_SPECTRAL | 8 | Images.cellsize (simager) | The spatial pixel size for the image cubes. Must be specified. |
| REST_FREQUENCY_SPECTRAL | HI | Images.restFrequency (simager) | The rest frequency for the cube. Can be a quantity string (eg. 1234.567MHz), or the special string ‘HI’ (which is 1420.405751786 MHz). If blank, no rest frequency will be written to the cube. |
| SPECTRAL_IMAGE_MAXUV | 2000 | MaxUV (Data Selection) | A maximum UV distance (in metres) to apply in the data selection step. Only used if a positive value is applied. |
| SPECTRAL_IMAGE_MINUV | 0 | MinUV (Data Selection) | A minimum UV distance (in metres) to apply in the data selection step. Only used if a positive value is applied. |
| DO_MASK_BAD_CHANNELS | true | none | Whether to mask out bad or blank channels from the spectral cube. |
| Gridding | |||
| GRIDDER_SPECTRAL_SNAPSHOT_IMAGING | false | snapshotimaging (Gridders) | Whether to use snapshot imaging when gridding. |
| GRIDDER_SPECTRAL_SNAPSHOT_WTOL | 2600 | snapshotimaging.wtolerance (Gridders) | The wtolerance parameter controlling how frequently to snapshot. |
| GRIDDER_SPECTRAL_SNAPSHOT_LONGTRACK | true | snapshotimaging.longtrack (Gridders) | The longtrack parameter controlling how the best-fit W plane is determined when using snapshots. |
| GRIDDER_SPECTRAL_SNAPSHOT_CLIPPING | 0.01 | snapshotimaging.clipping (Gridders) | If greater than zero, this fraction of the full image width is set to zero. Useful when imaging at high declination as the edges can generate artefacts. |
| GRIDDER_SPECTRAL_WMAX | 2600 (GRIDDER_SNAPSHOT_IMAGING=true) or 35000 (GRIDDER_SNAPSHOT_IMAGING=false) | WProject.wmax (Gridders) | The wmax parameter for the gridder. The default for this depends on whether snapshot imaging is invoked or not (GRIDDER_SNAPSHOT_IMAGING). |
| GRIDDER_SPECTRAL_NWPLANES | 257 | WProject.nwplanes (Gridders) | The nwplanes parameter for the gridder. |
| GRIDDER_SPECTRAL_OVERSAMPLE | 4 | WProject.oversample (Gridders) | The oversampling factor for the gridder. |
| GRIDDER_SPECTRAL_MAXSUPPORT | 512 (GRIDDER_SNAPSHOT_IMAGING=true) or 1024 (GRIDDER_SNAPSHOT_IMAGING=false) | WProject.maxsupport (Gridders) | The maxsupport parameter for the gridder. The default for this depends on whether snapshot imaging is invoked or not (GRIDDER_SNAPSHOT_IMAGING). |
| GRIDDER_SPECTRAL_SHARECF | true | WProject.sharecf (Gridders) | Whether to use a (static) cache for the convolution functions in the WProject gridder. |
| Cleaning | |||
| SOLVER_SPECTRAL | Clean | solver (Solvers) | Which solver to use. You will mostly want to leave this as ‘Clean’, but there is a ‘Dirty’ solver available. |
| CLEAN_SPECTRAL_ALGORITHM | BasisfunctionMFS | Clean.algorithm (Solvers) | The name of the clean algorithm to use. |
| CLEAN_SPECTRAL_MINORCYCLE_NITER | 800 | Clean.niter (Solvers) | The number of iterations for the minor cycle clean. |
| CLEAN_SPECTRAL_GAIN | 0.2 | Clean.gain (Solvers) | The loop gain (fraction of peak subtracted per minor cycle). |
| CLEAN_SPECTRAL_PSFWIDTH | 256 | Clean.psfwidth (Solvers) | The width of the psf patch used in the minor cycle. |
| CLEAN_SPECTRAL_SCALES | "[0,3,10,30]" | Clean.scales (Solvers) | Set of scales (in pixels) to use with the multi-scale clean. |
| CLEAN_SPECTRAL_THRESHOLD_MINORCYCLE | "[45%, 3.5mJy, 0.5mJy]" | threshold.minorcycle (Solvers) | Threshold for the minor cycle loop. |
| CLEAN_SPECTRAL_THRESHOLD_MAJORCYCLE | 0.5mJy | threshold.majorcycle (Solvers) | The target peak residual. Major cycles stop if this is reached. A negative number ensures all major cycles requested are done. |
| CLEAN_SPECTRAL_NUM_MAJORCYCLES | 3 | ncycles (Solvers) | Number of major cycles. |
| CLEAN_WRITE_AT_MAJOR_CYCLE | false | Images.writeAtMajorCycle (simager) | If true, the intermediate images will be written (with a .cycle suffix) after the end of each major cycle. |
| CLEAN_SPECTRAL_SOLUTIONTYPE | MAXCHISQ | Clean.solutiontype (see discussion at Multi-Scale and/or Multi-Frequency deconvolution) | The type of peak finding algorithm to use in the deconvolution. Choices are MAXCHISQ, MAXTERM0, or MAXBASE. |
| Preconditioning | |||
| PRECONDITIONER_LIST_SPECTRAL | "[Wiener, GaussianTaper]" | preconditioner.Names (Solvers) | List of preconditioners to apply. |
| PRECONDITIONER_SPECTRAL_GAUSS_TAPER | "[20arcsec, 20arcsec, 0deg]" | preconditioner.GaussianTaper (Solvers) | Size of the Gaussian taper - either single value (for circular taper) or 3 values giving an elliptical size. |
| PRECONDITIONER_SPECTRAL_WIENER_ROBUSTNESS | 0.5 | preconditioner.Wiener.robustness (Solvers) | Robustness value for the Wiener filter. |
| PRECONDITIONER_SPECTRAL_WIENER_TAPER | "" | preconditioner.Wiener.taper (Solvers) | Size of gaussian taper applied in image domain to Wiener filter. Ignored if blank (ie. ""). |
| Restoring | |||
| RESTORE_SPECTRAL | true | restore (simager) | Whether to restore the image cubes. |
| RESTORING_BEAM_SPECTRAL | fit | restore.beam (simager) | Restoring beam to use: ‘fit’ will fit the PSF in each channel separately to determine the appropriate beam for that channel, else give a size (such as “[30arcsec, 30arcsec, 0deg]”). |
| RESTORING_BEAM_CUTOFF_SPECTRAL | 0.5 | restore.beam.cutoff (simager) | Cutoff value used in determining the support for the fitting (ie. the rectangular area given to the fitting routine). Value is a fraction of the peak. |
| RESTORING_BEAM_REFERENCE | mid | restore.beamReference (simager) | Which channel to use as the reference when writing the restoring beam to the image cube. Can be an integer as the channel number (0-based), or one of ‘mid’ (the middle channel), ‘first’ or ‘last’ |
| New imager parameters | |||
| DO_ALT_IMAGER_SPECTRAL | "" | none | If true, the spectral-line imaging is done by imager (doc:../calim/imager). If false, it is done by simager (simager). When true, the following parameters are used. If left blank (the default), the value is given by the overall parameter DO_ALT_IMAGER (see User Parameters - Pipeline & job control). |
| NCHAN_PER_CORE_SL | 64 | nchanpercore (imager (under test)) | The number of channels each core will process. |
| USE_TMPFS | false | usetmpfs (imager (under test)) | Whether to store the visibilities in shared memory. This will give a performance boost at the expense of memory usage. Better used for processing continuum data. |
| TMPFS | /dev/shm | tmpfs (imager (under test)) | Location of the shared memory. |
| NUM_SPECTRAL_WRITERS | 16 | nwriters (imager (under test)) | The number of writers used by imager. Unless ALT_IMAGER_SINGLE_FILE=true, this will equate to the number of distinct spectral cubes produced. In the case of multiple cubes, each will be a sub-band of the full bandwidth. No combination of the sub-cubes is currently done. The number of writers will be reduced to the number of workers in the job if necessary. |
| ALT_IMAGER_SINGLE_FILE | true | singleoutputfile (imager (under test)) | Whether to write a single cube, even with multiple writers (ie. NUM_SPECTRAL_WRITERS>1). Only works when IMAGETYPE_SPECTRAL=fits |
| FREQ_FRAME_SL | bary | freqframe (imager (under test)) | The reference frame in which to write the spectral cube - one of topo, bary, lsrk. Anything else (or an unset value) default to bary. |
| OUTPUT_CHANNELS_SL | "" | Frequencies (imager (under test)) | The output channels for the spectral cube. Should be of the form [number,start,width], with the start and width parameters are in Hz. If not given, the behaviour is to use the same frequency values as the input MS, albeit in the requested frequency frame. |
| Image-based continuum subtraction | |||
| DO_SPECTRAL_IMSUB | true | none | Whether to run an image-based continuum-subtraction task on the spectral cube after creation. |
| JOB_TIME_SPECTRAL_IMCONTSUB | JOB_TIME_DEFAULT (24:00:00) | none | Time request for image-based continuum subtraction |
| SPECTRAL_IMSUB_SCRIPT | "robust_contsub_mpi.py" | none | The name of the script from the ACES repository to use for image-based continuum subtraction. The only two accepted values are “robust_contsub_mpi.py” and “contsub_im.py”. Anything else reverts to the default. |
| NCORES_IMCONTSUB | 256 | none | Number of cores to use for the robust_contsub_mpi.py script. Ignored if contsub_im.py is used. Must divide evenly into the number of spatial pixels in the spectral cube. |
| SPECTRAL_IMSUB_VERBOSE | true | none | Whether to use verbose output in the logging for the image-based continuum subtraction. |
| SPECTRAL_IMSUB_THRESHOLD | 2.0 | none (‘threshold’ parameter in robust_contsub_mpi.py) | Threshold [sigma] to mask outliers prior to fitting the continuum baseline in the “robust_contsub_mpi.py” version of the image-based continuum-subtraction. |
| SPECTRAL_IMSUB_FIT_ORDER | 2 | none (‘fit_order’ parameter in robust_contsub_mpi.py) | Order of the polynomial to fit to the continuum baseline in the “robust_contsub_mpi.py” version of the image-based continuum subtraction. |
| SPECTRAL_IMSUB_CHAN_SAMPLING | 1 | none (‘n_every’ parameter in robust_contsub_mpi.py) | If set to n, we use only every nth channel in the polynomial fit (1 uses every channel). Only for “robust_contsub_mpi.py” |
| SPECTRAL_IMSUB_LOG_SAMPLING | 1 | none (‘log_every’ parameter in robust_contsub_mpi.py) | How frequently the log messages from “robust_contsub_mpi.py” should be written (1 means every channel). |
| SPECTRAL_IMSUB_SG_FILTERWIDTH | 200 | none (‘filterwidth’ parameter in contsub_im.py) | The half-width of the Savitzky-Golay filter for baseline smoothing in the “contsub_im.py” script. |
| SPECTRAL_IMSUB_SG_BINWIDTH | 4 | none (‘binwidth’ parameter in contsub_im.py) | The bin width used for binning the spectrum before continuum subtraction (“contsub_im.py” only). |