User Parameters - Spectral-line Imaging

There are several steps involved in running the spectral-line imaging, with several optional pre-processing steps:

  1. A nominated channel range can optionally be copied to a new MS with mssplit.
  2. The gains solution from the continuum self-calibration can be applied to the spectral-line MS using ccalapply.
  3. The continuum can be subtracted from the spectral-line MS using ccontsubtract. The continuum is represented by either the clean model from the continuum imaging, or – as the default – a model image constructed by Cmodel from the component catalogue generated by Selavy. The Selavy parameters used are those described on User Parameters - Continuum Source-finding.
  4. A new tool remUVcont allows removal of both continuum as well as instrument-induced non-idealities. As the name suggests this is a visibility-based modeling and subtraction method using basis functions comprising of polynomials and harmonics with useful frills suitable for dealing with instrumental systematics.

Following this pre-processing, the resulting MS is imaged by either the imager task (the default), or the old simager, creating a set of spectral cubes. Imager provides the ability to image in the barycentric reference frame, and allows (for efficiency purposes) the option of writing out multiple sub-cubes (each having a subset of the full range of channels).

Following imaging, the cube statistics are calculated, using a distributed task within the same slurm job as the spectral imaging. This produces a file listing a series of statistics for each channel, as well as a plot of the statistics. See Validation and Diagnostics for examples. These statistics are then used to identify problematic channels (for example, due to divergence in the imaging caused by RFI) that are then masked. Blank channels (those not imaged, e.g. due to barycentric correction or missing data) are also masked. The statistics file is then regenerated.

A final task can perform image-based continuum subtraction. The default tool is the imcontsub tool. This fits and subtracts a low-order polynomial to each spectrum in the cube separately. It is able to fit independently in blocks of channels, which allows it to take into account potential discontinuities at the edges of beam-forming intervals.

Alternatively, there are two python tools available for this step, selected by setting IMCONTSUB_USE_PYTHON=true, and choosing the script via the SPECTRAL_IMSUB_SCRIPT parameter. These were originally developed as demonstration tools, which led to the development of the imcontsub tool. The first option uses the script in the ACES directory, which has a similar behaviour to the imcontsub tool. The second uses the script which uses a Savitzky-Golay filter to find and remove the spectral baseline, again in each spectrum of the cube separately. The requested python script is assumed to be in $ACES/tools - if it is not found, the task will not run. The former script is MPI-enabled, allowing it to run on multiple cores (defined by NUM_CORES_IMCONTSUB). The latter is still a serial task, so will take considerably longer.

The image-based continuum subtraction can take into account the potential discontinuities at the edges of beam-forming intervals. This is assumes the intervals are a constant size (given by SPECTRAL_IMSUB_BLOCKSIZE), and with some offset relative to the start of the spectrum (given by SPECTRAL_IMSUB_SHIFT, where it takes the value of the number of channels from the start of the first interval to the start of the spectrum). By default, these parameters are derived from the edge frequencies recorded in the SB obs variables, but this may be over-ridden by specifying one or both of these parameters (if only one is given, the other defaults to zero). Providing these parameters will be necessary for SBs older than 26120. Note also that does not provide this functionality.

The variables presented below work in the same manner as those for the continuum imaging, albeit with names that clearly refer to the spectral-imaging.

Whether or not the spectral processing is done is governed by the DO_SPECTRAL_PROCESSING parameter - the default approach of the pipeline is to not do any of the processing above, but if this parameter is set to true then it falls to the individual switches for each task. Each of these default to true, so if DO_SPECTRAL_PROCESSING is turned on then everything will be done.

A note on the imagers and the output formats. The default approach is to use the new imager imager (imager) to produce the spectral-line cubes. The legacy spectral imager application simager can be used by setting DO_ALT_IMAGER_SPECTRAL or DO_ALT_IMAGER to false. The latter is the switch controlling all types of imaging, but can be overridden by the former, if provided.

The default output format is CASA images, although FITS files can be written directly by setting IMAGETYPE_SPECTRAL to fits (rather than casa). This will only work with the new imager, as simager does not have this functionality. This mode is still in development, so may not be completely reliable. The recommended method for getting images into FITS format is still to use the DO_CONVERT_TO_FITS flag, which makes use of the FITS conversion application. A single FITS file can be produced by setting ALT_IMAGER_SINGLE_FILE=true.

Optional Image products

Imager allows users to specify whether or not to write optional output products such as the residual, weights, natural psf, preconditioned psf etc. images. These are aimed at minimising disk I/O and occupancy where possible.

Note that information on weights is required by the linear mosaicking applications: linmos and linmos-mpi. For non A-Projection gridders like WProject, where the weights value across the image extent is constant, imager can write out the weight value into an ascii text file that linmos can make use of. This can be specified using WRITE_WEIGHTS_LOG_SPECTRAL parameter for the spectral line imaging case.

For A-project gridders, or when snapshot imaging is turned on, one will have to necessarily write out the weights images for use in linmos. An exception will be thrown by if this condition is violated.

The imager can also optionally output the gridded visibility cubes. This can be requested by setting WRITE_UVGRIDS_SPECTRAL=true.

More details on writing these optional image products are discussed in the sections below. See also :doc: ScienceFieldContinuumImaging and :doc: ScienceFieldMosaicking.

Variable Default Parset equivalent Description
DO_SPECTRAL_PROCESSING false none Whether to do the spectral-line processing.
JOB_TIME_SPECTRAL_IMAGE JOB_TIME_DEFAULT (24:00:00) none Time request for imaging the spectral-line data
IMAGETYPE_SPECTRAL fits imagetype (imager) Image format to use - can be either ‘casa’ or ‘fits’, although ‘fits’ can only be given in conjunction with DO_ALT_IMAGER_SPECTRAL=true.
Preparation of spectral dataset      
DO_COPY_SL false none Whether to copy a channel range of the original full-spectral-resolution measurement set into a new MS. If the original MS is, this will create
JOB_TIME_SPECTRAL_SPLIT JOB_TIME_DEFAULT (24:00:00) none Time request for splitting out a subset of the spectral data
CHAN_RANGE_SL_SCIENCE “1-NUM_CHAN_SCIENCE channel (mssplit (Measurement Splitting/Averaging Utility)) The range of channels to copy from the original dataset (1-based).
TILENCHAN_SL 1 stman.tilenchan (mssplit (Measurement Splitting/Averaging Utility)) The number of channels in the tile size used for the new MS. The tile size defines the minimum amount read at a time.
DO_APPLY_CAL_SL true none Whether to apply the gains calibration determined from the continuum self-calibration (see GAINS_CAL_TABLE in User Parameters - Continuum imaging).
JOB_TIME_SPECTRAL_APPLYCAL JOB_TIME_DEFAULT (24:00:00) none Time request for applying the gains calibration to the spectral data
DO_CONT_SUB_SL true none Whether to subtract a continuum model from the spectral-line dataset. If true, the clean model from the continuum imaging will be used to represent the continuum, and this will be subtracted from the spectral-line dataset (either the original full-spectral-resolution one, or the reduced-channel-range copy), which gets overwritten.
JOB_TIME_SPECTRAL_CONTSUB JOB_TIME_DEFAULT (24:00:00) none Time request for subtracting the continuum from the spectral data
Continuum subtraction      
CONTSUB_METHOD CleanModel none This defines which method is used to determine the continuum that is to be subtracted. It can take one of four values: Cmodel, which uses a model image constructed by Cmodel (cmodel) from a continuum components catalogue generated by Selavy (Selavy Basics); Components, which uses the Selavy catalogue directly by in the form of components; CleanModel, in which case the clean model from the continuum imaging will be used; or remUVcont, that models each visibility spectrum using basis functions comprising of a combination of polynomials (equivalemt to miriad UVLIN) and harmonics to remove continuum as well instrumental artifacts.
REMUVCONT_POLY_ORDER “” none Order of the polynomial function used in fitting
REMUVCONT_HARM_ORDER “” none Order of the harmonic function used in fitting
REMUVCONT_N_WIN “” none The data will be divided into nWin windows and a moving fit is done to derive a smooth model
REMUVCONT_N_TAPER “” none Gaussian Taper width in number of channels to be used in an intermediate step where the spectrum is passed through a low pass filter to interpolate across flagged channels. The good data are not altered. This helps make the fitting robust.
REMUVCONT_N_ITER “” none The intermediate los pass filter is an iterative method based on FFT. This parameter specifies the number of iterations. Usually a few tens of iterations suffices. To avoid the low-pass filter step, use 0
REMUVCONT_F54 “” none Set this to an integer value N such that 54 x N = beam-forming width in units of number of channels. If this parameter is set, the fitting will be done only within beam-forming intervals. This is necessary to be robust against any discontinuities at the beam-forming and/or ODC update intervals. Note1: When this option is turned ON REMUVCONT_N_WIN is computed internally. Note2: For small windows, choose REMUVCONT_POLY_ORDER and REMUVCONT_HARM_ORDER carefully. Use low orders.
CONTSUB_SELAVY_NSUBX 6 nsubx (Selavy Basics) Division of image in x-direction for source-finding
CONTSUB_SELAVY_NSUBY 3 nsuby (Selavy Basics) Division of image in y-direction for source-finding
CONTSUB_SELAVY_THRESHOLD 6 snrCut (Selavy Basics) SNR threshold for detection with Selavy in determining components to go into the continuum model.
CONTSUB_MODEL_FLUX_LIMIT 10uJy flux_limit (cmodel) Flux limit applied to component catalogue - only components brighter than this will be included in the model image. Parameter takes the form of a number+units string.
CONTSUB_SELAVY_FLAG_ADJACENT true flagAdjacent (Selavy Basics) Whether to enforce pixels in islands to be contiguous.
CONTSUB_SELAVY_SPATIAL_THRESHOLD 5 threshSpatial (Selavy Basics) If CONTSUB_SELAVY_FLAG_ADJACENT=false, this is the threshold in pixels within which islands are joined.
CONTSUB_SELAVY_SPECTRAL_INDEX_THRESHOLD “” spectralTerms.threshold (Post-processing of detections) Threshold applied to component peak fluxes in determining which have a spectral index (and curvature) value reported in the component catalogue. Not used if left blank. Takes precedence over CONTSUB_SELAVY_SPECTRAL_INDEX_THRESHOLD_SNR.
spectralTerms.thresholdSNR (Post-processing of detections) Threshold applied to component peak signal-to-noise values in determining which have a spectral index (and curvature) value reported in the component catalogue. Not used if left blank.
Basic variables for imaging      
DO_SPECTRAL_IMAGING true none Whether to do the spectral imaging
NUM_CORES_SPECIMG_SCI 200 none The total number of cores allocated to the spectral-imaging job. One will be the master, while the rest will be devoted to imaging individual channels. This is not used when DO_ALT_IMAGER_SPECTRAL=true - instead, it is set to NCHAN / NCHAN_PER_CORE_SL + 1. If NCHAN_PER_CORE_SL does not evenly divide into NCHAN, then an error is raised and no jobs are submitted.
CORES_PER_NODE_SPEC_IMAGING 20 none The number of cores per node to use (max 20).
IMAGE_BASE_SPECTRAL i.%t.SB%s.cube Helps form (simager) The base name for image cubes: if IMAGE_BASE_SPECTRAL=i.blah then we’ll get image.i.blah, image.i.blah.restored, psf.i.blah etc. The %s wildcard will be resolved into the scheduling block ID, and the %t will be resolved into the “target”, or scheduling block alias.
DIRECTION_SCI none Images.direction (simager) The direction parameter for the image cubes, i.e. the central position. Can be left out, in which case it will be determined from the measurement set by mslist. This is the same input parameter as that used for the continuum imaging.
NUM_PIXELS_SPECTRAL 1024 Images.shape (simager) The number of spatial pixels along the side for the image cubes. Needs to be specified (unlike the continuum imaging case).
CELLSIZE_SPECTRAL 8 Images.cellsize (simager) The spatial pixel size for the image cubes. Must be specified.
REST_FREQUENCY_SPECTRAL HI Images.restFrequency (simager) The rest frequency for the cube. Can be a quantity string (eg. 1234.567MHz), or the special string ‘HI’ (which is 1420.405751786 MHz). If blank, no rest frequency will be written to the cube.
SPECTRAL_IMAGE_MAXUV 2000 MaxUV (Data Selection) A maximum UV distance (in metres) to apply in the data selection step. Only used if a positive value is applied.
SPECTRAL_IMAGE_MINUV 0 MinUV (Data Selection) A minimum UV distance (in metres) to apply in the data selection step. Only used if a positive value is applied.
DO_MASK_BAD_CHANNELS true none Whether to mask out bad or blank channels from the spectral cube.
MASK_CHANS_USE_SIG false none Whether to use the “signifcance”, or the ratio of the 1-percent statistic to the MADFM, to determine the bad channels.
none The significance level at which to reject channels.
MASK_CHANS_USE_NOISE true none Whether to mask out bad channels on the basis of the MADFM value alone.
none The value of MADFM (in mJy/beam), above which a channel is deemed bad.
MASK_CHANS_BLANK true none Whether to mask out blank channels from the continuum cube.
GRIDDER_SPECTRAL_SNAPSHOT_IMAGING false snapshotimaging (Gridders) Whether to use snapshot imaging when gridding.
GRIDDER_SPECTRAL_SNAPSHOT_WTOL 2600 snapshotimaging.wtolerance (Gridders) The wtolerance parameter controlling how frequently to snapshot.
GRIDDER_SPECTRAL_SNAPSHOT_LONGTRACK true snapshotimaging.longtrack (Gridders) The longtrack parameter controlling how the best-fit W plane is determined when using snapshots.
GRIDDER_SPECTRAL_SNAPSHOT_CLIPPING 0.01 snapshotimaging.clipping (Gridders) If greater than zero, this fraction of the full image width is set to zero. Useful when imaging at high declination as the edges can generate artefacts.
GRIDDER_SPECTRAL_WMAX 2600 (GRIDDER_SNAPSHOT_IMAGING=true) or 35000 (GRIDDER_SNAPSHOT_IMAGING=false) WProject.wmax (Gridders) The wmax parameter for the gridder. The default for this depends on whether snapshot imaging is invoked or not (GRIDDER_SNAPSHOT_IMAGING).
GRIDDER_SPECTRAL_NWPLANES 257 WProject.nwplanes (Gridders) The nwplanes parameter for the gridder.
GRIDDER_SPECTRAL_OVERSAMPLE 4 WProject.oversample (Gridders) The oversampling factor for the gridder.
GRIDDER_SPECTRAL_MAXSUPPORT 512 (GRIDDER_SNAPSHOT_IMAGING=true) or 1024 (GRIDDER_SNAPSHOT_IMAGING=false) WProject.maxsupport (Gridders) The maxsupport parameter for the gridder. The default for this depends on whether snapshot imaging is invoked or not (GRIDDER_SNAPSHOT_IMAGING).
GRIDDER_SPECTRAL_SHARECF true WProject.sharecf (Gridders) Whether to use a (static) cache for the convolution functions in the WProject gridder.
SOLVER_SPECTRAL Clean solver (Solvers) Which solver to use. You will mostly want to leave this as ‘Clean’, but there is a ‘Dirty’ solver available.
CLEAN_SPECTRAL_ALGORITHM BasisfunctionMFS Clean.algorithm (Solvers) The name of the clean algorithm to use.
CLEAN_SPECTRAL_MINORCYCLE_NITER 800 Clean.niter (Solvers) The number of iterations for the minor cycle clean.
CLEAN_SPECTRAL_GAIN 0.2 Clean.gain (Solvers) The loop gain (fraction of peak subtracted per minor cycle).
CLEAN_SPECTRAL_PSFWIDTH 256 Clean.psfwidth (Solvers) The width of the psf patch used in the minor cycle.
CLEAN_SPECTRAL_SCALES "[0,3,10,30]" Clean.scales (Solvers) Set of scales (in pixels) to use with the multi-scale clean.
CLEAN_SPECTRAL_THRESHOLD_MINORCYCLE "[45%, 3.5mJy, 0.5mJy]" threshold.minorcycle (Solvers) Threshold for the minor cycle loop.
CLEAN_SPECTRAL_THRESHOLD_MAJORCYCLE 0.5mJy threshold.majorcycle (Solvers) The target peak residual. Major cycles stop if this is reached. A negative number ensures all major cycles requested are done.
CLEAN_SPECTRAL_NUM_MAJORCYCLES 3 ncycles (Solvers) Number of major cycles.
CLEAN_WRITE_AT_MAJOR_CYCLE false Images.writeAtMajorCycle (simager) If true, the intermediate images will be written (with a .cycle suffix) after the end of each major cycle.
CLEAN_SPECTRAL_SOLUTIONTYPE MAXBASE Clean.solutiontype (see discussion at Multi-Scale and/or Multi-Frequency deconvolution) The type of peak finding algorithm to use in the deconvolution. Choices are MAXCHISQ, MAXTERM0, or MAXBASE.
CLEAN_SPECTRAL_DETECT_DIVERGENCE true Clean.detectdivergence (Solvers) Whether to detect that the deconvolution is starting to diverge (in which case it is stopped).
PRECONDITIONER_LIST_SPECTRAL "[Wiener, GaussianTaper]" preconditioner.Names (Solvers) List of preconditioners to apply.
PRECONDITIONER_SPECTRAL_GAUSS_TAPER "[20arcsec, 20arcsec, 0deg]" preconditioner.GaussianTaper (Solvers) Size of the Gaussian taper - either single value (for circular taper) or 3 values giving an elliptical size.
PRECONDITIONER_SPECTRAL_GAUSS_TAPER_IS_PSF "false" preconditioner.GaussianTaper.isPsfSize (Solvers) Decide if rather than applying the specified taper, the taper should be adjusted to ensure that the output image planes have the specified resolution.
PRECONDITIONER_SPECTRAL_GAUSS_TAPER_TOL 0.005 preconditioner.GaussianTaper.tolerance (Solvers) Fractional tolerance for the fitted beam size when PRECONDITIONER_SPECTRAL_GAUSS_TAPER_IS_PSF = true The default is set to 0.5%
PRECONDITIONER_SPECTRAL_WIENER_ROBUSTNESS 0.5 preconditioner.Wiener.robustness (Solvers) Robustness value for the Wiener filter.
PRECONDITIONER_SPECTRAL_WIENER_TAPER "" preconditioner.Wiener.taper (Solvers) Size of gaussian taper applied in image domain to Wiener filter. Ignored if blank (ie. "").
RESTORE_SPECTRAL true restore (simager) Whether to restore the image cubes.
RESTORING_BEAM_SPECTRAL fit restore.beam (simager) Restoring beam to use: ‘fit’ will fit the PSF in each channel separately to determine the appropriate beam for that channel, else give a size (such as “[30arcsec, 30arcsec, 0deg]”). NB: If PRECONDITIONER_SPECTRAL_GAUSS_TAPER_IS_PSF=true the restoring beam will be derived such that the output image planes have the fixed specified resolution.
RESTORING_BEAM_CUTOFF_SPECTRAL 0.5 restore.beam.cutoff (simager) Cutoff value used in determining the support for the fitting (ie. the rectangular area given to the fitting routine). Value is a fraction of the peak.
RESTORING_BEAM_REFERENCE mid restore.beamReference (simager) Which channel to use as the reference when writing the restoring beam to the image cube. Can be an integer as the channel number (0-based), or one of ‘mid’ (the middle channel), ‘first’ or ‘last’
New imager parameters      
DO_ALT_IMAGER_SPECTRAL "" none If true, the spectral-line imaging is done by imager (doc:../calim/imager). If false, it is done by simager (simager). When true, the following parameters are used. If left blank (the default), the value is given by the overall parameter DO_ALT_IMAGER (see User Parameters - Pipeline & job control).
NCHAN_PER_CORE_SL 64 nchanpercore (imager) The number of channels each core will process.
USE_TMPFS false usetmpfs (imager) Whether to store the visibilities in shared memory. This will give a performance boost at the expense of memory usage. Better used for processing continuum data.
TMPFS /dev/shm tmpfs (imager) Location of the shared memory.
NUM_SPECTRAL_WRITERS "" nwriters (imager) The number of writers used by imager. Unless ALT_IMAGER_SINGLE_FILE=true, this will equate to the number of distinct spectral cubes produced. In the case of multiple cubes, each will be a sub-band of the full bandwidth. No combination of the sub-cubes is currently done. The number of writers will be reduced to the number of workers in the job if necessary. If left blank, the number of writers will be the number of worker nodes in the imaging job.
ALT_IMAGER_SINGLE_FILE true singleoutputfile (imager) Whether to write a single cube, even with multiple writers (ie. NUM_SPECTRAL_WRITERS>1). Only works when IMAGETYPE_SPECTRAL=fits
FREQ_FRAME_SL bary freqframe (imager) The reference frame in which to write the spectral cube - one of topo, bary, lsrk. Anything else (or an unset value) default to bary.
OUTPUT_CHANNELS_SL "" Frequencies (imager) The output channels for the spectral cube. Should be of the form [number,start,width], with the start and width parameters are in Hz. If not given, the behaviour is to use the same frequency values as the input MS, albeit in the requested frequency frame.
Optional outputs      
WRITE_RESIDUAL_SPECTRAL true write.residualimage (imager) Whether to write the residual image cube
WRITE_PSF_RAW_SPECTRAL false write.psfrawimage (imager) Whether to write the naturally weighted psf image cube
WRITE_PSF_IMAGE_SPECTRAL false write.psfimage (imager) Whether to write the preconditioned psf image cube
WRITE_WEIGHTS_IMAGE_SPECTRAL false write.weightsimage (imager) Whether to write the weights image cube
WRITE_WEIGHTS_LOG_SPECTRAL true write.weightslog (imager)

Option to write the weight spectra into an ascii text file

Note 1: Should not be used with A-project gridders. The
ASKAPpipeline in any case supports WProject only.
Note 2: For snapshot imaging, this option is internally
forced to “false” by the pipeline scripts. Set WRITE_WEIGHTS_IMAGE_CONT=true in this case.
WRITE_MODEL_IMAGE_SPECTRAL true write.modelimage (imager)

Whether to write the model image cube

Note: The pipeline ensures that intermediate model images
get generated if required by selfcal jobs.
WRITE_UVGRIDS_SPECTRAL false write.grids (imager) Whether to write the UV-grids (cubes only)
Image-based continuum subtraction      
DO_SPECTRAL_IMSUB true none Whether to run an image-based continuum-subtraction task on the spectral cube after creation.
JOB_TIME_SPECTRAL_IMCONTSUB JOB_TIME_DEFAULT (24:00:00) none Time request for image-based continuum subtraction
IMCONTSUB_USE_PYTHON false none Whether to use one of the python tools (see next item), instead of the default tool imcontsub (imcontsub).
SPECTRAL_IMSUB_SCRIPT "" none The name of the script from the ACES repository to use for image-based continuum subtraction. The only two accepted values are “” and “”. Anything else reverts to the default.
NUM_CORES_IMCONTSUB 256 none Number of cores to use for the script. Ignored if is used. Must divide evenly into the number of spatial pixels in the spectral cube.
SPECTRAL_IMSUB_VERBOSE true none Whether to use verbose output in the logging for the image-based continuum subtraction.
SPECTRAL_IMSUB_THRESHOLD 2.0 none (‘threshold’ parameter in Threshold [sigma] to mask outliers prior to fitting the continuum baseline in the “” version of the image-based continuum-subtraction.
SPECTRAL_IMSUB_FIT_ORDER 2 none (‘fit_order’ parameter in Order of the polynomial to fit to the continuum baseline in the “” version of the image-based continuum subtraction.
SPECTRAL_IMSUB_CHAN_SAMPLING 1 none (‘n_every’ parameter in If set to n, we use only every nth channel in the polynomial fit (1 uses every channel). Only for “”
SPECTRAL_IMSUB_BLOCKSIZE “” none (‘blocksize’ parameter in Do the fitting and subtracting in blocks of n channels. A value of 0 will result in all channels being used at once. By default, the value for this will be determined from the weights information in the SB obsvariables.
“” none (‘shift’ parameter in Shift the edges of the blocks to the left by this many channels. By default, the value for this will be determined from the weights information in the SB obsvariables.
SPECTRAL_IMSUB_INTERLEAVE false none (‘interleave’ parameter in Interleave the blocks by a half, but only take the middle of each interleave. This reduces edge effects
SPECTRAL_IMSUB_LOG_SAMPLING 1 none (‘log_every’ parameter in How frequently the log messages from “” should be written (1 means every channel).
SPECTRAL_IMSUB_SG_FILTERWIDTH 200 none (‘filterwidth’ parameter in The half-width of the Savitzky-Golay filter for baseline smoothing in the “” script.
SPECTRAL_IMSUB_SG_BINWIDTH 4 none (‘binwidth’ parameter in The bin width used for binning the spectrum before continuum subtraction (“” only).