Thresholds in Selavy

Global thresholds

The Duchamp package uses a single detection threshold for the entire image being searched. This can be either specified as a flux threshold Selavy.threshold, or as a signal-to-noise threshold via Selavy.snrCut. In the latter case, the noise is calculated from the image statistics over either the entire dataset or a subsection given by Selavy.StatSec.

The way the statistics are calculated (for a signal-to-noise threshold) is determined by the Selavy.flagRobustStats parameter. If true (the default), the noise statistics are characterised by the median and MADFM (median absolute deviation from the median). This provides a robust estimate not strongly biased by the presence of bright pixels. It can take longer to process however. If this parameter is false, the noise statistics will be characterised by the mean and standard deviation.

When Selavy is run in distributed mode, using a flux threshold is still straightforward, but a signal-to-noise threshold requires extra work to get an appropriate global noise estimate. Each worker finds the mean (or median), sends it to the master process which averages them to find the global mean estimator. This is then distributed to the workers who then find their local standard deviation or MADFM. These are again averaged by the master to provide a global estimator of the standard deviation, and hence the threshold. A more complete description of this process can be found in Whiting & Humphreys (2012), PASA 29, 371.

However, if the sensitivity varies across the field, this will either mean some regions are not searched as deep as they could be and/or some are searched too deeply, resulting in too many spurious detections. Selavy deals with this in one of two ways, described in the following section.

Varying the threshold

The first way to allow the threshold to vary is to use a weights image, such as that produced by the ASKAPsoft imager (and included in most of the ASKAP simulations), to scale the image according to the sensitivity. In practice, this takes the square root of the normalised weights and divides this into the pixel values. This has the effect of scaling down the low-sensitivity regions of the image, making it less likely that they present many spurious detections. Set Selavy.WeightScaling=true to utilise this mode - the weights image is specified via Selavy.WeightScaling.weightsimage, which has no default. The detection thresholds are provided in the usual fashion. The pixel values are only affected for the detection phase - parameter calculations are not affected.

The alternative method, which provides a bit more flexibility, is to impose a signal-to-noise threshold based on the local noise surrounding the pixel in question. This threshold then varies from pixel to pixel based on the change in the local noise. This mode is turned on using the Selavy.VariableThreshold parameter, which defaults to false.

This “local” level is estimated by measuring the noise properties of pixels within a box centred on the pixel in question. An array is thus built up containing the signal-to-local-noise values for each pixel in the image, and this array is then searched with a SNR threshold (Selavy.snrCut) and, if necessary, grown to a secondary SNR threshold (Selavy.growthCut). The way the noise properties are calculated is governed by the Selavy.flagRobustStats parameter. A value of true means robust statistics will be used, specifically the median and the median absolute deviation from the median (MADFM) – the latter will be converted to the equivalent standard deviation for a Gaussian noise distribution for the purposes of calculating the signal-to-noise threshold. A value of false means we use the mean and the standard deviation.

The searching can be done either spatially or spectrally, and this affects how the SNR values are calculated. If spatially (the default), a 2D sliding box filter is used to find the local noise. If spectrally, only a 1D “box” is used. Note that the edges (ie. all pixels within the half box width of the edge) are set to zero, and so detections will not be made there. This probably won’t affect the 2D case, as often the edges of the field have poor sensitivity (certainly the ASKAP simulations mostly have a padding region around the edge), but in the 1D case this will mean the loss of the first & last channels. The choice between 2D and 1D is made with the Selavy.searchType parameter (which actually comes out of the Duchamp package).

When run on a distributed system as above, this processing is done at the worker level. Note that having an overlap between workers of at least the half box width will give continuous coverage (avoiding the aforementioned edge problems). Selavy will increase the overlap to account for this if necessary. The amount of processing needed increases quickly with the size of the box, especially in the case of robust statistics due to the use of medians, and particularly for the 2D case.

The various maps created can be written out to disk – see section below. If you have run this once and written out the images, specifically the SNR map, then you can re-run the searching with a different threshold without having to re-do the calculations. Simply give Selavy.VariableThreshold.reuse=true (this defaults to false).

A final option for varying the threshold spatially is to use a different threshold for each worker. In this scenario, switched on by setting thresholdPerWorker = true, each worker finds its own threshold based on the noise within it, using the snrCut signal-to-noise ratio threshold. No variation of the threshold within a worker is done, so you get discrete jumps in the threshold at worker boundaries. Use of the overlap can mitigate this. This mode was implemented more as an experiment than out of any expectation it would be useful, and limited trials indicate it’s probably not much use. For completeness we include the parameter here.

Threshold-related parameters

Parameter	Type	Default	Description
Selavy.threshold	float	no default	The flux threshold applied to the entire image. Not compatible with the variable threshold parameters. If given, takes precendence over Selavy.snrcut.
Selavy.snrCut	float	5.0	The signal-to-noise threshold, in units of sigma above the mean.
Selavy.Weights	bool	false	Whether to scale the fluxes by the weights for the purposes of source detection.
Selavy.Weights.weightsimage	string	“”	The filename of the weights image to be used to scale the fluxes prior to searching.
Selavy.Weights.weightsCutoff	float	-1	If positive (not by default), pixels with a weight below this value are set to zero for the searching step. The value provided should be a fraction (between 0 and 1) of the maximum weight value in the image. Note that this is different to the cutoff used by linmos (Linear Mosaic Applicator), which is a cutoff in the gain value. (You need to square the value given to linmos to provide the same value to Selavy.)
Selavy.VariableThreshold	bool	false	If true, a sliding box function is used to find the local noise properties, which are used to make a signal-to-noise map that can be used for searching.
Selavy.VariableThreshold.boxSize	int	50	The half-width of the box used in the SNR map calculation. The full width of the box is 2*boxSize+1.
Selavy.VariableThreshold.reuse	bool	false	If true, Selavy will load the signal-to-noise ratio map from the image named by the SNRimageName parameter (see table below). If this image does not exist, the calculations will proceed as normal.
Selavy.searchType	string	spatial	In which sense to do the searching: spatial=2D searches, one channel map at a time; spectral=1D searches, one spectrum at a time. The variable searches are affected by this, in that the spatial search uses a 2D box, while the spectral search uses a 1D box.
Selavy.flagRobustStats	bool	true	Whether to calculate the noise properties with robust statistics (that is, the median and the median absolute deviation from the median), or (if false) the mean and standard deviation.
Selavy.thresholdPerWorker	bool	false	If true, each worker’s subimage sets its own threshold.

Saving threshold maps

Selavy provides the option of writing out the various arrays created for the VariableThreshold mode. These include the signal-to-noise map, the noise map and the threshold map. The format of the images are governed by the VariableThreshold.imagetype parameter - if this is “fits”, then a “.fits” extension will be added to the filename. If the name is not given, no image will be written. The images will be created with the same size as the full input image (any search subsection is ignored - pixels outside this are set to zero). These maps are able to be reused when Selavy.VariableThreshold.reuse=true.

The parameters controlling this behaviour are listed below.

Parameter	Type	Default	Description
Selavy.VariableThreshold.imagetype	string	fits	The image format for the ouptut images - either “fits” or “casa”
Selavy.VariableThreshold.SNRimageName	string	“”	The name of the CASA image containing the SNR map
Selavy.VariableThreshold.ThresholdImageName	string	“”	The name of the CASA image containing the threshold map
Selavy.VariableThreshold.NoiseImageName	string	“”	The name of the CASA image containing the noise map
Selavy.VariableThreshold.AverageImageName	string	“”	The name of the CASA image containing the background average map