Selavy Basics

Summary

Selavy is the ASKAP implementation of the source-finding algorithms of Duchamp (publicly available at the Duchamp website).

Duchamp is a source-finder designed for three-dimensional datasets (such as HI spectral-line cubes), but is easily applied to two- or one-dimensional data. There is an extensive user’s guide available on the Duchamp website that provides details about the algorithms. Users are also referred to the Duchamp journal paper Whiting (2012), MNRAS 421, 3242 for further details and examples. The name ‘Duchamp’ comes from Marcel Duchamp, the artist who pioneered the art of the “readymade”, or “found object”.

Selavy uses the core Duchamp algorithms, and adds new features as dictated by the requirements of ASKAP processing. See Whiting & Humphreys (2012), PASA 29, 371 for descriptions and some benchmarks. The new features include support for distributed processing, Gaussian fitting to two-dimensional images, variable-threshold detection, and additional pre- and post-processing algorithms specified by the Survey Science Teams. The name ‘Selavy’ comes from Rrose Selavy, a pseudonym of Marcel Duchamp.

Basic Execution and Control Parameters

This section summarises the basic execution of Selavy, with control of specific types of processing discussed in later sections. The default approach of Selavy is to follow the Duchamp method of applying a single threshold, either a flux value (specified by the user) or a signal-to-noise level, to the entire image. Pixels above this threshold are grouped together in objects - they may be required to be contiguous, or they can be separated by up to some limit. Detected objects may be “grown” to a secondary (lower) threshold, to include faint pixels without having to search to a faint level.

One of the aims of Selavy is to provide distributed processing support, and details on how this is done can be found below. To get a single SNR threshold in this case, the noise stats are calculated on a subimage basis and combined to find the average noise across the image. Selavy also permits variable detection thresholds to be determined in several ways - these are described on the Thresholds page.

The input image can be a FITS format image or a CASA image (as is produced by the ASKAPsoft imaging pipeline). The code to read CASA images is new: it extracts the flux, WCS and metadata information and stores them in the classes used by the Duchamp code. While the Duchamp software is able to read FITS images, this has issues with very large cubes depending on the memory of the system being used - sometimes even the getMetadata() function would break (e.g. when subsections were being requested) when the number of pixels in the subsection was more than memory could hold. For this reason, the CASA code is default, unless the useCASAforFITS parameter is set to false.

General parameters

The following table lists the basic parameters governing the searching. All parameters listed here should have a Selavy. prefix (ie. Selavy.image).

All parameters from Duchamp can be provided in an input parameter set, although some are not implemented in Selavy. A list of these and brief explanations can be found on the Duchamp Exclusions page.

Parameter Type Default Description
image string none The input image - can also use imageFile for consistency with Duchamp parameters
useCASAforFITS bool true Whether to use the casacore functionality for FITS images (if false, the native Duchamp code is used - see discussion above)
searchType string spatial How the searches are done: in 2D channel maps (“spatial”) or in 1D spectra (“spectral”)
snrCut float 4.0 Threshold value, in multiples of the rms above the mean noise level
threshold float no default Threshold value in units of the image (if not given, the signal-to-noise threshold given by snrCut will be used instead.)
flagGrowth bool false Whether to grow detections to a lower threshold
growthCut float 3.0 Signal-to-noise ratio to grow detections down to
growthThreshold float 0.0 Threshold value to grow detections down to (takes precedence over growthCut)
flagNegative bool false Whether to invert the cube and search for negative features
flagRobustStats bool true Whether to use robust statistics when evaluating image noise statistics.
minPix int 2 Minimum number of pixels allowed in a detection
minChannels int 3 Minimum number of channels allowed in a detection
minVoxels int minPix + minChannels + 1 Minimum number of voxels allowed in a detection. Will be at least minPix + minChannels + 1, but can be higher.
maxPix int -1 Maximum number of pixels allowed in a detection. No check is made if value is negative.
maxChannels int -1 Maximum number of channels allowed in a detection . No check is made if value is negative.
maxVoxels int -1 Maximum number of voxels allowed in a detection. No check is made if value is negative.
flagAdjacent bool true When merging sources, whether sources need to be adjacent to be merged
threshSpatial float 3.0 When flagAdjacent=false, this is the spatial threshold (in pixels) within which objects are combined
threshVelocity float 7.0 When flagAdjacent=false, this is the frequency threshold (in channels) within which objects are combined
flagRejectBeforeMerge bool false A flag indicating whether to reject sources that fail to meet the minimum size criteria before the merging stage. Default behaviour is to do the rejection last.
flagTwoStageMerging bool true A flag indicating whether to do an initial merge of newly-detected sources into the source list as they are found. If false, new sources are simply added to the end of the list for later merging.
flagSubsection bool false Whether to look at just a subsection of the given image
subsection string full image A subsection string, e.g. [1001:2000, 50:1572, *, *] - only these pixels will be examined
flagStatSec bool false Whether to use a subsection of the image to calculate the image statistics
statsec string full image A subsection string indicating the pixel region to be used to calculate statistics (mean, rms,...)
flaggedChannels string no default A comma-separated list of channels and channel ranges (for instance, 1,2,5-9,13) that should be ignored for the purposes of detecting objects. Replaces the old (for Duchamp versions <1.5) flagMW/minMW/maxMW parameters.
beamArea float 10.0 The area of the beam in pixels. This parameter is only used when the image does not provide beam information. When this is used, a circular beam is assumed.
beamFWHM float -1.0 The FWHM of the beam in pixels. This parameter is only used when the image does not provide beam information. When this is used, a circular beam is assumed. This value takes precedence over beamArea but is ignored if negative (the default).
spectralUnits string no default The units desired for the spectral axis. If no value is given, the units in the image header are used.
spectralType string no default An alternative WCS type that the spectral axis is to be expressed in. If no value is given, the type held by the image header is used. The specification should conform to the standards described in Greisen et al (2006), although it is possible to provide just the first four letters (the ‘S-type’, e.g. ‘VELO’).
restFrequency float -1.0 If provided, this will be used in preference to the rest frequency given in the image header. If not provided, the image header value will be used if required.

Control switches

The following table lists parameters that control different modes of Selavy. They are not switched on by default.

Parameter Type Default Description
flagATrous bool false Use the à trous wavelet reconstruction algorithm prior to source-finding. See the Preprocessing page for details.
flagSmooth bool false Use spectral or spatial smoothing prior to source-finding. See the Preprocessing page for details.
recon2D1D bool false Use the 2D1D wavelet reconstruction algorithm (provided by WALLABY). See the Preprocessing page for details.
Weights bool false If true, scale the fluxes in the image by the normalised weights, to remove sensitivity varations. See the Thresholds page for details.
VariableThreshold bool false If true, use a sliding box to find the noise local to a pixel and set the (spatially-varying) detection threshold accordingly. See the Thresholds page for details.
optimiseMask bool false Whether to use the mask optimisation algorithm to optimally increase the size of each object. See the Postprocessing page for details.
extractSpectra bool false Extract a spectrum (to a CASA image) for each detected source. See the Extraction page for details.
extractNoiseSpectra bool false Extract a noise spectrum (to a CASA image) for each detected source. See the Extraction page for details.
Fitter.doFit bool false Fit Gaussian components to objects detected in a two-dimensional image. See the Postprocessing page for details.

Distributed processing

Description

The primary innovation in the ASKAPsoft implementation has been to allow distributed processing of images, to test the likely functionality of pipeline processing. The image is split up into subimages according to a user-specified scheme (the user provides the number of subdivisions in the x-, y- and z-directions). Neighbouring subimages can be overlapped by a certain amount (which may be desirable, particularly in the case of variable thresholds - see below).

Processing is performed under a master-worker framework, where a single master process coordinates the processing, and each worker handles a single subimage. Each of these subimages is searched independently, then the worker sends the list of detected sources to the master process. Once the master has accumulated the full set of detected sources, objects near the overlap regions are merged (if necessary) and have their parameters recalculated. The results are then written out.

Distributed processing parameters

Parameter Type Default Description
nsubx int 1 The number of subdivisions in the x-direction when making the subimages.
nsuby int 1 The number of subdivisions in the y-direction when making the subimages.
nsubz int 1 The number of subdivisions in the z-direction when making the subimages.
overlapx int 0 The number of pixels of overlap between neighbouring subimages in the x-direction
overlapy int 0 The number of pixels of overlap between neighbouring subimages in the y-direction
overlapz int 0 The number of pixels of overlap between neighbouring subimages in the z-direction
subimageAnnotationFile string selavy-SubimageLocations.ann The filename of a Karma annotation file that is created to show the boundaries of the subimages (see description below). If empty, no such file is created.

Output files

Standard Duchamp output

Selavy builds on the Duchamp library, and provides the ability to write standard Duchamp output files. Whether these are written is governed by the writeDuchampFiles parameter, which defaults to true. Standard Duchamp provides for flexibility in naming the output files it generates. They are summarised here, listed by the parameter name with the default value in square brackets.:

  • resultsFile [selavy-results.txt] - the list of detected sources and their parameters. Also includes (if flagSeparateHeader=false, the default case) a summary of the input parameters.
  • headerFile [selavy-results.hdr] - if flagSeparateHeader=true, this contains just the input parameter summary from the results file.
  • karmaFile [selavy-results.ann] - a Karma annotation file, showing the location of detected sources. This is produced when flagKarma=true.
  • ds9File [selavy-results.reg] - a DS9 region file, showing the location of detected sources. This is produced when flagDS9=true.
  • casaFile [selavy-results.crf] - a CASA region file, showing the location of detected sources. This is produced when flagCASA=true.
  • logFile [selavy-Logfile.txt / selavy-Logfile-Master.txt / selavy-Logfile-?.txt] - the logfiles, showing lists of intermediate detections (before the final merging), as well as pixel-level details on the final detection list. The first default listed is the default when running serial processing. The other two come from the distributed-processing case. In this case, the parameter’s value has either ‘-Master’ or ‘-?’ (where ? is replaced by the worker number, starting at 0) inserted before the suffix, or at the end if there is no suffix in the name provided. Only the master file (or the sole logfile in the serial case) has the pixel-level details of the final detections. These files will not be produced unless you set flagLog=true.
  • votFile [selavy-results.xml] - a VOTable of the final list of detections. This is produced when flagVOT=true (the default, unlike standard Duchamp).
  • binaryCatalogue [selavy-catalogue.dpc] - a binary format catalogue of detected sources that can be re-used by Selavy or Duchamp.

ASKAP-specific output

The following files are produced as a result of the new features implemented in the ASKAP source finder:

  • subimageAnnotationFile [selavy-SubimageLocations.ann] - a Karma annotation file showing the locations of the subimages used (see “Distributed Processing” section above). Lines are drawn showing the outer borders of each worker’s subimage, and each subimage is labelled with the worker number (starting at 1).

  • CASDA-suitable catalogues - in conjunction with the CASDA project (CSIRO ASKAP Science Data Archive), the following catalogues may be created:

    • continuum islands
    • continuum components (fitted 2D Gaussians)
    • spectral-line (HI) sources

    These are written in both ASCII (.txt) and VOTable (.xml) formats. These take their names from the resultsFile detailed above: replacing the .txt extension with islands.txt, components.txt and hiobjects.txt (so if the resultsFile parameter is selavy-results.txt, the islands catalogue will be selavy-results.islands.txt, the components catalogue will be selavy-results.components.txt and the HI emission catalogue will be selavy-results.hiobjects.txt). The VOTable equivalents will have a .xml extension. An example of the components catalogue can be found at Post-processing of detections, while an example island catalogue is shown below. The HI catalogue is only produced when HiEmissionCatalogue=true.

  • Fitting results - when Gaussian fitting is done for the continuum sources, several files can be written: a catalogue in ASCII & VOTable format (differing from the CASDA-format components catalogue), and annotation files showing the location of fitted components. These will be called selavy-results.fitResults.txt etc. To produce these files, set writeFitResults=true (default is false). See Post-processing of detections for details of the content of these files.

  • Images: when the variable-threshold option is used, the user can opt to write out relevant maps to CASA images. These include the noise map, detection threshold, and signal-to-noise ratio. These are described in more detail in Preprocessing for Selavy. Additionally, when the curvature-map option in the Gaussian fitting is used, the curvature map can be written to a CASA image - consult Post-processing of detections for information. Finally, whenever Gaussian fitting is used, maps of the fitted components and the residuals after subtracting them from the input image are always created.

  • Extracted spectra and images: Extraction of Spectra, Images and Cubelets describes various ways to extract data from the input image relating to individual detections. These can include integrated spectra, moment maps, cutout images or cubelets. All are saved to CASA-format images.

  • There will also be a log file produced by Selavy that contains the stdout logging information - this is described in Logging (note the difference with the Duchamp log file described in the previous section).

The island catalogue will look something like the following:

#                                        island_id    island_name n_components ra_hms_cont dec_dms_cont ra_deg_cont dec_deg_cont    freq maj_axis min_axis pos_ang flux_int flux_int_err  flux_peak mean_background background_noise max_residual min_residual mean_residual max_residual max_residual x_min x_max y_min y_max    n_pix solid_angle beam_area   x_ave   y_ave   x_cen   y_cen x_peak y_peak flag_i1 flag_i2 flag_i3 flag_i4                                                                                             comment
#                                               --                                                            [deg]        [deg]   [MHz] [arcsec] [arcsec]   [deg]    [mJy]        [mJy] [mJy/beam]      [mJy/beam]       [mJy/beam]   [mJy/beam]   [mJy/beam]    [mJy/beam]   [mJy/beam]   [mJy/beam]                                    [arcmin2] [arcmin2]
   image.i.ngc7232.cont.linmos.taylor.0.restored_1 J221655-452150            3  22:16:55.2    -45:21:50  334.229793   -45.363969  1400.5     0.41     0.40   16.87  399.155        0.000   1726.833           0.000            0.000        0.000        0.000         0.000        0.000        0.000  1445  1466  1708  1734       48        1.92      0.22 1456.00 1720.56 1454.54 1721.79   1454   1722       0       0       0       0
   image.i.ngc7232.cont.linmos.taylor.0.restored_2 J221021-454251            5  22:10:21.4    -45:42:51  332.589201   -45.714269  1400.5     0.73     0.42   43.78  253.272        0.000    766.953           0.000            0.000        0.000        0.000         0.000        0.000        0.000  1790  1809  1611  1627       59        2.36      0.22 1800.68 1620.15 1800.22 1619.95   1800   1620       0       0       0       0
   image.i.ngc7232.cont.linmos.taylor.0.restored_3 J221416-425710            1  22:14:16.8    -42:57:10  333.569943   -42.952993  1400.5     0.46     0.27   51.68  112.661        0.000    493.966           0.000            0.000        0.000        0.000         0.000        0.000        0.000  1582  1589  2445  2449       31        1.24      0.22 1585.55 2447.06 1585.47 2447.08   1585   2447       0       0       0       0
   image.i.ngc7232.cont.linmos.taylor.0.restored_4 J215840-471934            2  21:58:40.7    -47:19:34  329.669445   -47.326162  1400.5     0.63     0.40   86.95   87.930        0.000    329.294           0.000            0.000        0.000        0.000         0.000        0.000        0.000  2390  2400  1123  1128       41        1.64      0.22 2394.37 1125.68 2393.31 1125.52   2393   1126       0       0       0       0

The columns used are:

  • island_id is the unique identifier for the island. The ID string is made up of a scheduling block ID, the image name (the above example used an image called image.fits), followed by a unique identifier, which is just a numerical counter.
  • island_name is the “IAU-format” name taken from the J2000 position of the island’s centroid.
  • n_components indicates how many components were fitted to this island.
  • The position of the island is indicated by both HMS/DMS-formatted strings and decimal degrees for RA and DEC.
  • freq indicates the frequency of the image.
  • Estimates of the size and orientation of the island are provided with maj_axis, min_axis and pos_ang – these are not fitted values, but calculated by the Duchamp code based on detected pixels.
  • The integrated and peak fluxes are given by flux_int (with an error) and flux_peak*.
  • The background level and noise in the vicinity of the island are indicated by the mean_background and the background_noise.
  • The effectiveness of the Gaussian fits can be seen by the residual statistics - these are the residuals from subtracting the fitted Gaussians of the island from the image, and the statistics are evaluated using only the pixels within the island.
  • The pixel ranges are shown by x_min, x_max, y_min and y_max, with n_pix giving the total number of detected pixels.
  • The size of the island is also indicated by the solid_angle (in square-arcmin), with the comparable size of the restoring beam in beam_area.
  • Three estimates of the “centre” of the island are shown: x_ave & y_ave give the average pixel in each axis; x_cen & y_cen give the centroid, or flux-weighted-average; and x_peak & y_peak give the location of the peak pixel.
  • There are placeholders for four flags, but these are not yet used.