ASKAPsoft Release Notes¶
This page summarises the key changes in each tagged release of ASKAPsoft and/or ASKAP pipelines.
Pipeline patch releases for 2.10.x¶
A list of minor patch releases for version 2.13 of the pipeline scripts
2.13.1 (9/9/24) - A new optional parameter, plus several patches. The new parameter is
GRIDDER_WMAXCLIP
, which needs to be set to true to turn on the use of thewmaxclip
gridder parameter. The patches include:Minor changes to transient imaging parameters
Better consistency of the how beam labels are handled in bandpass processing, particularly when only a single beam is being processed.
Taking the ingest mode into account along with weights ID in verifying the match of science & bandpass observations
Robustness in casdaupload execution when archiving individual beam model images (catalogues were being missed for deposit)
2.13.2 (9/9/24) - Minor fix to the handling of the reference antenna number for bandpass & leakage calibration.
2.13.3 (9/9/24) - Fixing an issue with the default job priorities (high priority wasn’t being set properly)
Pipelines 2.13.0 (21/8/24)¶
A minor update to the pipeline scripts, making available the fast-imaging (or transient imaging) part of the pipeline. This is enabled by setting DO_TRANSIENT_IMAGING=true
, and results in the following steps for each beam:
Subtraction of the deep continuum model from the visibilities
Imaging of the continuum in pre-defined time intervals
Feeding of those images into VAST’s “vaster” transient-detection pipeline
The release also adds the possiblity of setting the slurm --cpu-freq
parameter for jobs, in the same manner as the job time, memory or priority.
Pipelines 2.12.0 (29/7/24)¶
A minor update to the pipeline scripts, that changes a number of imaging/gridding parameter defaults, and adds a new option to the external catalogue self-cal mode:
The gridding parameter changes make use of MPIWProject by default, and provide for a larger number of w-planes and a higher degree of oversampling.
The minorcycle thresholds in all imaging cases have been changed to signal-to-noise thresholds, typically 5-sigma threshold with a 0.5 sigma deep-clean threshold.
There are now specific options of “racs_mid” and “racs_low” that can be given to EXTERNAL_CATALOGUE, that allow the querying of the online sky model service to determine the local sky model used for initial calibration in the selfcal.
Pipeline patch releases for 2.11.x¶
A list of minor patch releases for version 2.11 of the pipeline scripts
2.11.1 (21/7/24) - Fixing bug in rapid-mode preimaging that was preventing the preflagging directives being applied
2.11.2 (25/7/24) - Correctly determine the FIELD_ID string when a specific field (that isn’t the first) is selected through the config file.
Pipelines 2.11.0 (11/7/24)¶
A minor update to the pipeline scripts, to allow the archiving of bandpass results as part of the regular pipeline processing. This includes both depositing into CASDA, and the option of copying the raw data to a location on the Acacia object store at Pawsey.
Earlier patches compared to 2.10.0 (already released in 2.10.x versions) include the following key additions (beyond various bugfixes):
Use of
scalemask
in imagingUse of FITS table output for extracted spectra in beam-wise source-finding
Improvements to single-OST use for spectral cubes during source-finding
A prototype of selfcal that uses an external catalogue to start with
Better parameters for the parallel-write-MS option
Better handling of modules, including singularity
ASKAPsoft patch releases for 1.17.x¶
These are the minor patch releases for askapsoft version 1.17
1.17.1 (16/5/24) - Fixing gitlab CI tests that were failing to run properly.
1.17.2 (20/5/24) - Patch to the selavy fix on the consistency of subdivisions & workers. Imager now uses a mask to handle overlaps of fields when running with offset imaging.
1.17.3 (11/6/24) - A couple of bug fixes for imager, avoiding it either hanging in continuum mode (where it stopped at a threshold), or crashing when run in DEBUG mode. Also a few updates to cmodel, to change the way the dcoffset term is applied and to make no PB correction if no PB parameter is provided.
1.17.4 (12/6/24) - A minor bug fix to correct the image names created by cmodel. Also adding a
usescalepixels
option to the imager1.17.5 (14/6/24) - A minor bug fix to cflagger to prevent Slicer errors.
ASKAPsoft 1.17.0 (13/5/24)¶
A release of the ASKAPsoft metapackage with new features and bug fixes:
Features:
Selavy source-finder:
Fix instances of output catalogues having negative RA values (e.g. RA=-30deg instead of RA=330deg)
Ensure consistency between numbers of subdivisions and worker ranks, exiting if they don’t match
Ensure noise maps etc that are written out in spectral mode have all channels filled, not just the first
Yandasoft
Performance improvements to the imager, particularly for continuum cube imaging, through refactoring and better handling of FFTs
Imager now has an option to add an initial (scale-specific) clean mask
Improvements to the self-calibration using offset images, allowing thresholds to be used from the initial image
Simplification of logging within cbpcalibrator to improve performance when run at large scale
Fixing race condition within cflagger & ccontsubtract when run in parallel
Linmos performance improvements
Sky model
Better performing XML interfaces used to read large catalogues into the sky model
Improved error handling for unit mis-matches when reading catalogues
Other
Improvements to the parallel write access to the DATA column in measurement sets
Test script for linmos performance
Pipeline patch releases for 2.10.x¶
A list of minor patch releases for version 2.10 of the pipeline scripts
2.10.1 (22/3/24) - Turning off the
clearcache
parameter in the cube imaging, as this wasn’t working properly in askapsoft/1.162.10.2 (27/3/24) - Minor bug fix for splitting bandpass data.
2.10.3 (22/4/24) - A few small changes:
allowing the use of the
scalemask
parameter in imagingallow the use of the FITS table output (use in spectral extraction) to be specified separately for the beam-wise source-finding compared to the final mosaic source-finding
propagating the holography footprint position angle to linmos & POSSUM validation
ensure the selavy directory is cleaned for pre-existing output images
2.10.4 (10/5/24) - Allowing the spectral cube to be copied to a single OST in the source-finding job, to speed up processing. The relevant pipeline parameter has changed from
CONTCUBES_ON_SINGLE_OST
toCUBES_ON_SINGLE_OST
.2.10.5 (17/5/24) - Ensuring the pol-imaging jobs have the
write.scalemask
parameter properly defined.2.10.6 (12/6/24) - Several minor changes & bug fixes:
changes to allow the use of
WRITE_PARALLEL_MS=true
, with job sizes etc set correctly and parameters properly progagated to the slurm jobsa prototype version of selfcal that starts with calibration against an existing source catalogue
modifications to the use of FFTW wisdom files, to deal with baremetal versions (getting libraries correct)
fixing job stats reporting for use with askapsoft/1.17.2
fixing job labelling when certain beams are stopped through the pipeline control flags
better handling of singularity versions, including a new -S command-line option to override defaults
2.10.7 (13/6/24) - A bug-fix to correct the use of readscalemask in the continuum cube & contpol imaging jobs. Also a minor fix to the legacy bandpass validation script to find the pickle files correctly.
2.10.8 (21/6/24) - Minor patch that unloads the askapsoft module prior to running beamwise convolution & possum validation, to avoid python conflicts when running with bare-metal builds of askapsoft on setonix
2.10.9 (25/6/24) - Additional option to allow archiving of all beam clean model images
2.10.10 (3/7/24) - A patch release addressing two issues and interface for a new feature:
Improvements to the single-OST copying done prior to source-finding, to avoid conflicts when done in per-beam mode
Patches to the POSSUM validation to ensure flag summary data is available to the validation scripts
hooks are in place to allow specification of pixel lists in the cleaning. This is not yet available in askapsoft/1.17.5 (current version at time of this release), but will be in the next one
Pipelines 2.10.0 (19/3/24)¶
An update to the pipeline code, focussed around performance and addressing a few flagging-related bugs:
Reduce the number of times the raw data is accessed - it is now read once to determine which MS corresponds to which beam + channel-range, and then a local data file is read for subseuquent jobs that require that information.
Adding a Stokes-V clip level to the flagging
Ensure the finaliseScience job will run properly on the setonix work queue (where it can’t see the askapbuffer filesystem), by moving copying tasks to earlier in the workflow.
Ensuring the beamwise source-finding results are archived whenever they are requested (previously it was only when POSSUM validation was turned on).
Ensure we properly track the various variables recording what flagging has been done for a given beam
Adding an additional flagging step to catch nearly-but-not-entirely flagged channels (particularly for time-split data)
Deprecation of the use of tmpfs (in line with recent askapsoft releases) in the imaging
Removal of legacy code, including moving some
acesops
python tools to the pipeline area
ASKAPsoft 1.16.1 (14/3/24)¶
A minor bug fix for askapsoft/1.16 to address an issue where the Cflag.selection_flagger.uvrange
option was not having any effect.
ASKAPsoft 1.16.0 (28/2/24)¶
A release of the ASKAPsoft metapackage with new features and bug fixes:
Features:
Implementation of offset-fields in imager, so that a number of additional fields at defined positions can be imaged in conjunction with the main image.
Improvements to direction dependendent calibration
Clean thresholds can now be specified in terms of the rms noise - e.g.
threshold.minorcycle=[30%, 5sigma, 0.5sigma]
Improvements to the image trimming that is applied in linmos, so that it works more robustly trimming all axes.
Improved handling of fftw wisdom files
Deprecation of the use of tmpfs in imager
Pipeline patch releases for 2.9.x¶
A list of minor patch releases for version 2.9 of the pipeline scripts
2.9.1 (5/2/24) - Removing the single-OST directory at completion of the selavy job
2.9.2 (12/2/24) - Fixing relaunch errors for references to a pipeline script that was no longer present.
2.9.3 (12/2/24) - Further fix for a typo in dependencies being passed to sbatch
2.9.4 (14/2/24) - Fixes for OH processing that were left out of the 2.9 deployment: ensuring the full-resolution leakage table has BPLEAKAGE_VALID values set correctly
2.9.5 (17/2/24) - Several minor changes: better behaviour for killAll.sh -x (relaunch=stop isn’t persistent); cleaning up lock files for pipelineControl; error checking for copies to guard against I/O errors; adding an explanatory PDF to the WALLABY validation directory; reducing repeated access to raw data through splitTimeRange.py; moving copying of images to single-OST directory to after their headers are updated
2.9.6 (18/2/24) - Fix to the handling of check parameters in the beamwise-convolution jobs
2.9.7 (22/2/24) - Update to the default CASDA deposit directory, following a CASDA release
2.9.8 (26/2/24) - Adding checks for the successful copy of bandpass PDFs
2.9.9 (26/2/24) - Getting correct dependencies between preImaging & preSpectralImaging (when continuum imaging isn’t done, as for the WALLABY MW processing)
2.9.10 (28/2/24) - Fixing a bug with the application of preflagging in rapid mode. Also distinguishing check parameters for contcube convolution
2.9.11 (29/2/24) - Ensuring the beamwise sourcefinding info is copied correctly when the extracted spectra tarballs are not present.
2.9.12 (29/2/24) - Fixes to the beamwise convolution control/check parameters.
2.9.13 (1/3/24) - Further fixes to the beamwise convolution control/check parameters.
2.9.14 (4/3/24) - Ensuring the amplitude flagging (and application of preflags) is correctly done for the averaged data
2.9.15 (5/3/24) - Ensuring the additional flagging step is done for averaged data, and fixing the way OMP_NUM_THREADS is specified for continuum imaging
Pipelines 2.9.0 (2/2/24)¶
An update to the pipeline code with changes to the structure of the finalisation jobs, to the way slurm jobs are submitted, and to the way modules are loaded in the slurm jobs. There is new handling of footprints to respond to the “align_to_beam” observing mode, and several parameter updates
Submission of slurm jobs is now abstracted into a submitJobs function, that allows for several retries should the submission fail.
The separate jobs for diagnostics, FITSconvert, makeThumbnails and validationScience have been merged in to a single finaliseScience job.
Bugfixes for the beamwiseCubeStats & beamwisePSFstats scripts to make them more robust to missing beams
Combining the gatherAll & launch scripts into a single launch script
Updates to the killAll.sh script to allow killing of just the pending jobs, and to prevent the relaunch of the pipeline.
Consolidation of module load commands at the start of slurm jobs and other minor fixes to avoid unnecessary output to the terminal.
New parameters
LINMOS_CONT_TRIMMING
(and same forCONTCUBE
&SPECTRAL
) to allow the use oflinmos.trimming
in the mosaickingHandling of a fully-flagged reference antenna in the leakage calibration, to mimic that done for the bandpass calibration (itself released in version 2.8.7)
Pipeline patch releases for 2.8.x¶
A list of minor patch releases for version 2.8 of the pipeline scripts
2.8.1 (20/12/23) - Fixing the dependency relationships between downstream tasks and the preimaging jobs.
2.8.2 (21/12/23) - Correcting the memory & priority slurm parameters for the pre-spectral-imaging jobs, and fixing the directory path when tarring the LinmosBeamImages.
2.8.3 (9/1/24) - Different approach to file-locking for the pipelineControl files. Setonix scratch has been mounted with localflock rather than flock, so cannot rely on flock to preserve the locks from different nodes. Instead use a mkdir-based locking scheme. Also fixing some minor issues with job states. The capability to use FFTW wisom files is incorporated (but needs an askapsoft release to be useful). RACS validation directories are now tarred properly (no absolute path to the directory).
2.8.4 (11/1/24) - Fixing dependency issue with flagAdditionalAveraged.sh so it comes after the pre-imaging job.
2.8.5 (11/1/24) - Fixing continuum sourcefinding dependencies, so that it properly picks up the mosaicking jobs (following the change in 2.8.0 of doing the linking in the linmos job)
2.8.6 (12/1/24) - Ensuring the FIELDBEAM is set correctly in the possum validation script (so it doesn’t change for downstream scripts)
2.8.7 (15/1/24) - If the nominated reference antenna for bandpass calibration is fully flagged in at least one beam, we switch to a different one. We also now make flagSummary files for the MSs used in the bandpass calibration (these are used in assessing the reference antenna suitability).
2.8.8 (25/1/24) - A change to how the heterogeneous jobs (used for continuum imaging) are run through srun, with the removal of the –ntasks-per-node parameter.
2.8.10 (28/1/24) - A patch to the pre-imaging script, to get the flagging of averaged data properly set up. (Note there was no 2.8.9)
2.8.11 (30/1/24) - A further fix to the heterogeneous jobs. The –ntasks-per-node is reinstated for the workers, with code in place to ensure this is correctly sized to no more than the number of workers (not total tasks).
Pipelines 2.8.0 (20/12/23)¶
An update to the pipeline code to provide a reduced slurm footprint and enhanced compatibility with RACS processing:
The jobs run before the continuum imaging, and before the spectral imaging, are now combined into a single slurm job. This reduces the overall number of jobs submitted for a given pipeline run, particularly when run in time-split mode
The linking of single-field mosaics to the top level directory is now done in the linmos slurm job itself. This reduces the need for additional jobs just to do the linking.
The copying of per-beam sourcefinding results (and spectra) is now done in the validationScience script, so that it is done when the POSSUM validation is not required.
A RACS/VAST validation script has been included, producing a number of CSV files in a directory that is tarred up and archived as an evaluation file.
The ccalibrator tasks now use $GRIDDER, so can respond to the use of MPIWProject (even though they are run serially).
A config_petrichor.sh file is added, to facilitate testing on the CSIRO cluster.
ASKAPsoft 1.15.0 (8/12/23)¶
An update to the askapsoft metapackage to allow a change in the MS data accessor. The visibilities in the MS (DATA & FLAG columns) are presented in (pol, chan, row) order, and this update changes the accessor classes (IConstDataAccessor.h) to access them in this way - rather than the previous (row, chan, pol) order.
ASKAPsoft 1.14.2 (1/12/23)¶
A further patch to askapsoft/1.14 to fix traditional weighting issues with joint deconvolution.
Pipeline patch releases for 2.7.x¶
A list of minor patch releases for version 2.7 of the pipeline scripts
2.7.1 (29/11/23) - Properly handling the BEAM variable when dealing with the beamwise-sourcefinding results in the possum validation script.
2.7.2 (1/12/23) - Ensuring we have the full path to the PolData directories when trying to remove them in the possum validation script.
2.7.3 (1/12/23) - Using the local version of the holography PB image cube in the possum validation script.
2.7.4 (1/12/23) - Scripting bugfix to improve some calling of the logJob function.
2.7.5 (15/12/23) - Properly defining the averaged MS names for the bandpass processing, so that the leakage calibration works properly.
2.7.6 (18/12/23) - Two key fixes for operational processing:
Adding includeflagged=false to the continuum imaging in circumstances where there are bad UVW values (as identified by ingest).
Copying continuum cubes onto a single OST to allow more efficient operation when extracting large numbers of spectra.
2.7.7 (18/12/23) - Further fix for the bandpass MS names, making things fully consistent across the processing.
Pipelines 2.7.0 (28/11/23)¶
Another major update to the pipeline scripts:
The pipeline is able to run predominantly on other slurm partitions (for instance, the work queue on setonix). Some tasks remain on the askaprt partition (as they will need access to the /askapbuffer filesystem), and the partitions are mediated through the
QUEUE
andQUEUE_OPS
variables.The pipeline better handles the averaging of channels, where it will read from the scheduling block parameter set how many fine channels make up 1MHz, and use that to determine the default value of
NUM_CHAN_TO_AVERAGE
.Data with
NUM_CHAN_TO_AVERAGE=1
will not have the averaging or its matching flagging job run (since the data is already averaged). This will naturally apply to data taken in continuum mode.Extracted spectra, particularly those done beamwise, are removed from disk once they are tarred up. If they are needed by the POSSUM validation job, they will be extracted from the tar files.
The interaction with the HIPASS catalogues for MS cutouts is improved, using files on disk instead of Vizier queries, and using more accurate velocity measures to better centre galaxies in the cutout window.
The
usefitstable
Selavy parameter is exposed through the pipeline parameters.
ASKAPsoft 1.14.1 (27/11/23)¶
A fix to askapsoft/1.14 that addresses a bug that was preventing the joint imaging working in conjunction with Doppler tracking.
ASKAPsoft 1.14.0 (22/11/23)¶
A release of the ASKAPsoft metapackage, with a number of new features and developments:
Better feature rejection in the imcontsub image-based continuum subtraction tool.
Enabling imager to make more than one image at once
A bug fix to address the zeroing of data when flagging bad solutions in ccalapply
Improvements to the table-based storage of extracted spectra
Various fixes related to both traditional weighting, and the implementation of MPIWProject (in serial mode).
Pipeline patch releases for 2.6.x¶
A list of minor patch releases for version 2.6 of the pipeline scripts
2.6.1 (10/11/23) - Bug fix for the logic of verifying the destination location for MSs being copied elsewhere.
2.6.2 (13/11/23) - Bug fix for additional channel-wise flagging for spectral data, so that the list of files gets resest for each beam.
2.6.3 (13/11/23) - Further bug fix for leakage calibration job.
2.6.4 (13/11/23) - Typo fixing in processDefaults.sh
2.6.5 (13/11/23) - Ensuring removeDir function calls are not run under an srun call
2.6.6 (14/11/23) - Fixing the POSSUM validation check parameters for consistency
2.6.7 (15/11/23) - Fixing bug in spectral msconcat where the wrong list of MSs was being used.
2.6.8 (16/11/23) - Further fix for the spectral msconcat, so it doesn’t concatenate the original MSs when they aren’t needed.
2.6.9 (20/11/23) - Fix to the POSSUM validation job submission command.
2.6.10 (21/11/23) - Ensuring the self-cal calibration tables are properly archived (they had been missed since the release of 2.6.0).
2.6.11 (23/11/23) - A bash-based version of the pipelineControl is now used (to avoid container/loop-device issues on setonix). Also minor fixes to spectral imaging check parameters, joint imaging parset and the dependency string for the MS cutouts.
2.6.12 (27/11/23) - Fixing script errors in the application of leakage calibration, particularly in the rapid mode.
2.6.13 (27/11/23) - Further fix to the POSSUM validation check parameters.
Pipelines 2.6.0 (9/11/23)¶
A major update to the pipeline scripts, adding a number of new features:
Spectral-line joint imaging of all beams is now available, through a new scienceCalim workflow, triggered by setting
DO_JOINT_IMAGING_SPECTRAL=true
.Full-stokes imaging is now possible for the spectral-line datasets. This includes on-axis leakage calibration by adapting the coarse-resolution leakage table to full spectral resolution.
Doppler correction can be done in mssplit (the job used to make a spectral MS to be imaged), with the option of interpolation to improve the accuracy.
The MPIWProject gridder may be used in all gridding stages that are run as parallel jobs.
Improvements to ccontsubtract, allowing the use of uvlin.
Providing the option of copying the spectral-line MSs (either pre- or post-contsub) to a defined place on disk.
Ensuring the tarred-up extracted spectra created by selavy are deleted once the tarring has completed.
Archiving the diagnostics directory (tarred up) as an evaluation file in CASDA, alongside the usual calibration-metadata-processing-logs file
Use of check parameters with validation scripts to prevent unnecessary re-runs.
ASKAPsoft 1.13.0 (20/9/23)¶
A release of the ASKAPsoft metapackage, aimed primarily at correcting the treatment of Stokes Q and U in linmos. Changes made include:
renormalising the off-axis correction by subtracting off the on-axis leakage (which will already have been applied to the data before imaging)
transforming Q and U leakage corrections for the rotation wrt to sky frame when the beam is rotated to match the holography with a specific pol_axis setting
ASKAPsoft 1.12.0 (15/8/23)¶
A more major release of the ASKAPsoft metapackage:
Cflag & Ccalapply have parallel capabilities enabled
Selavy has the capability of writing out extracted spectra to a single FITS binary table (rather than many individual FITS files)
Bug fixes for imager (nyquist gridding normalisation and use of ‘combinechannels’ parameter) and cdeconvolver-mpi (MPI distribution).
Further work on a tool to trim blank edges around images
Bug fix in casdaupload to present correct floating-point precision field centre values.
Improvements to the sky model service query interface
ASKAPsoft 1.11.1 (17/5/23)¶
Bug fix release for ASKAPsoft to address some issues of slow imaging jobs. Required restricting the image sizes to avoid large prime factors.
Pipeline patch releases for 2.5.x¶
A list of minor patch releases for version 2.5 of the pipeline scripts
2.5.1 (15/5/23) - Ensuring the completion of splitting jobs for spectral bandpass processing is recorded properly. Also that the casda polling job is properly launched on setonix.
2.5.2 (16/5/23) - Ensuring that all bandpass jobs, even those run after the bandpass calibration task, are run at high-priority by default.
2.5.3 (19/5/23) - Allowing job-dependent memory specification, so that multiple jobs can run on the same node on setonix. Also fixing the determination of pipeline errors in the gatherStats job, so that they properly get aggregated and copied elsewhere for later assessment.
2.5.4 (24/5/23) - Allowing the EMU validation script to use or not use the component residual image to calculate the overall RMS noise level. Default is to not use it.
2.5.5 (2/6/23) - Adding an environment variable FI_CXI_DEFAULT_VNI to slurm scripts to help address an I/O problem.
2.5.6 (14/6/23) - Several changes:
Default values of memory for slurm scripts run on setonix, applied largely to pre-imaging jobs to allow them to share compute nodes.
A copy script that is run prior to casda upload, to copy data from /scratch to /askapbuffer.
More robustness in making the yaml file given to CLINK (to ensure it is complete).
Various fixes for check and control parameters, as well as job dependencies, to address issues seen in operations.
2.5.7 (15/6/23) - A further check parameter fix (for spectral splitting).
2.5.8 (19/6/23) - Ensuring the observation.xml.checksum is correctly made in the copying script introduced in 2.5.6. Also fixing a bug that meant the preflagging directives were not being applied in the science flagging.
2.5.9 (19/6/23) - Ensuring the SEFD data is properly copied to the diagnostics directory.
2.5.10 (20/6/23) - Turning off the calls to ‘lfs setstripe’ when running in a directory under /scratch (so that we instead use the default striping setup).
2.5.11 (23/6/23) - Turning off the copy to /askapbuffer prior to CASDA deposit, as the CASDA servers can now see /scratch.
2.5.12 (27/6/23) - Better check parameter handling for flagging, and protection of summary plots when using multiple templates.
2.5.13 (27/6/23) - Correcting the determination of parent weights.
2.5.14 (27/6/23) - Further fix to summary plot names
2.5.15 (30/6/23) - Further fix to summary plot names
2.5.16 (1/7/23) - Further fix to flagging check parameters
2.5.17 (4/7/23) - Further fix to flagging check parameters
2.5.18 (5/7/23) - Further fix to flagging check parameters for rapid mode.
2.5.19 (27/7/23) - Avoid removing time-window MSs before the averaging is complete.
2.5.20 (4/9/23) - Add a check for $HOME at the start of slurm scripts.
2.5.21 (5/9/23) - Update the singularity version for setonix
2.5.22 (8/9/23) - Fix for exclusive node jobs with upgraded setonix, by specifying number of nodes directly.
2.5.23 (8/9/23) - Further update for exclusive node jobs
2.5.24 (12/9/23) - Updated bptool needed pipeline changes; the casdaupload job needed a check parameter; and dealing with a corner-case in finding the parent weights (ref_ws=-1)
2.5.25 (12/9/23) - Fix to the parent weights determination.
2.5.26 (12/9/23) - Fix for dealing with the bptool update.
Pipelines 2.5.0 (12/5/23)¶
A pipeline release aimed at providing improved performance to support moving the operational processing to setonix.
Pipeline changes to the workflow functionality include:
Breaking up the pipelineControl.in parset into a series of smaller parsets, one per beam, that gives better efficiency particularly when launching the pipeline.
Improved CLINK notifications, including for job failures, job timeouts, and pipeline restarts.
The
#SBATCH --clusters=
directive has been removed from the slurm scripts, to provide for more reliable performance.The priority level for jobs is able to be changed from job to job, with the final jobs of the workflow (validation etc) having high priority by default to allow a pipeline run to finish off as soon as possible.
The generate-preflags task is now done outside of a slurm script, to reduce the impact on the slurm queue and the dependency management.
Output from beamwise-sourcefinding is now put into distinct directories, to allow better use by downstream users.
The schedblock metadata file is now used if there is one already in place, rather than regenerating it each time.
Changes to input parameters include:
There’s a new parameter
REMOVE_LEAKAGE_OFF_AXIS_STOKES_V
that allows the Stokes V mosaic to be made without the removeleakage option, even when the other Stokes are made with it turned on.The direction used for doppler tracking in spectral imaging can be specified by the
DOPPLER_TRACKING_DIRECTION
parameter.
Other small changes include:
The
common.target.src%d.beam_list
parameter can be read from the SB parset, and used to limit which beams are processed.Updates to the use of check parameters for various tasks including imaging and beamwise-convolution.
Improvements to the building of the catalogue used for MS cutouts - ‘HIPASS’ now includes all three HIPASS catalogues, so that we can look north of the equator.
The WALLABY validation script makes use of the new aces-apps module, which allows the use of SEFD data when available.
Fixes were made to the EMU validation script so that the correct residual image is passed to it, and so that the raw image is correctly defined.
The POSSUM validation script has been updated to use the more recent version of POSSUM’s script.
The capability of interpolating self-cal solutions in time is exposed through the pipeline, although not used by default.
Minor fixes for the stats plot appearance
ASKAPsoft 1.11.0 (03/04/23)¶
An update to the askapsoft metapackage, with the following key updates to yandasoft (now version 1.10.0):
The imcontsub tasks can now do I/O in “individual” mode, where each rank is done in sequence. While slow it gets around the collective I/O MPI bug seen in containers on setonix.
Fixing an mssplit change that led to table conformance errors
Mssplit can now do spectral frame conversion while splitting or rebinning data, including interpolation to improve the accuracy.
Selavy bug fixes: better rejection of not-a-number entries in catalogues; fixing calculations to properly handle circular components when calculating the deconvolved sizes.
Fixes addressing a comms issue that could lead to stalled/hung imaging jobs
Improvements to joint imaging performance & memory usage.
Improvements to cdeconvolver to work better importing UV grids
Adding a spheroidal taper to the AProjectWStack gridder
Underlying interfaces - handle uv-weigthts, and adding a class to calculate weights for robust weighting.
Fixing conversion to HPX in linmos
Pipeline patch releases for 2.4.x¶
A list of minor patch releases for version 2.4 of the pipeline scripts
2.4.1 (20/12/22) - Additional fix for case where no jobs were submitted.
2.4.2 (21/12/22) - Fixing a typo in the final casda submission job
2.4.3 (21/12/22) - Fixing up plots: making sure the contcube cubestats plots are produced, and fixing the beam numbering for the flag summary plots.
2.4.4 (22/12/22) - Robustness to slurm failures for the submit_casda job.
2.4.5 (25/12/22) - Beamwise PSF plotting was failing in the presence of NaN values.
2.4.6 (25/12/22) - Fixing the job parameters from piplineControl.in passed to gatherAll
2.4.7 (30/12/22) - Fixing python errors in POSSUM validation.
2.4.8 (20/1/23) - Robustness to slurm failures for the CASDA polling job; better handling of jobid parameters in pipelineControl.in, and new check parameters for splitting & averaging.
2.4.9 (23/1/23) - Ensuring the MS cutouts are included in the CASDA deposit.
2.4.10 (25/1/23) - Getting the contcube weights mosaic links correct.
2.4.11 (26/1/23) - Fixing a typo in the CASDA polling job
2.4.12 (2/2/23) - Another typo fix
2.4.13 (6/2/23) - Robustness in determining pipeline restarts, and additional checks for missing beams in the selfcal plotting.
2.4.14 (6/2/23) - Typo fix to 2.4.13
2.4.15 (11/2/23) - Ensuring MS cutouts are included in the observation.xml
2.4.16 (16/2/23) - Fix missing preconditioning parameters in the contpol imaging parset. Also remove jobids from the control parset when restarting.
2.4.17 (28/2/23) - Several fixes:
Use the new dopplertracking parameter in the spectral imaging
Improve the use of alternative catalogues for MS cutouts, and add robustness to finding no sources.
Updating the EMU validation script to the latest version
Use checkparameters for msconcat.
2.4.18 (7/3/23) - Ensure linmosFieldsCont knows about the alt image extension
2.4.19 (8/3/23) - Fixes to ensure multi-field linmos jobs run without error
2.4.20 (10/3/23) - Robustness for the GASKAP validation dir, changing the parameter to
GASKAP_VALDIR_BASE
(fromGASKAP_VALDIR
)2.4.21 (13/3/23) - Ensuring thumbnail plotting & catalogue handling can work with Galactic-coordinate images.
2.4.22 (22/3/23) - Update the dopplertracking to use a direction parameter, properly crafted to be the target direction.
2.4.23 (23/3/23) - Exposing the new parameter imcontsub.imageaccess (available from askapsoft/1.11 onwards) through the pipeline config.
2.4.24 (23/3/23) - Further fix for imcontsub.imageaccess.
2.4.25 (25/3/23) - Making the holography weights check work for the new format of parset parameters (look for “holography.target_ws”)
2.4.26 (1/5/23) - Removing already-existing MSs prior to running the initial split/merge/copy jobs at the start of the pipeline.
2.4.27 (1/5/23) - Further patch to 2.4.26
Pipeline 2.4.0 (20/12/22)¶
A pipeline release aimed at improving the performance of restarts. When slurm errors stop the running of processASKAP, the launch script is now re-submitted, so that the pipeline processing can continue after all jobs that were submitted are done.
Pipeline patch releases for 2.3.x¶
A list of minor patch releases for version 2.3 of the pipeline scripts
2.3.1 (19/12/22) - Various fixes to the validation scripts, particularly getting the directories correct as well as the use of modules. Also turn off their archiving when the relevant
ARCHIVE_IMAGE_xxx
parameter is off.2.3.2 (19/12/22) - Preserving the SB_PB parameter for subsequent pipeline runs, and ensuring robustness to slurm failures for the CASDA polling job.
2.3.3 (20/12/22) - Fixing the final observation.xml file for multiple templates, and further fixes to the archiving when
ARCHIVE_IMAGE_xx
is off.
Pipeline 2.3.0 (13/12/22)¶
A small pipeline release to better handle the renaming of images, particularly how that affects the continuum-subtraction jobs. This includes tidying up how the dependencies between jobs work.
Pipeline patch releases for 2.2.x¶
A list of minor patch releases for version 2.2 of the pipeline scripts
2.2.1 (6/12/22) - Minor patches to validationScience.sh
2.2.2 (7/12/22) - Fix to the logic for the selfcal check params
2.2.3 (7/12/22) - Move the EMU validation script to use RACS for the comparison. Also fix the checksums in the final casdaupload job, and extend the job time for the polling.
2.2.4 (8/12/22) - Minor changes to launch job time and fixing typos.
Pipeline 2.2.0 (6/12/22)¶
A smaller pipeline release, incorporating new check parameters governing the self-calibration, to prevent it re-running when the images are renamed. Also some further fixes to the parameters in the validation scripts, following 2.1.3.
Pipeline patch releases for 2.1.x¶
A list of minor patch releases for version 2.1 of the pipeline scripts
2.1.1 (2/12/22) - Setting the module directory for galaxy work.
2.1.2 (4/12/22) - Couple of CASDA-upload fixes: setting the correct image type for the ‘raw’ images, and getting the READY file written properly.
2.1.3 (5/12/22) - Fixing parameters in the validation scripts, and removing the use of the ‘bc’ tool
Pipeline 2.1.0 (2/12/22)¶
A further consolidated update to the pipeline scripts:
We now catch slurm submission errors and exit the pipeline, emitting a CLINK notification in the process.
The MS cutout script now allows the use of a CSV catalogue file as the base catalogue to query.
Better checking for partial completion of selfcal - we need to re-start the selfcal process from the beginning upon a pipline re-run.
Ensure the beam-convolution doesn’t start before all of the source-finding jobs have finished, to avoid issues with the image renaming.
Better handling of the singularity commands in a number of scripts.
Adding ‘raw’ to the list of images for which we get cubestats
Fix to the creation of the final observation.xml file
Adding additional filter input to the EMU validation script.
Ensure the FIELDBEAM variable is correctly unset at the end of the rapid processing.
Pipeline patch releases for 2.0.x¶
A list of minor patch releases for version 2.0 of the pipeline scripts
2.0.1 (18/11/22) - Default values given for singularity modules for galaxy & setonix.
2.0.2 (20/11/22) - Main change: keep the .conv extension unless otherwise told. Also fix up issues reading templates when the SB_SCIENCE is not given, and reordering the application of cluster config files.
2.0.3 (21/11/22) - Ensure askappy version is determined early enough in the workflow.
2.0.4 (21/11/22) - Fix to formatting of QOS in slurm files.
2.0.5 (22/11/22) - Several small bug fixes:
Issue reading the schedblock information when SBs not given on command-line.
Formatting of launch script to get the command-line arguments correct.
Prevent the gatherAll job being added to the kill script
2.0.6 (23/11/22) - Defining the default location of the holography PB images.
2.0.7 (1/12/22) - Re-enabling –cluster and –exclude in the slurm scripts.
Pipeline 2.0.0 (18/11/22)¶
A major update to the pipeline scripts, providing improvements in functionality and performance. This provided readiness to start the trial of the full-scale ASKAP surveys
Key features:
Flexibility is introduced to allow the pipelines to run on different clusters, by allowing for different configs based on the
CLUSTER
parameter (which should be set in the environment, not the config/template).The slurm scripts are now able to make use of containers for the askapsoft and other modules.
The pipeline allows for automatic restarts upon completion of all launched jobs. This looks at the outcomes of the jobs that have run, and if there were failures then the processASKAP call is re-run
This is facilitated by a pipelineControl.in parset, that records the state of jobs that are either queued, running, or completed (either failed or ok). This parset also replaces the checkfiles that were stored as empty files on disk - these are now ‘check parameters’, stored as parset parameters. There is a python utility function pipelineControl.py that provides for interactions with the parset.
When all pipeline jobs are completed, the pipeline emits an event via CLINK.
Multiple processing templates/config files can now be used for a single SBID, by giving
SUBSEQUENT_TEMPLATE
and/orSUBSEQUENT_CONFIG
. Once all processing for the first run is complete, a processASKAP run using the second template/config will be started.The pipelineControl.in parset also provides for go/no-go switches at a beam level, although these are not used as yet (the aim being to turn off processing for a beam should it fail basic QA tests mid-pipeline-run).
The filenames for the holography primary beam images are now determined automatically, given a holography SBID. This can be provided through the new “-p” command-line option to processASKAP.sh.
The common-resolution images are now considered the primary image product, with the non-convolved images renamed with a “.raw” suffix.
The alternate preconditioner images now have their suffix able to be specified via the
RESTORE_PRECONDITIONER_EXTENSION
parameter (which defaults to ‘alt’, the previous behaviour). We also now make a common-resolution (across beams) version of this alternative image.There is new functionality to provide cutouts in frequency of a selection of channels for given measurement sets. These selections correspond to a specified catalogue of sources, having an RA/Dec position and a velocity. Such cutouts are archived in CASDA.
Image headers are updated by imager & linmos using the recently-released functionality. We also store the target name and its RA/Dec position as header keywords.
The slurm jobs can now be given a QOS (quality of service) level, via
QOS
.The casda_ingest_success polling jobs no longer use zeus, but rather the same cluster as all other jobs.
There is an option (by setting
ARCHIVE_CONT_MS_FULLY_CALIBRATED=false
) to archive the pre-selfcal averaged MSs to CASDA instead of the calibrated ones.The EMU continuum validation script is now run as a separate validation job (much like the other SST validation tasks), instead of at the end of the continuum source-finding.
ASKAPsoft 1.10.0 (15/11/22)¶
An update to the askapsoft metapackage, with the following key updates to yandasoft (now version 1.9.0):
Allows for the ability to interpolate calibration solutions in time, at point of application. This is with the new parameter
calibrate.interpolatetime=true
(default is false).Allow for an image-domain spheroidal taper to be applied to the A-function during gridding and degridding.
Additionally, some improvements and refactoring of EstimatorAdapter in base-scimath, and fixes to failing CI tests for askapparallel.
Pipeline 1.10.0 (4/11/22)¶
Selavy can now write out Galactic coordinates to the catalogues. Set
SELAVY_INCLUDE_GALACTIC_COORDS=true
to have it done. Also, ifLINMOS_DIRECTION_TYPE=GALACTIC
this will be turned on automatically.We make use of the new interface for spectral extraction in Selavy, available from askapsoft/1.9.0.
For uploads to CASDA, the
OBS_PROGRAM
variable now defaults to a blank string, but must be filled in if uploads are to happen.The validation report directories that are sent to CASDA are now much smaller in terms of size on disk - only the elements within the diagnostics directory (typically the plots) are kept for archiving purposes.
We no longer make copies of the continuum validation reports in a space on /group (for later archiving at ATNF) - since we are more production-oriented this has been replaced by CASDA.
Subsequent patches for 1.10:
1.10.1 (8/11/22) - ensuring the full paths to the evaluation files are used for casdaupload
ASKAPsoft 1.9.0 (2/11/22)¶
An update to the askapsoft metapackage, which incorporates the following releases of key subpackages:
askap-analysis 1.6.0:
The interfaces to the extraction classes have changed, so that it makes better use of the base-accessors package. From the parset point-of-view, this means parameters like
cube
andspectralCube
that indicate what file to extract from should now not give the “.fits” extension when FITS files are used. Theimagetype=fits
should also be given in that situation.Related bug fixes include:
Properly accounting for the size of the spectral cube being extracted from, so that if the source falls outside its bounds then it is ignored.
Better handle the case of the restoring beam not being present in the spectral cube.
Selavy will now handle Galactic-coordinate images. Positions of detected sources will be converted to RA/Dec for output into the catalogues, with the COOSYS in the VOTables hard-coded to J2000. There is a new parameter
catalogueHasGalacticCoords
(defaulting to false) that allows the additional writing of Galactic coordinates to the catalogues.
yandasoft
Pipeline patch releases for 1.9.x¶
A list of minor patch releases for version 1.9 of the pipeline scripts
1.9.1 (9/8/22): Minor fixes to the reporting of the dependencies for some linmos jobs (the actual dependencies used by sbatch were correct).
1.9.2 (17/8/22): Copying of beamwise source-finding results (catalogue & spectra tar files) to the POSSUM validation directory for later use.
1.9.3 (6/9/22): Several minor fixes:
Making use of the updated preflagging, to provide more information on the fitting and channel-rejection thresholds, thereby allowing channel-dependent flags to be included in the preflag directives.
Separating the contcube linmos tasks to have one Stokes per job.
Allowing the option of parallel writing in the contcube linmos jobs.
1.9.4 (8/9/22): Fixing the handling of bandpass/leakage tables & the preflagging, so that the tables and their plots are copied to the local BPCAL directory first, and the flagging directives generated therein.
1.9.5 (8/9/22): Ensuring the wildcards in the bandpass/leakage table names are resolved properly.
1.9.6 (20/9/22): Ensuring all calibration tables are properly archived as part of the calibration-metadata-processing-logs tar file, in their own directory called “CalibrationTables”.
1.9.7 (21/9/22): Adding an additional flagging step in the bandpass processing to flag channels that are above some percentage flagged (defaulting to 50%).
1.9.8 (7/10/22): Two minor changes: the additional flagging threshold level for bandpass data is increased to 60%; a fix for the determination of linmos offsets when using IMAGE_AT_BEAM_CENTRES=false
1.9.9 (18/10/22): Adjusting the way the beam-wise source-finding is treated for POSSUM validation. There is a new parameter to change the polSNR threshold governing how many source spectra are written, and the outputs are now in an ‘evaluation’ tar file separate to the validation report directory.
1.9.10 (28/10/22): Fixing a bug with the casda upload script, ensuring that the spectral types (for extracted spectra) are assigned correctly. Also removing explicit “module load askapenv” instances from the scripts, as they are not needed when askapsoft is loaded.
Pipeline 1.9.0 (5 August 2022)¶
The pipeline has the following new features:
Linmos can create mosaics with different direction or projection types, via the new parameters
LINMOS_DIRECTION_TYPE
andLINMOS_PROJECTION_TYPE
.The ability to average channels together in the spectral dataset prior to spectral-line imaging. This is done via the mssplit job that is used to split out a nominated channel range.
Bugfixes:
Spectral sub-bands (when more than one cube is being created by the spectral or contcube imaging) were not being named or handled correctly.
BPCAL metadata files were not getting the “good” data flag set, which meant they got regenerated each time, causing problems when the raw data was not available.
The job dependencies for the contcube imaging did not correctly depend on the application of selfcal gains (following fixes in version 1.8.0)
ASKAPsoft 1.7.1 (3 August 2022)¶
A bug-fix for linmos that ensures the change of direction or projection type applies for mosaics of mosaics.
ASKAPsoft 1.7.0 (19 July 2022)¶
An update to the askapsoft metapackage, which incorporates the following releases of key subpackages:
Yandasoft 1.6.0:
Improved handling of metadata tables in FITS images, following the initial work in ASKAPsoft/1.6.0
Improvements to the memory usage within the minor cycle deconvolution, with FFT optimisation
Tests of multi-threading support for FFT (not yet in production). Also affects base-scimath 1.7.0.
Work to further improve joint-deconvolution, to add nyquist-gridding support and fix issues with the way the mosaic direction is specified.
Bug fix: Output of multiple casa-format cubes is now possible from imager
Bug fix: Residual & model images when nyquist-gridding was used were coming out incorrectly-sized.
Imagemath 1.6.0:
Improve the
outputcentre
parameter to accept a description of the direction and projection types, to allow the output of the mosaic in different frames (for instance HPX or GALACTIC).Handling within linmos of the metadata tables introduced in ASKAPsoft/1.6.0
base-askap 1.5.0:
Thread-safety for the formatting of the print command for directions
askap-pipelinetasks 1.7.0:
Allow cmodel to apply a primary beam model consistent with that used by linmos to both the output image, and to a list of components that can be written out for further use.
base-accessors 1.6.0:
Provide a standard interface for reading & writing tabular data to/from FITS files (as extensions).
askap-analysis 1.5.0:
Making use of the new image history interface.
Other:
base-components and askap-analysis were patched to deal with interface changes made in casacore
Pipeline patch releases for 1.8.x¶
A list of minor patch releases for version 1.8 of the pipeline scripts
1.8.1 (26/5/22): Changes to the UV grid export pipeline script to separate the single-node finalisation tasks from the actual export (and cube-stats) calculation, aimed at improving the overall pipeline efficiency.
1.8.2 (15/6/22): Fixing job dependencies for the combineFlagSummaryAveraged job, so that it runs at the appropriate time and not straight away.
1.8.3 (23/6/22): Changing the default CASDA staging area to /askapbuffer/casda/prd (from /group).
1.8.4 (1/7/22): Fixing a bug that was overwriting the selfcal parsets with the subsequent ccalapply parsets.
Pipeline 1.8.0 (19 May 2022)¶
A pipeline update largely around a few bug fixes and consolidation of slurm jobs:
The rapid mode was still using the on-the-fly derivation of preflag directives, which meant manually-added flags were not being picked up. The behaviour is now consistent with the standard mode.
Also, the flagging in the rapid mode was made more robust so that if no rules are specified the amplitude flagging task is turned off (as for standard mode).
The numerous per-beam jobs for application of the selfcal & leakage calibration, plotting of the gains, and running flag summary, have been consolidated either into the imaging/selfcal job or into the final combine-plots jobs. This will greatly reduce the number of individual slurm jobs submitted to the queue for a given SBID.
The assignment of project codes in the source-finding job has been fixed to be consistent with the change in 1.7, where all contcubes go to the polarisation project.
The option to use nyquist gridding for the spectral & contcube imaging has been turned off by default (while a bug with that mode of imager is worked on - this applies to askapsoft/1.6.0).
Pipeline patch releases for 1.7.x¶
A list of minor patch releases for version 1.7 of the pipeline scripts
1.7.1 (11/4/22): A small addition to the pipeline to allow generation of cube statistics and plots for the exported UV grid cubes. These are stored within the diagnostics directory.
1.7.2 (12/4/22): A further addition to speed up the cube statistics generation for the UV grids, removing the robust statistics (now controllable via an option for findCubeStatistics.py) and parallelising the gzipping of the FITS files.
1.7.3 (13/4/22): A bugfix to ensure that image names used in the science validation scripts are correctly formed. Also fixing the UVgrid export job to correctly use the stats checkfile, and to ensure the job name written to the stats-all file is not too long.
1.7.4 (15/4/22): A minor fix to ensure that imager failures in the UV grid export are properly logged in the stats files.
1.7.5 (22/4/22): Recovering the parallelism of the cubeStats tasks for the UV grid export, since the NCORES/NPPN get reset in updating the FITS headers
1.7.6 (23/4/22): Further UV-grid fix : ensuring the .hdr files are cleared away prior to creating the header and stats and tarring the directory.
1.7.7 (27/4/22): Turning off the UV grid export from within the spectral imaging job, unless explicitly asked for by using UVGRID_EXPORT_WHEN_IMAGING
1.7.8 (28/4/22): Fixing errors in the job stat reporting within the UVgrid export task
ASKAPsoft 1.6.0 (11 April 2022)¶
An update to the askapsoft metapackage, which incorporates the following releases of key subpackages:
Yandasoft 1.5.0:
Improvements to the image metadata handling in imager & linmos:
Additional header keywords are written at file creation: TELESCOP, PROJECT, SBID, DATE-OBS, DURATION
History entries are added and supported
Capability to store arrays in the metadata - such as PSF as a function of channel
The beam table is now able to be written to the header instead of to a separate beam log
Fixing a bug in imcontsub where occasional channels could be skipped when operating with chunks on barycentric-frame data.
Patching delaysolver to better handle difficult or bad data
Correcting the weightstate value within linmos for the joint deconvolution case (to WEIGHTED)
Some fixes to improve the memory usage within the spectral-line mode of imager
Imager now writes the scales and the cleaned flux per scale to the log as INFO level messages.
askap-pipelinetasks 1.6.0:
casdaupload is able to handle multiple MS input (to cater for UV grids), by combining into a single block in the XML file
cmodel has a ‘nearest’ parameter to switch from nearest-neighbour allocation to a lanczos-interpolation
mssplit was fixed to avoid reading an entire row just to get nPol
askap-analysis 1.4.0:
Selavy’s interface to read spectral terms (to measure the spectral index & curvature of components) has changed, so that only nterms need be given and the input parset doesn’t need to specify explicitly which images to read from.
Other general fixes include:
Improvements to the version reporting in each of the tools within askapsoft
Pipeline 1.7.0 (11 April 2022)¶
A range of pipeline updates:
The major update for this version is the ability to create and archive the spectral UV-gridded visibilities. These will be written to directories named similarly to the spectral MSs, and archived alongside them to CASDA.
There is now the option to automatically use Nyquist-sampled gridding cell sizes. This will not affect the oversampling chosen for the output image.
Using the changed selavy interface for specifying the spectral terms to use in spectral index/curvature calculations, whereby we only give the number of terms to be used.
The PSF recorded in the FITS headers of the convolved cubes corresponds to the same frequency channel as that of the PSF recorded in the headers of the input image cube.
Introducing new parameters to allow the use of Cmodel.nearest in both selfcal and continuum-subtraction.
The SEFD data, where produced for the bandpass calibration observation, is now copied to the diagnostics directory for future reference.
The Stokes-I continuum cube will now be archived with the project indicated by POLARISATION_PROJECT_ID (which defaults to CONTINUUM_PROJECT_ID if not given)
Bugfixes:
The casda.sh script would hang when MS_METADATA was provided in the config file.
The beam convolution was failing on channels where all input beams were completely flagged (this entailed a patch to the convolve module).
There is now clearer handling of the Cmodel.flux_limit parameter across all tasks that use it.
Pipeline 1.6.9 (25 March 2022)¶
A further small fix to update the checksums made for the spectral ms.tar files, so that they use the MD5 sum in accordance with the latest CASDA practice.
Pipeline 1.6.8 (24 March 2022)¶
A small fix correcting a bug with the mode triggered by BANDPASS_FLAG_HI_ABSORPTION, where the affected channels were not properly being identified.
Pipeline 1.6.7 (22 February 2022)¶
A small addition to the pipeline scripts, providing parameters that turn on the usemedian functionality in mssplit when creating the continuum MS. These options are off by default.
Pipeline 1.6.6 (21 February 2022)¶
A fix to the pipeline for selfcal & contsub “Components” methods - the reference direction provided in the component parset was incorrectly set to the pointing centre, rather than the beam centre. This has been rectified and is now consistent with the other equivalent methods.
Pipeline 1.6.5 (17 February 2022)¶
Several small fixes to the pipeline scripts:
The setting of DATE-OBS in the restored cube header at the start of the imcontsub slurm jobs is now only done when it is not already present (or the image is being converted from CASA format).
Corrections were made to the calculation of the bandpass smoothing windows, to take into account the total number of channels - this was a problem largely for zoom-mode datasets.
The BPCAL directory is now ensured to be present prior to the creation of this window file.
ASKAPsoft 1.5.0 (8 February 2022)¶
A new version of askapsoft that only changes the casdaupload utility. The checksums generated are now MD5 checksums, in accordance with requirements from CASDA following the move to the new Acacia storage system at Pawsey. No other applications have changed from 1.4.1.
Pipeline 1.6.4 (4 February 2022)¶
Adding some reporting in slurm outputs of the pipeline scripts to trace module paths, to aid with debugging of occasional issues running on galaxy. Also moving to use $USER instead of whoami.
Pipeline 1.6.3 (19 January 2022)¶
Several patches to the pipeline scripts:
Ensuring the continuum validation directory & metrics file are included in the CASDA deposit.
Ensuring the tar files that contain extracted spectra for polarisation processing do not have a ‘%p’ in them - this was causing problems with the CASDA software upon deposit. The names now leave out the image.restored.%p string.
Ensuring the statsPlotter does not fail when encountering the FLASH_validation stats entry.
Ensuring the stats files are copied to the diagnostics directory alongside the stats plots, at the completion of a pipeline run.
Correcting erroneous job dependencies related to the combineFlagStatistics job.
ASKAPsoft 1.4.1 (13 January 2022)¶
A minor patch release affecting only the casdaupload utility, to
ensure that the commensal_projects
parameter provided for
measurement sets is correctly applied and visible in the XML file.
Pipeline 1.6.2 (21 December 2021)¶
Two patches to the pipeline scripts:
Ensuring backwards-compatiblity in dealing with the pre-flagging directive files. These are now generated if they do not exist, which will allow pipeline runs using 1.6.x to be run in a directory previously done with version 1.5.x
A symbolic link to the footprint file in the metadata directory is created, named in the same way as previous versions (without the “-src1” string). This should allow continued compatibility with validation scripts.
ASKAPsoft 1.4.0 (16 December 2021)¶
A number of changes to the processing code:
Imager:
Some fixes for the behaviour of the spectral imager when a given channel has very sparse data (ie. lots of flagging).
Improvements to the performance of the GaussianTaper.isPsfSize option.
Improved options for the export of UV grids, allowing FITS output to real & imaginary images, with sky or UV coordinates, CASA UV grids (which are done serially), or CASA/FITS grids FFT’d to the sky plane (real part only).
Joint imaging is able to handle holography-based beam patterns through A-projection.
Bug fix - spectral imaging was failing with MPI-based error when a worker had no valid channels to process.
Cflag: fixed situation where there could be slight differences in the flagging results when there are fully-flagged rows in the MS.
Imcontsub: fixed a bug that prevented image metadata from being read correctly.
Casdaupload:
Improved handling of checksums, so that they are not re-generated if they exist and are newer than the corresponding file.
New ability to handle UV grid data, incorporating as part of the measurement set section.
Selavy:
Ensured the project code for RM synthesis spectra is taken from the Q contcube, not the I.
Bug fix - catalogue name determination was failing when the resultsFile input did not end in “.txt”
Fixed general spectral extraction code to ensure the OBJECT and OBJID headers are correctly written to the output spectra.
There were also a number of fixes to the build recipes for yandasoft and some base packages, as well as fixing up the dependencies for the base packages.
Pipeline 1.6.1 (16 December 2021)¶
A few patches to the pipeline scripts:
Ensuring the correct project codes are applied to extracted spectra produced by Selavy, by making sure the headers of the contcubes are given the correct project ID.
Correcting a bug that meant the beamlog files were still being missed by Selavy.
Providing the observation start & end time explicitly to the CASDAupload tool, to cover the possibility of no MSs being provided.
Fixing job dependencies that were allowing the source-finding job to start before all mosaics were present (particularly the contcube mosaics).
Pipeline 1.6.0 (1 December 2021)¶
Changes to the Pipeline scripts:
New parameters for the continuum & spectral-line source-finding:
SELAVY_VARIABLE_THRESHOLD_REUSE
andSELAVY_SPEC_VARIABLE_THRESHOLD_REUSE
respectively, that control whether to re-use any noise maps generated from a previous run of Selavy. Both default to false.Options to generate plots of flagging Statistics for:
bandpass measurement sets
spectrally averaged science measurement sets
science measurement sets used for spectral imaging
Feature to conduct additional flagging of spectrally averaged data to flag out all visibilities for coarse channels that exceed a specified threshold for flagged percentage.
Increase robustness to link-failures/maintenance at MRO: the observatory generates and keeps a copy of the schedblock and footprint metadata after a scheduling block is observed removing the processing pipeline’s dependence on services hosted at MRO.
The preconditioning parameters are now able to be changed with the selfcal loop.
Updated information on the use of the askapsoft imcontsub tool as first preference in the pipeline.
The bandpass processing in the pipeline now has the ability to automatically identify channels that correspond to the Milky Way HI absorption seen towards PKS B1934-638, and have them flagged prior to determining the bandpass solutions (so that the smoothed bandpass interpolates over those channels).
There are new pipeline parameters
ARCHIVE_IMAGE_CONT
,ARCHIVE_IMAGE_CONTCUBE
,ARCHIVE_IMAGE_SPECTRAL
,ARCHIVE_CONT_MS
, to go withARCHIVE_SPECTRAL_MS
that can govern which types of data products are sent to CASDA.The specific image codes that are archived are now governed by
ARCHIVE_IMAGECODE_CONT
,ARCHIVE_IMAGECODE_CONTCUBE
, andARCHIVE_IMAGECODE_SPECTRAL
, which by default match what is given to the equivalentMOSAIC_IMAGECODE_
parameters.A fix has been made to the handling of Selavy catalogue names, so that non-standard names specified with
CATALOGUE_BASE_NAME
will get identified and uploaded to CASDA.
Pipeline 1.5.5 (15 November 2021)¶
A patch for the POSSUM validation pipeline script, fixing a mis-labelled variable for the parameter file.
Pipeline 1.5.4 (12 November 2021)¶
A small patch to the three initial linmos scripts in the pipeline, to
ensure that the use of weights logs in linmos is only turned on when
the relevant USE_WEIGHTS_LOG_x
parameter is true
.
Pipeline 1.5.3 (11 November 2021)¶
A patch to the pipeline to ensure that the copy of the pipeline configuration (written to the slurmOutputs directory) incorporates the template config, even when the template is taken from the central TEMPLATE_DIR directory. The reading of the template file from the SB parset is also made more robust.
Pipeline 1.5.2 (28 October 2021)¶
A small patch release to fix an issue with the thumbnail update in 1.5.1
Pipeline 1.5.1 (21 October 2021)¶
A patch release for the pipeline scripts. This addresses an issue with the mosaicking and the linking of mosaics to the top level directory. There were circumstances where the linking would fail (due, for instance, to an existing link of the same name), and this meant subsequent links handled by the job were not made. This could lead to some mosaics (weights or higher-order Taylor term images) not being present, and even being missed in the CASDA deposit.
The new code is more robust with the linking, having separate jobs to handle it and doing more error-checking.
Additionally, the identification of thumbnail images now fully reflects the desired suffix (as given by the parameter THUMBNAIL_SUFFIX).
Scripts used by ASKAP operations to assess pipeline errors have been added to the code repository, although are not (yet) part of the regular pipeline workflow.
ASKAPsoft 1.3.0 (21 September 2021)¶
New features for yandasoft (1.3.0 release):
Imager:
Imager & cimager have the ability to select which image outputs are written to disk, choosing from restored, clean model, psf (natural & preconditioned), residual, mask, sensitivity and weights.
The weights may, for gridders such as WProject that create constant weights images, be written as a text file (called a ‘weightslog’). This is able to be interpreted correctly by linmos.
Nyquist gridding an option in imager for Hogbom and BasisfunctionMFS solvers. This reduces the size of the grids to the smallest possible (2x the ‘Nyquist’ synthesised beam oversampling), saving on memory.
mssplit - optional feature to spectrally average visibilities using medians instead of mean.
linmos
Option to use stokes-I model instead of restored image in wide-field leakage correction.
Optional explicit specification of beam numbers for primary beam and leakage correction.
Ensuring linmos and linmos-mpi produce the same output.
Put checks in linmos-mpi to ensure that the input images have all 4 dimensions (RA, Dec, Stokes, Freq).
Sky-model service code updated.
Bug fixes:
Selavy - fix the calculation of island image sizes for situations where the entire island and background have negative values (which was resulting in NaN-valued size parameters).
Pipeline 1.5.0 (21 September 2021)¶
For imaging, the user has the ability (in concert with askapsoft/1.3) to decide whether or not to write out optional image products:
natural psf
preconditioned psf
weights and/or weights log
residual
model
mask (relevant for continuum only)
sensitivity (relevant for continuum imaging only)
the first type of restored image when a restored image with an alternate resolution is also specified (relevant for continuum only).
gridded visibilities: if opted, the grids are exported in casa image formats (cubes only).
For images with constant weights (as in the case of WProjection without snapshot imaging), the pipeline user can instruct the imager to write the constant weights into ASCII text files instead of large fits images. The weights log files are recognised by linmos.
There is the option to conduct additional flagging of spectral line data to flag out all visibilities for channels that exceed a specified threshold for flagged percentage.
Convolution to common resolution:
convolution codes updated to v1.2.0 of RACS-tools https://pypi.org/project/RACS-tools/2.0.0/
option to force the convolving beam to a circular Gaussian
better exception handling.
When ‘restored’ was not included in the
MOSAIC_IMAGECODE_CONT
variable, the weights images were not being archived. This is now fixed.Other feature updates and bug fixes
The resolution of the ‘large’ thumbnail images has been reduced from 300dpi to 100dpi, and the default thumbnail image formats changed from png to jpg.
Improved catching of failures in the FITS header updates prior to imcontsub.
Ensure plots from bandpass processing and bandpass validation are correctly archived.
Improved handling of metadata files needed by validation scripts.
Job ID initialisation to avoid inheriting spurious dependencies.
When a processing template is provided, and is not a direct path to a file, a standard directory location is searched for a file with that name. If no such file is found an error is raised. The standard location is provided in the default config (as
TEMPLATE_DIR
).
ASKAPsoft 1.2.2 (26 July 2021)¶
Implementation of off-axis leakage correction in linmos, using Q,U,V images/models.
ASKAPsoft 1.2.1 (15 July 2021)¶
Single change, to the CASDAupload tool, to allow specification of a commensal_project tag for each artefact.
ASKAPsoft 1.2 (2 July 2021)¶
Selavy:
Improvements to the calculation of error terms in the Gaussian component catalogue. These now follow the prescription from Condon (1997).
As part of yandasoft/1.2.0:
Imager:
Improvements to the flux scaling, so that the state of the decoupled parameter doesn’t affect the fluxes.
Improvements to the memory usage, including changing some double arrays to float
Cimager.dumpgrids=true fixed to no longer dump a core file
Solvers:
Fixes to the SVD solver used in calibration, when using the more recent GSL library.
Patches to Pipeline 1.4¶
There were a number of patch releases for version 1.4 of the pipeline scripts. They are summarised here along with the patch number and release date:
FITS header updates made more robust for the HISTORY entries (1.4.1 June 21; 1.4.2 June 22)
Correcting anomalous addition of UV cuts to contcube imaging parsets (1.4.3 June 23)
Allowing the spectral MSs to be tarred and checksummed in the flagSummary job, to save time in the final casdaupload job (1.4.4,5 June 24, 1.4.9 July 8)
Ensuring the correct PSF size is propagated to all beams in the beam-convolution (1.4.6 Jun 27)
Ensure UV cutoffs specified in selfcal and leakage calibration are robust against corrupt records (with zero UVW values) in the ms (1.4.7 June 29)
Fix a bug that missed archiving the leakage calibrated msdata for all but the last beam (1.4.8 July 1)
Ensuring the PROCESSED_ON file is created correctly (1.4.10 July 8)
Allowing specification of commensal project codes in the CASDA deposit (1.4.11 July 15)
Ensuring the contsub mosaics are controlled by the correct parameter (1.4.12 July 19; 1.4.13 July 20)
Ensuring the SB state is properly transitioned at the start of processing, for both science and bandpass SBs (1.4.14 July 22), and at the end, upon completion of the CASDA deposit (1.4.15 July 28)
Ensure the footprint rotation is propagated correctly to the primary-beam models used in linmos (1.4.16 August 16)
Pipeline 1.4.0 (21 June 2021)¶
Further enhancements to the pipeline:
New parameters to set spectral-index thresholds in the Selavy jobs for self-calibration & continuum-subtraction. These thresholds define the lower flux limit for sources to have a spectral-index determined from the taylor-term or continuum-cube images. The new parameters are SELFCAL_SELAVY_SPECTRAL_INDEX_THRESHOLD, SELFCAL_SELAVY_SPECTRAL_INDEX_THRESHOLD_SNR, CONTSUB_SELAVY_SPECTRAL_INDEX_THRESHOLD, and CONTSUB_SELAVY_SPECTRAL_INDEX_THRESHOLD_SNR.
The removeleakage parameter used by linmos is now available through the pipeline, using the new parameters REMOVE_LEAKAGE_OFF_AXIS_CONT, REMOVE_LEAKAGE_OFF_AXIS_CONTCUBE, and REMOVE_LEAKAGE_OFF_AXIS_SPECCUBE (all defaulting to false).
Minor improvements include:
Changing the name of the archived config file to pipelineConfig-<date>.sh (rather than the .in suffix that version 1.3 had)
Defining the new PROJECT_ID parameters in the defaultConfig.sh script
Using the averaged MS rather than the full-resolution MS as the reference for the leakage plotting.
Pipeline 1.3.0 (13 June 2021)¶
A larger set of changes to the pipeline, with a focus on preparing for Phase II Pilot survey processing and further support for reprocessing.
The command-line inputs to processASKAP.sh have been enhanced, allowing the specification of a template config (-t) and the two SBID parameters (-s SB_SCIENCE -b SB_1934). The template can also be read from the science SB parset.
Multiple project ID codes can be used in the CASDA upload. There are potentially different codes for continuum, polarisation & spectral data products, and further different codes for the continuum and spectral MSs. There are new input parameters: CONTINUUM_PROJECT_ID, POLARISATION_PROJECT_ID, SPECTRAL_PROJECT_ID, MS_CONT_PROJECT_ID and MS_SPEC_PROJECT_ID.
There are now checks for the weights ID used for the two SBs (science & bandpass) - they must be the same else the pipeline stops (unless the new parameter OVERRIDE_WEIGHTS_ID_CHECK is set to true).
If the spectral MSs are being archived, they are tarred up in the flagSummary jobs, to help speed up the final casdaupload job.
Better handling within the pipeline of the feed/polarisation position angle.
Linmos for the non-Stokes-I images now depends on DO_CONTPOL_IMAGING.
Pipeline 1.2.7 (27 May 2021)¶
Several small pipeline fixes:
Ensuring the leakage tables are archived alongside the bandpass tables.
Ensuring the image name has a “%p” wildcard properly resolved for the GASKAP validation script.
Ensuring the beam log produced by the PSF convolution has all beam images included, not just the last one.
Further changes to ensure operation on leakage-calibrated data without other MSs present. This includes ensuring the contpol clean arrays are properly defined when continuum imaging isn’t being done.
Fixing a typo with the archiving of the config files
Ensuring the correct list of Stokes polarisations is used in the beam convolution jobs.
Pipeline 1.2.6 (23 May 2021)¶
Minor fix to ensure the contcube linmos jobs proceed correctly in the case of multi-field SBIDs.
Pipeline 1.2.5 (22 May 2021)¶
Several bug fixes for the pipeline:
Ensuring the correct output PSF size is determined for the 2D beam convolution.
Separating the convolution options for continuum and continuum-cube, with new parameters BMAJ_CONVOLVE_CONTCUBE, BMIN_CONVOLVE_CONTCUBE, BPA_CONVOLVE_CONTCUBE, BMAJ_CUTOFF_CONVOLVE_CONTCUBE.
Ensuring the cube-stats calculations for the imcontsub jobs are properly parallelised (to match the imcontsub job itself), resulting in much faster execution.
Changes in the logic of pipeline switches, so that reprocessing of leakage-calibrated MSs is possible without all the pre-imaging jobs. Now can just give DO_CONTPOL_IMAGING=true and DO_CONT_IMAGING=false to get it to work.
Ensuring the job stats info for the spectral linmos jobs is written to the correct place.
Pipeline 1.2.4 (14 May 2021)¶
Several pipeline fixes:
Ensuring the continuum mosaicking makes use of the multi-threading capabilities of linmos, with a new parameter NUM_OMP_THREADS_CONT_LINMOS.
Typos fixed related to the application of pre-flagging.
Resolving an inconsistency in the specification of the growth threshold used in the Selavy jobs.
Pipeline 1.2.3 (12 May 2021)¶
A further bug-fix release of the pipeline that addresses a few issues:
Most importantly, it corrects a bug that has been in place since 1.0.18.3 that prevented the pre-flagging from being applied. Data processed with versions from 1.0.18.3 to 1.2.2 will not have had the pre-flagging (derived from the bandpass solutions) applied to the science data.
When the bandpass calibration is done elsewhere (that is, the table is not generated as part of the current processing), there was a problem copying the table to the BPCAL directory (in preparation for archiving). This has been fixed so that only the table will be in the BPCAL directory.
Additionally, the solver used for the self-calibration can now be
specified via SELFCAL_SOLVER
, which allows the use of LSQR instead
of the default of SVD.
Pipeline 1.2.2 (6 May 2021)¶
A further patch to ensure that the continuum-selfcal jobs work correctly when run in the “Fat-node” mode. The version of slurm on galaxy was upgraded, and resulted in the way these jobs were specified (using the “hetjob” mechanism) no longer worked. The new approach uses srun to distribute the tasks in a heterogeneous way.
Pipeline 1.2.1 (30 April 2021)¶
A small patch release of the pipeline to ensure that the TARGET_NAME is set properly when the rapid mode of the pipeline is used.
Pipeline 1.2.0 (29 April 2021)¶
New pipeline features:
Added the capability to make use of holography-based beam images/cubes in the linmos tasks. These are provided using new pipeline parameters
ASKAP_PB_TT_IMAGE
andASKAP_PB_CUBE_IMAGE
. If these are not given, linmos reverts to using the Gaussian-shaped primary beams.The beam models that are used will be archived to CASDA in the calibration-metadata-processing-logs tar-file.
Also for linmos, it is now possible to fix the direction of the mosaic image to the pointing direction of the field. This is controlled by the new parameter
FORCE_FIELD_MOSAIC_CENTRES
- turning this to false reverts to the previous behaviour of using the most central of the input beams. The direction can also be nominated specifically by givingFIELD_MOSAIC_CENTRE
.The default names of images, MSs and calibration tables have changed. They now incorporate the “target” name, or alias, of the Scheduling Block. This can be represented in the base name string as a
%t
wildcard. Here is a summary of the pipeline parameters that have changed:
Variable |
old value |
new value |
---|---|---|
|
1934_SB%s_%b.ms |
1934.SB%s.%b.ms |
|
calparameters_1934_bp_SB%s.tab |
calparameters.1934_bp.SB%s.tab |
|
calparameters_1934_bpleakage_SB%s.tab |
calparameters.1934_bpleakage.SB%s.tab |
|
scienceData_SB%s_%b.ms |
scienceData.%t.SB%s.%b.ms |
|
i.SB%s.cont |
i.%t.SB%s.cont |
|
cont_gains_cal_SB%s_%b.tab |
calparameters.selfcal.%t.SB%s.%b.tab |
|
i.SB%s.contcube |
i.%t.SB%s.contcube |
|
i.SB%s.cube |
i.%t.SB%s.cube |
Should you have a dataset partially processed with an earlier version, you may want to put the old values in your config file, so that the filenames are consistent with any that are already made.
Additionally, the following bugs have been fixed:
The slurm output file for continuum imaging jobs are now written in the correct place (the hetjob setup was putting the output files in the wrong directory with the wrong name).
The slurm output filename for the spectral linmos has been fixed.
Adding some robustness for failures when USE_CLI=false.
Pipeline 1.1.1 (15 April 2021)¶
Minor pipeline patches:
Disabling the turning off of DO_SOURCE_FINDING_FIELD_MOSAICS when only 1 field is present - this had undesirable results when mosaicking was turned off.
Using the solver.Clean.decoupled=true in the continuum-cube and spectral imaging, to better handle issues with askapsoft/1.1.0
Ensure we check whether the calibrated averged MS is present (along with the full-resolution and averaged MSs) in determining whether to go ahead with a pipeline run.
Slight changes to the handling of deprecated parameters (related to CORES_PER_NODE_x and NUM_CORES_x parameters) to ensure the behaviour is as described.
Changes to the leakage application parameters to avoid using only the first beam/channel
1.1.0 (31 March 2021)¶
This major release is the first made out of our new bitbucket repositories - a collection of over 30 independent repositories most of which have been extracted out of the earlier monolithic ASKAPSDP subversion repository. The breaking out of the monolithic repository into small independent ones was aimed at ease of distributing the core calibration and imaging functionalities (packaged now in to the YANDAsoft repo) amongst the wider radio astronomy community. The new software inherits all of the previous features essential for ASKAP operations and has a bunch of new features developed to enhance performance efficiency of data processing.
Processing Software
imager
Speed up CLEANing and Preconditioning stages (using OpenMP).
Speed up from combined FFT of any number of channels in an allocation as opposed to a channel-wise FFT during the inversion.
added feature to export UV-grids.
Calibrators
cbpcalibrator ability to derive bandpass leakages.
ccalibrator ability to derive bandpass gains and bandpass leakage. Advantage of ccalibrator over cbpcalibrator is its ability to additionally derive a smooth solution spectra, optionally taking beam forming intervals into account.
Note: This smoothing feature requires the LSQR solver, and is at an experimental stage.
imcontsub
Ability to work with data that has had the barycentric correction applied in imager
subtracting the continuum within beam-forming intervals requires taking in to account the shift of the spectra (along channels) due to barycentric corrections done at imaging.
linmos
Ability to accept user-defined field centre - default behaviour is to use the direction of the most central field from the input list.
Speed up correction and regridding (using OpenMP).
Ability to use MPI collective I/O when reading/writing FITS files - can be 2x faster for large files (and larger stripe factors).
Ability to make use of ASKAP holography information for PB correction (Stokes-I only).
Future release will have leakage correction.
general
Added mssplit parameter to control how storage managers are mapped to files.
Better estimation of WMax (using unflagged data). Helps set up the memory required for gridding kernels correctly and avoid downstream processing issues.
New test tool tImageWrite for writing simulated images via the standard interfaces to assess I/O performance under various settings.
bugfix for 1934-638 flux model - although present from the beginning, the error is negligible at ASKAP’s frequencies and could only be discovered when processing high-frequency ATCA data.
Pipelines
Added user parameter
NUM_OMP_THREADS_CONT_IMAGING
to make use of the multi threading features in imager. The default is set to 20.Made changes to the distribution of master and workers of the imager on allocated nodes - the default ensures an exclusive node (FAT memory node) for the master, while the the workers are distributed cyclically on the remaining nodes. This speeds up the processing as well as ensures larger memory for the master and the workers.
The combined FFT feature is turned on for speeding up continuum imaging.
Efficient leakage Calibration process: The pipeline can now derive a single bandpass leakage table for all beams and antenna (using cbpcalibrator), and apply the correction to the science data (using ccalapply) without splitting the science measurement sets channel-by-channel. To use this feature users need to set:
WRITE_BPLEAKAGE=true
(default)LEAKAGE_CAL_SOLVER=LSQR
(default)BANDPASS_CAL_SOLVER=LSQR
(default) NB 1: The version 1.1.0 of the software requires usingsolver=LSQR
for cbpcalibrator. The calibrator code has a known bug that gets triggered when SVD is used instead. This bug has been fixed. The fixed version will be available in the next release.NB 2: The solutions derived using LSQR however are rather sensitive to the damping parameter
alpha
of this solver. Pipeline users may need to change the defaults for this parameter currently set toBANDPASS_ALPHA=1000
andLEAKAGE_ALPHA=3e5
(derived from experiments on a number of data sets).NB 3: Users can revert to the old method by setting
WRITE_BPLEAKAGE=false
in the the pipeline configuration file.Makes use of multi-thread option to speed up mosaicing continuum images. This feature was available in the past release, but now with the new linmos application, this will be put to use.
The default for stman.files is now set to
combined
in all the mssplit tasks. This may be useful in improving the file system related IO by reducing the total number of files on the file system. Users can change the default throughFILETYPE_MSSPLIT
pipeline configuration parameter.
1.0.20 (31 March 2021)¶
The final 1.0 pipeline update. This addresses a few issues:
The continuum validation file has fields that get written as type unicodeChar converted to type char (to prevent issues downstream with the CASDA software).
When the SUBMIT_JOBS parameter is false, we now do not create various files (the stats summary, the jobList and PROCESSED_ON, and the copy of the config file)
We no longer use the askapcli module - tools like footprint and schedblock are provided by the askappy module.
A number of parameter names have been changed to use better terminology. Those that had
CPUS_PER_CORE_
are nowCORES_PER_NODE_
, andNUM_CPUS_
are nowNUM_CORES_
. The old parameters are accepted (a warning message is written at the start of the pipeline, and the new parameters will take on the values of the old).There are new parameters CORES_PER_NODE_CONTCUBE_LINMOS and CORES_PER_NODE_SPECTRAL_LINMOS to better control how the linmos jobs behave.
There were also further bug-fixes for 1.0.19 that were released as patch versions 1.0.19.1-1.0.19.4:
Better checks on existence of files before plotting beamlogs.
Fixing EOF indentation in a couple of slurm scripts
Ensuring the beam-wise source-finding is done in the correct place (after the spectral imaging is launched, in case spectral extraction is needed).
Ensuring the pre-imaging script in rapid mode is available for all beams and not overwritten.
Ensuring the source-finding checks the values of both DO_MOSAIC and DO_MOSAIC_FIELDS
Getting the paths to the extracted spectra correct when uploading to CASDA.
1.0.19 (10 March 2021)¶
A further pipeline update:
The beam-convolution for continuum images has a different structure, where the output PSF is determined first, then each image sequentially is convolved by itself (rather than all at once). This reduces the memory footprint of the overall job.
The names of logs, parsets and output files have been made more consistent, so that they have the same essential name for a given job, and that name is also more descriptive.
When linmos pipeline jobs fail to find any images to mosaic, they will return a failure code. This will prevent dependent jobs from running after them.
There were further bug-fixes for 1.0.18 (most released as patch versions 1.0.18.1-1.0.18.5):
The IFS parameter was being set incorrectly in the beam-convolution, which had effects for downstream jobs. A typo in the beam convolution job was also fixed.
There were a couple of erroneous cases of the askapenv module being unloaded, which is not necessary with the updated bptool module. The askapenv loading has been moved to a subshell.
The imcontsub jobs now only move into the working directory when doing the imcontsub task, so that cubes can be found on a re-run.
1.0.18 (28 February 2021)¶
A pipeline update addressing a few issues:
The handling of modules within the pipeline scripts and the slurm scripts has been simplified, and made compatible with the updated bptool and forthcoming askapsoft modules.
Checkfiles are now used with the Selavy jobs, to allow ease of re-running should the selavy job fail mid-way through.
The job stats information is now written progressively to the stats-all files. The intermediate files in the stats directory are no longer written, and there is no stats directory.
The convolution-of-beams will now run properly when DO_MOSAIC_FIELDS=false (calculating the common PSF separately for each field). There is also now the ability to provide the desired output PSF (see below).
New pipeline parameters:
LEAKAGE_CAL_SOLVER
BPTOOL_VERSION
convolution parameters: NUM_CPUS_CONVOLVE_2D, BMAJ_CONVOLVE, BMIN_CONVOLVE, BPA_CONVOLVE, BMAJ_CUTOFF_CONVOLVE
1.0.17.1 (8 February 2021)¶
A bug-fix for 1.0.17, getting the WCS handling in the thumbnail-plotting script correct.
1.0.17 (5 February 2021)¶
A pipeline update primarily to fix the thumbnail-plotting script, updating the code so that it runs better with the astropy WCSAxes package, rather than APLpy. Also ensuring that the tests for redoing the thumbnails are done properly.
Also a bug-fix for the GASKAP validation script.
1.0.16.2 (12 January 2021)¶
Minor fix to the pipeline scripts, to get the continuum validation module working correctly with the new askappy module.
1.0.16.1 (8 January 2021)¶
Minor fix to the pipeline scripts, so that the file type reported for extracted spectra in the CASDA upload XML interface document is correct in the FLASH style of extrating spectra.
1.0.16 (18 December 2020)¶
Improved functionality for the pipeline:
Moving to python3-compatibility in the python scripts in the pipeline
Use of the new askappy module, doing away with the python2.7-based virtualenv provided by askapsoft
Addition of the GASKAP validation script, and improved module handling in the rest
More robustness within the cubestats calculations to empty arrays
Correcting a bug where the beam-convolving jobs were using the wrong FIELD ID
Fixing a bug where check-files for multi-field linmos jobs were not able to be created.
Minor errors fixed in the getArtifactsList script
Allowing the rapid mode to run the beam-convolution jobs (released as 1.0.15.1)
Additionally, askapsoft/1.0.16 has been deployed at Pawsey out of the subversion repository, with no functional change other than without the python virtual environment and the python packages.
1.0.15 (26 November 2020)¶
A larger release of the pipeline scripts, providing new functionality:
The ability to convolve continuum images & cubes to a common resolution
Better ability to recover from errors in the spectral imaging or mosaicking, particularly if the imaging succeeded but the stats calculations failed.
Incorporation of science teams’ validation scripts at the end of the pipeline workflow.
The mean map from the continuum selavy output is included in the payload uploaded to CASDA.
The imaging jobs now use the “detectDivergence” parameter, to allow the deconvolution to stop if it starts to diverge.
The final copy of mosaics from a single field directory is now a symbolic link
Cleaner use of the non-standard extract stats functions, for better time reporting.
1.0.14.4 (16 October 2020)¶
Minor fixes to the pipeline scripts to better handle SBs with spaces in the field names. This primarily addresses issues in the spectral processing & source-finding jobs.
1.0.14.3 (7 October 2020)¶
Another quick update for 1.0.14 with a couple of minor fixes:
A further check in the rapid-mode preimaging script to check for already-existing averaged MSs
Updating the FITS headers of the restored spectral cube prior to image-based contsub, to go with a fix to the imcontsub script (that allows the correct handling of intervals after frame conversion).
1.0.14.2 (25 September 2020)¶
A further quick update to 1.0.14 that fixes a bug with the rapid-mode preimaging script.
1.0.14.1 (24 September 2020)¶
A quick update to 1.0.14 that fixes a couple of minor bugs in that release:
The “FITSconvert” job label in the stats files was breaking the stats plotting, so it is now “fitsconvert”
The checks for existing MSs when the raw data is missing were not looking for the right filenames when time-splitting was being used.
1.0.14 (10 September 2020)¶
A single change to the processing, where the casdaupload tool is now able to correctly tar measurement sets that are symbolic links, by properly dereferencing the link.
Most of the changes are in the pipeline scripts:
The pipeline now has the ability to re-run when the raw data is absent. The user can add parameters NUM_CHAN_1934, NUM_CHAN_SCIENCE, MS_METADATA_CAL, and MS_METADATA to the config file, with the latter two pointing to existing mslist metadata files, and the pipeline should be able to pick up from where it left off (assuming there are local versions of the MS data).
The linmos jobs and those following are now submitted with an “afterok” dependency, meaning they will only run if all the preceding beam jobs have succeeded.
The final jobs of the pipeline run (convertToFITS, makeThumbnails, validationScience) are now done before the gatherStats job, which allows reporting of their job stats.
A bug was fixed where the spectral imaging findStats & maskChannels job stats were not being recorded correctly.
The version of the numpy module is able to be specified through the config file.
1.0.13 (17 August 2020)¶
The main addition is a patch to the GaussianTaper preconditioner. The change made in 1.0.12 had a bug with the internal normalisation of the PSF, which led to incorrect fluxes in the output image, particularly in the residuals. This change fixes this so that the fluxes should be accurate.
There are a few minor fixes to the pipeline scripts as well:
The tolerance parameter for the GaussianTaper is exposed through the pipeline.
The validationScience script has a couple of bugs fixed that were preventing all necessary files ending up in the diagnostics directory.
A bug in processDefaults was fixed that meant certain spectral imaging parameters were not being set when spectral processing was turned on but spectral imaging was turned off.
The ability of the smooth_bandpass tool in v1.10 to take a file indicating a list of spectral windows has been exposed through the pipeline.
1.0.12 (26 July 2020)¶
A further patch release for both askapsoft and the pipeline
A new feature is exposed for the preconditioning. The GaussianTaper is now able to be specified as the desired output size (rather than the taper size), by use of the GaussianTaper.isPsfSize=true parameter. This means the taper size will be interpreted as the desired size of the PSF after tapering, and the size of the taper itself will be adjusted accordingly.
This mechanism has been exposed through the pipeline scripts as well.
Linmos has been patched to ensure that spectral-line mosaicking is able to be done without error - coordinate axis errors were causing job failures with 1.0.11.
Selavy has had its memory handling greatly improved, which should allow it to run without excessive use of memory. Part of this involved a patch to the underlying Duchamp library, which is now at 1.6.3.
The master node uses noticeably more than the workers, so to ensure there is sufficient headroom the default number of cores/node for the pipeline sourcefinding has been set to 10 by default.
Selavy also sees improvements to the way the component fitting is done, in the mode where it does not use the number of components from the initial estimate. The new approach is to use the initial estimates for as many as there are, but then find new places to put components by removing all previously-fitted components and searching for further peaks.
Additionally, a small change was made to the CASDA interface to ensure that validation reports are properly identified, so that the new CASDA release can show them via web links.
1.0.11 (10 June 2020)¶
Another patch release for askapsoft and the pipeline
Selavy sees additional fixes for the RM synthesis module, where the handling of masks is further improved through their extraction and use. This should deal correctly with spectra missing masked channels.
The FITS writer is also now able to write out masks to FITS files.
Linmos has a couple of bugs fixed:
Defining a beam centre that is not the centre of an input image will now do the expected thing.
The “removebeam” parameter will now work as expected.
In the pipeline scripts:
The default value for CONTSUB_METHOD has changed to CleanModel
You can now use CONTSUB_METHOD=remUVcont and DO_COPY_SL=true - previously the method would have quietly changed to Cmodel.
There is a new parameter SPECTRAL_IMSUB_INTERLEAVE that allows the interleaving of blocks in the image-based continuum subtraction.
1.0.10 (1 June 2020)¶
A further patch release for askapsoft and the pipelines
There are several bug fixes for Selavy:
The RM synthesis module has had a few updates:
The writing of spectra is now only done for components that are detected in polarisation.
The Stokes I modelling is better constrained, so that null values of -99 for spectral index and curvature are not used.
The pol_peak_debias parameter is no longer provided in the catalogue for non-detections.
The error on the polarised SNR is now reported.
The component catalogue has a couple of changes:
The minor axis error is now corrected (it was previously reporting the error on the axial ratio).
The errors on the deconvolved sizes are also now correctly reported.
The cubelet extraction had been broken by changes in 1.0.9, which meant that island statistics (such as the mean background etc) were not being reported correctly.
The extraction and creation of component map/residual images will now work with input images of only two dimensions (ie. with no spectral axis).
Additionally, the maskBadChannels tool has a new set of parameters, allowing better control over the different determinations of channels to be flagged.
The pipeline scripts have a few changes, mostly to cover the above:
There are new parameters governing the channel masking: MASK_CHANS_USE_SIG, MASK_CHANS_SIG_THRESHOLD, MASK_CHANS_USE_NOISE, MASK_CHANS_NOISE_THRESHOLD, and MASK_CHANS_BLANK. Generally the defaults should be OK, although this will mean a less stringent masking regime.
There are new environment variables in each slurm job, to help with PMI-related errors that are occasionally seen on galaxy (PMI_NO_FORK and PMI_NO_PREINITIALIZE).
1.0.9.1 (22 May 2020)¶
A quick update to get an additional feature in for operational purposes.
A pipeline user can now specify a list of image codes that should be mosaicked. Use the new parameters MOSAIC_IMAGECODE_CONT, MOSAIC_IMAGECODE_CONTCUBE, MOSAIC_IMAGECODE_SPECTRAL to provide the list relevant to your needs. The default values are the same as previous use.
1.0.9 (19 May 2020)¶
A patch release for both pipelines and applications
The pipeline scripts see the following fixes:
Fixes to the updateFITSheaders script, so that all headers are replaced correctly (the previous version’s fix introduced a bug that limited the update to a single header card).
Extracted continuum component spectra are now included in the CASDA deposit.
Minor fixes to the source-finding pipeline script to get directories and file paths correct.
Correcting the dependencies and the range of mosaicking jobs submitted for different combinations of with/without altrestored or contsub imaging methods
The user configuration will now be able to over-ride the PROJECT_ID for the observation, for the purposes of archiving. If not given, we fall back to the value in the scheduling block database.
Ensuring the validation files are named distinctly and reported correctly to the casdaupload tool, particularly for the case of multiple fields with their own validation file.
The applications see the following fixes:
The msmerge tool is now more robust against missing integrations in one or more of the input measurement sets. The missing data in the merged spectrum is now set to zero and flagged.
Selavy has a few corrections:
The flagging arrays used in the spectral extraction were the inverse of what they should have been. This meant no valid values were being passed to the downstream tasks (such as RM synthesis). This is now done correctly.
The extraction from a spectral cube that had a different spatial extent to the continuum cube could have resulted in errors due to the location being outside the image. Now, if that is the case, no extraction is attempted.
If a Gaussian fit results in components of very low S/N (ie. less than one for the peak or major/minor axes), then the fit is rejected and redone with one fewer component.
1.0.8 (27 April 2020)¶
A patch release for both pipelines and applications.
The pipeline scripts see the following fixes:
A new option for UV-based continuum subtraction - CONTSUB_METHOD=”remUVcont”, that uses a new tool to fit harmonic functions to the visibility spectra. The remUVcont tool is still a prototype, but this mode allows testing of it to be integrated into the pipeline.
The ability to run extraction of source and noise spectra for bright continuum components, extracting from the contsub or restored spectral cube.
Access to new parameters in the image-based continuum subtraction tool robust_contsub_mpi.py, that allow fitting & subtracting in blocks of channels.
More robustness of the pipeline scripts for the presence of spaces in the MS field names.
There is more checking done to avoid re-running jobs unncessarily when the pipeline is re-run. This includes not re-running the source-finding jobs, as well as the updating of FITS headers, creation of thumbnails and tar files needed for CASDA upload.
The SELAVY_SPECTRAL_INDEX_THRESHOLD_SNR now defaults to 50, so that we apply that threshold to components before reporting their spectral index and curvature in the catalogue. Fainter sources will get values of -99.
The applications see the following fixes:
The casdaupload tool has a new parameter “clobberTarfiles” that allows it to use existing ms.tar files or create new ones.
Selavy had several bugs fixed in the extraction of spectra and the handling of those spectra in RM synthesis, particularly for data that exhibits masked or flagged channels.
The extraction of source & noise spectra for continuum components is now better parameterised (through parameters called Components.extractSpectra & Components.extractNoiseSpectra).
The ordering of components in the component catalogue is now properly done by island/component order (treating the island IDs as integers rather than strings). This means component 1a (the brightest component of the brightest island) is always first.
There is more control over the precision of various columns in the output catalogues.
1.0.7 (29 March 2020)¶
A pipeline patch, covering the following:
Introduction of a user-specified set of beam-wise flagging commands. These are provided through a single input file, and can define sets of flagging commands that can differ from beam to beam.
Introduction of BANDPASS_CAL_SOLVER to set the solver parameter for cbpcalibrator.
Allowing the re-use of existing directory tar files in the casdaupload job.
Making those tar files more specific, so they only keep files relevant for the current SBID and set of fields.
1.0.6 (10 March 2020)¶
A simple patch to the pipeline scripts, allowing control over the version of the python module used in the slurm scripts. This is now governed by the PYTHON_MODULE_VERSION parameter.
1.0.5 (24 February 2020)¶
A further patch to the 1.0 release, affecting just the pipeline scripts:
Leakage-calibrated measurement sets are now included in the CASDA upload, in preference to those with just the self-cal gains applied.
There is now a (user-parameterisable) sleep period inserted before the second and subsequent srun calls in slurm jobs. The length is governed by SLEEP_PERIOD and defaults to 30 (seconds). This aims to help with slurm-related bus errors on the compute nodes.
The findCubeStatistics jobs are now reported in the stats files alongside the maskBadChannels. Both these have clear descriptions as well.
Fixed a bug where one couldn’t specify both CIMAGER_MAXUV and CIMAGER_MINUV.
Fixed an issue where the symbolic links to reportProgress, killAll and jobList were not being made correctly.
Fixed a bug in the wrapper script used in the rapid-mode version of the apply-leakage-correction job.
1.0.4 (17 February 2020)¶
A patch to the 1.0 release, affecting both pipeline scripts and applications
Pipelines:
A change to the leakage calibration scripts so that we do not use srun calls within loops over short-lived jobs. This will reduce the load on the Pawsey slurm controller.
The finding of leakage parameters now does not involve a split into multiple measurement sets.
When SUBMIT_JOBS=false, we now do not save a copy of the config file.
Applications:
Selavy will now only write out polarisation spectra for sources with a polarised flux signal-to-noise above a particular value.
1.0.3 (5 February 2020)¶
A patch to the 1.0 release, covering both pipeline scripts and one application
Pipelines:
Better error handling in a number of scripts, so that the return values from slurm jobs accurately reflect what failed (if anything).
Improved determination of footprint files for observation metadata. These are now fully determined from the scheduling block parameter set, so an observation that was only partially completed will have all planned fields with defined footprints.
Applications:
The maskBadChannels app now exits if the input stats file is not able to be opened.
1.0.2 (30 January 2020)¶
A further patch release to both the pipeline scripts and certain applications.
Pipelines:
Ensuring the BPCAL can run without DO_SCIENCE_FIELD=true - the case of MSs needing to be merged was not previously running correctly.
The FEED table in the MS created by msconcat (in the split-by-time case) is ensured to be identical to the original.
Channel masking is enabled in the pipeline - the image cube statistics are used to determine discrepant channels, which are masked in the image cube, along with channels that are completely blank.
Implementation of the MPI-enabled version of robust_contsub.py, to speed up the image-based continuum subtraction.
Allow flagging through the pipeline based on UV-range (through cflag’s selection flagger).
Applications:
Fix to the calculation of fluxes in the polarisation catalogue.
Improved maskBadChannels to better use the statistics to identify the outlying channels.
Fix to cflag’s selection flagger, so that it can make use of the uvrange option.
1.0.1 (16 January 2020)¶
A patch release to the pipeline scripts:
Better handling of repeated field names in the scheduling block parset.
Better handling of repeated fields and their corresponding tiles, particularly in the linmos scripts.
Ability to turn off the leakage calibration jobs when DO_LEAKAGE_CAL_CONT=false.
Allow use of the parallactic angle rotation in the gridding by extracting the pol_axis angle from the SB parset.
1.0.0 (06 December 2019)¶
A major release that facilitates derivation and application of on-axis instrumental leakage calibration.
This is also the final release to be made out of the monolithic ASKAPSDP subversion repository. Future releases will be made from our new bitbucket repositories. We have thus designated this version 1.0.0.
Processing Software:
Ccalibrator now allows derivation of leakages per beam. This is done by adding a new value “antennaleakages” for the parameter “solve” of ccalibrator. It functions in a similar way as “antennagains”.
The parallactic angle rotation can be incorporated into the gridding. This is either read from the FEED table or provided via a parset parameter.
Improvements to ccalapply
Better initial estimates for psf beam-fit
Detects divergence in BFMFS and stops clean early.
More robustness in the RM-synthesis module of Selavy to blank channels.
Pipelines:
The bandpass self-calibrated 1934-638 data are further processed (flagged, averaged to 1MHz, flagged again) to allow derivation of on-axis leakage solutions per 1MHz channels for each beam.
The derived leakages are applied on the science data after the continuum selfcal stage. For each beam, the leakage correction is done per 1MHz coarse channel.
The contPol (except for Stokes-I) and the contCube images will now be made using the on-axis leakage calibrated data.
Other minor fixes:
Parset fixes for the contPol imaging
Addition of a “-r” option to reportProgress, so that only the running jobs are shown
Tidying up of the job stats reporting & plotting, so that all jobs (including the new leakage calibration ones) are included.
0.24.9 (15 November 2019)¶
A further pipeline patch, ensuring that the rapid mode functions correctly. The preimaging script was hard-coding the output directory to be the final field in the list of fields, and so data was ending up in the wrong directory, and failing to image.
This has been corrected, along with work to ensure post-selfcal jobs are not run when the DO_CONT_IMAGING flag is turned off.
0.24.8 (8 November 2019)¶
An important patch to the pipeline scripts.
This fixes an issue with setting the lustre striping, where the value given for the striping had not yet been defined in the script. If lustre pools have been set in the current directory, these are removed if no argument is given, ie.: lfs setstripe -c “” .
The change sets the default parameters before running the initialisation script.
0.24.7 (24 October 2019)¶
A further patch to both processing and pipeline code.
Processing:
Further fixes to the PSF-fitting within the imager, to avoid divergence in the case of extended beams.
A check is now made within the imager to determine whether the clean is diverging. If very large residual values are seen the clean is halted.
The makeThumbnailImage script has been updated to address an issue making images with weights contours.
Pipelines:
The rapid-mode now properly checks for the return values of the pre-imaging tasks prior to running the imaging. A failure in one of those tasks will stop the imaging from running.
The flagSummary tasks are now turned on by default.
The bandpass plotting now shows the smoothed table when the smoothing option is used.
The use of nominated askapsoft modules has been made more robust, particularly in the prepareMetadata script.
The updating of the HISTORY cards in FITS headers now has a check for the existence of any HISTORY cards before checking for repeated entries (this was causing the script to fail).
The user can now specify the number of cores per node for the continuum source-finding job, via the CPUS_PER_CORE_SELAVY parameter.
The continuum cube imaging task will now use the correct MS when no selfcal calibration is applied to the data.
The following jobs are no longer needlessly submitted:
msconcat for the spectral data (upon re-running the pipeline)
Stokes V linmos when DO_CONTPOL_IMAGING=false
0.24.6 (3 October 2019)¶
A patch to provide a fix to Selavy, correcting a problem with the integrated fluxes in the Island catalogue - these were being corrected twice for the beam area, resulting in values that were too small.
There is also a new tool in Synthesis, maskBadChannels, that can mask channels in a spectral cube based on the cube statistics (noise as a function of channel).
There are also minor improvements to the pipeline scripts:
The HISTORY headers written to the FITS files should no longer have repeated entries.
The modules loaded while running processASKAP.sh have been updated to avoid python errors when running on Galaxy (should the user not have the python module loaded at time of running processASKAP.sh).
The findCubeStatistics script has a -n option to allow plotting of the stats without recomputing them.
0.24.5 (13 September 2019)¶
A patch release largely built around fixing issues with data presented to CASDA for release.
Ensuring the validation metrics file is correctly named when casdaupload is re-run.
Getting the correct RA/Dec positions for each beam’s MS for the casda metadata (rather than the pointing centre of the observation).
Allowing the correct annotation of Jira tickets when uploading to CASDA.
Ensuring the column widths within the VOTable catalogues produced by Selavy are consistent with CASDA expectations.
There are also several pipeline fixes:
A new “rapid” mode of the pipeline for wide-area continuum observations of many fields within the one scheduling block. This combines the pre-imaging tasks into the imaging slurm job, to reduce the overall load on the slurm controller.
New bandpass plotting is provided, along with flag summaries for each MS.
The reference frequency was not being set correctly for the CleanModel option of continuum subtraction.
We avoid writing out the schedblock variables to the metadata files, to avoid formatting issues that were causing errors.
The Stokes value had been set to lower-case in some labels on plots, which led to confusion. It is now back to upper case.
And for the askapsoft processing software:
The numpy, matplotlib, scipy and astropy 3rdParty modules have been removed - these need to be provided by the external system. This was to avoid segmentation faults occuring in the makeThumbnails jobs for the larger continuum images. This has resulted in changes to the modules loaded by the pipeline scripts as well (so that those python packages are provided by system modules).
The PSF fitting in the imager has been tweaked to avoid rare cases of divergence.
0.24.4 (6 August 2019)¶
A further small patch release, addressing a few critical issues:
The determination of beam locations in the pipeline is now done in a more robust way, that is insensitive to the specifics of the parset formatting (making use of the LOFAR getparsetvalue command).
The handling of the continuum validation directory in the CASDA upload is fixed, so that the filename is recognised correctly. It has also been shortened, to avoid problems with Windows filesystems.
The code to invert the sign of the Stokes V images in the pipeline has been removed. The images are now presented as-is, and work will be done to ensure the correct values are presented by the imager.
Selavy has had improvements made to its memory handling, to avoid out-of-memory errors due to memory leaks.
0.24.3 (2 August 2019)¶
A small release providing a patch to the processing software and a new option to the pipeline scripts.
Processing software:
0.24.2 had an issue with the application of self-cal gains. If integrations were substantially flagged, they would have a null row in the calibration table, and this could result in errors during the application, whether this was done from within the imaging or by ccalapply.
The new code is much more robust against flagged data, and completely flagged integrations will be left out of the table. This allows the application of the solutions to proceed without error.
Pipelines:
There is a new option to produce MFS images in different polarisations, following the application of the self-cal gains to the averaged dataset. The default set of polarisations is “I,V” (although I is done in the self-cal procedure and not repeated), and there are new parameters to govern the deconvolution for these images - consult the documentation for coverage and default values.
This also includes a script to invert the sign of the pixel values in the Stokes V images, to make the values accord with the definition of Stokes V from IAU/IEEE. This can be turned off by setting INVERT_SIGN_STOKES_V_IMAGE=false.
For consistency, there is a similar one for the continuum cubes: INVERT_SIGN_STOKES_V_CONTCUBE. These are kept separate to allow them to be switched on independently, in case (for instance) the memory requirements of the cubes mean you want to do it separately.
Additionally, the pipeline scripts now allow the averaged dataset to be flagged when the full-spectral-resolution dataset is not.
There are checks made at the start of the spectral splitting, to ensure we don’t do it when the concatenated version of the MS is present (similar to the change made in 0.24.2).
The dependencies for the continuum source-finding job are fixed so that the single-field cont-cube mosaicking job is included.
The mosaicking now appropriately handles the continuum images when nterms=1.
0.24.2 (25 July 2019)¶
This is largely a patch release, with a few minor additional features.
Pipelines:
Changes to the distribution of ranks within the imaging job, to smooth out the memory usage a little.
Use the bptool module to identify antennas to flag based on the bandpass calibration table. Triggered by the new parameter DO_PREFLAG_SCIENCE. The bptool module is now assumed to be in the same place as other ASKAP modules.
Checkfiles will now indicate the time-window where appropriate, to better distinguish work that has been done on datasets.
Checks for spectral-line parameters are now only done when DO_SPECTRAL_PROCESSING is set to true as well as DO_SPECTRAL_IMAGING.
Time-split data will now no longer be regenerated if the merged dataset is present.
The second mosaicking job is now only done if there is more than one field. There are also better error checking done for the copying of the mosaic in the case of a single field.
The altrestored mosaicking jobs are not launched when this mode is not used.
When restarting a partially failed imaging pipeline, the selfcal gains table is also searched for to see if a given beam needs re-doing (previously it was only looking for the image, which meant a partially-completed selfcal job could look like it was complete).
The gatherStats slurm job now has a longer time request, to prevent it timing out for SBs with large numbers of fields.
Additional checks are made when restarting selfcal, so that the gains table needs to be present (as well as the image) for the selfcal to be skipped.
When measuring the spectral indices from the continuum cubes, the pipeline scripts will now use the reference-beam’s beamLog to correctly scale the spectra prior to fitting.
When archiving measurement sets, the metadata directory is now copied into the MS prior to it being tarred, where it is called ASKAP_METADATA.
New or changed pipeline default parameters:
DO_PREFLAG_SCIENCE (true)
GRIDDER_SHARECF and GRIDDER_SPECTRAL_SHARECF, to allow performance enhancements to be realised in the pipeline.
TILENCHAN_AV, TILENCHAN_SL, TILE_NCHAN_SCIENCE & TILE_NCHAN_1934 all now default to 1.
SELAVY_NSUBY is now 6
ARCHIVE_SPECTRAL_MS is now false
MULTI_JOB_SELFCAL is now false
SELAVY_SPECTRAL_INDEX_THRESHOLD and SELAVY_SPECTRAL_INDEX_THRESHOLD_SNR, for the new Selavy threshold parameters (see below).
PURGE_FULL_MS_AFTER_COPY to remove the full spectral MS after copying a nominated channel range.
Processing:
Selavy:
Images created by Selavy, such as noise maps, will now have the same image mask as the input image.
There are new parameters “Selavy.spectralTerms.threshold” and “Selavy.spectralTerms.thresholdSNR” to provide a minimum flux or signal-to-noise ratio for the spectral index/curvature to be reported in the component catalogue (components fainter than this threshold will not have a spectral index/curvature value).
A fix to the computation of spectral indices from extracted spectra - this was failing due to uninitialised memory.
MFS imaging:
The higher-order Taylor-term residual images are now appropriately scaled to take into account the coupling with the Taylor-0 residual. The behaviour is now consistent with CASA. For reliable spectral indices, it is recommended to measure from cleaned sources.
The linear-mosaicking will now correct higher-order Taylor-term images for primary beam variation with frequency, through the use of the removebeam parameter.
General imaging:
There are numerous performance enhancements to imaging and linear-mosaicking, resulting in faster processing times.
The WProject gridder will now discard data above the wmax limit, rather than throwing an exception.
The WProject gridder now has the option of using a static cache of convolution functions, rather than re-generating for every instance of a gridder.
CASDA upload:
The number of time steps for each scan in the measurement set are now reported to CASDA via the observation.xml file.
Each scan also has a field-of-view reported. It is currently the FWHM of the beam (1.09 lambda/D), evaluated at the centre frequency.
0.24.1 (8 May 2019)¶
A patch release, focusing only on fixes for pipeline issues:
Default values for GRIDDER_WMAX_NO_SNAPSHOT and GRIDDER_SPECTRAL_WMAX_NO_SNAPSHOT have been changed to 35000
For the CASDA upload script, certain evaluation files have had the pipeline timestamp added to their filename
A few bugs were fixed:
The list of MSs given to the upload script were erroneously the final time-window MS (when that mode is used).
The continuum validation script was mistakenly making all files group-writeable (rather than just the copied validation directory).
The logic for checking the FREQ_FRAME parameter was incorrect and should now do the right thing.
The user documentation has also been updated, with a missing table added back in.
0.24.0 (16 April 2019)¶
A major release, with an improved framework for the pipeline scripts to handle large datasets efficiently, along with several performance improvements to the processing software.
Processing software:
Spectral imaging:
There is a change to the behaviour of the Channels keyword - now the syntax is [number,startchannel], where the channels are those in the input measurement set.
Introduction of more frequency frames. You can now specify one of three reference frames topocentric (default) barycentric and the local kinematic standard of rest. The new keyword is: Cimager.freqframe=topo | bary | lsrk
You can now specify the output channels via their frequencies if you choose. This is a very new feature but simple to test. Note that, as before, there is no interpolation in this mode. It is simply a “nearest” neighbour scheme. The syntax is: Cimager.Frequencies=[number,start,width], with the start and width parameters are in Hz.
The spectral line imager should be more robust to missing channels either via flagging or frequency conversion.
The restoring beam was often not being written to the header of spectral cubes, particularly when more than one writer was used.
The beamlog (the list of restoring beams per channel) for cubes was not being written correctly. It now will always be written, and will have every channel present, even those with no good data (their beams will be zero-sized).
The bandpass calibration had reverted to the issue solved in 0.22.0, where a failed fit to a single channel/beam would crash the job. This is now robust against such failures again.
The mssplit tool has had a memory overflow fixed, so that bucketsizes larger than 4GB can be used.
The linmos-mpi task would fail with an error when the first image being processed was not the largest.
Pipelines:
The pre-imaging tasks can now run more efficiently by dividing the dataset into time segments and processing each segment separately. The length of the time segment is configurable. The time-splitting applies to the application of bandpass solutions, flagging, averaging, and, for the spectral data, continuum-subtraction.
Additional continuum-imaging parameters are made selcal-loop-dependent: CLEAN_ALGORITHM, CLEAN_MINORCYCLE_NITER, CLEAN_GAIN, CLEAN_PSFWIDTH, CLEAN_SCALES, and CLEAN_THRESHOLD_MINORCYCLE. The last two parameters can have vectors for individual loops, and so this necessitates a new format, whereindividual loops are separated by ‘ ; ‘.
Overall control over the spectral processing is now provided by DO_SPECTRAL_PROCESSING. This defaults to false, meaning only the continuum processing will be done. Turning this to true will result in application of the selfcal gains, continuum subtraction, spectral imaging and image-based continuum subtraction being done - each of these are turned on by default.
Elevation-based flagging for the science observation is able to be configured through the pipeline parameters.
There are parameters to specify a given range of times to be used from the bandpass calibration observation - useful if you wish to use part of a long observation.
The arguments to the bandpass smoothing tools can be provided as a text string instead of the specific pipeline parameters - this will allow continued development of these tools without needing to keep the pipeline up-to-date with possible parameter inputs.
The new spectral imaging features (see above) are exposed through pipeline parameters FREQ_FRAME_SL and OUTPUT_CHANNELS_SL. The DO_BARY parameter has been deprecated.
The following bugs have been fixed
The continuum subtraction selavy jobs were using the wrong nsubx/nsuby parameters.
The pipeline scripts now check for the correct loading of the askapsoft & askappipeline modules, and exit if these are not loaded correctly.
Wildcards used in identifying polarisation data products are now correctly applied.
There are new parameters used in the pipeline (with their defaults):
DO_SPLIT_TIMEWISE=true
DO_SPLIT_TIMEWISE_SERIAL=true
SPLIT_INTERVAL_MINUTES=60
MULTI_JOB_SELFCAL=true
FAT_NODE_CONT_IMG=true
TILENCHAN_AV=18
SPLIT_TIME_START_1934=””
SPLIT_TIME_END_1934=””
BANDPASS_SMOOTH_ARG_STRING=””
BANDPASS_SMOOTH_F54=””
ELEVATION_FLAG_SCIENCE_LOW=””
ELEVATION_FLAG_SCIENCE_HIGH=””
OUTPUT_CHANNELS_SL=””
FREQ_FRAME_SL=bary
DO_SPECTRAL_PROCESSING=true
There are new default values for some pipeline parameters:
JOB_TIME_DEFAULT=”24:00:00”
CPUS_PER_CORE_CONT_IMAGING=6
FAT_NODE_CONT_IMG=true
NUM_PIXELS_CONT=6144
CELLSIZE_CONT=2
NCHAN_PER_CORE=12
NCHAN_PER_CORE_CONTCUBE=3
NCHAN_PER_CORE_SL=64
NUM_SPECTRAL_WRITERS=””
NUM_SPECTRAL_WRITERS_CONTCUBE=””
GRIDDER_WMAX_NO_SNAPSHOT=30000
GRIDDER_NWPLANES_NO_SNAPSHOT=257
CLEAN_MINORCYCLE_NITER=”[400,800]”
CLEAN_GAIN=0.2
CLEAN_SCALES=”[0,3,10]”
CLEAN_THRESHOLD_MINORCYCLE=”[30%, 0.5mJy, 0.03mJy]”
CLEAN_NUM_MAJORCYCLES=”[5,10]”
CLEAN_THRESHOLD_MAJORCYCLE=”0.035mJy”
SELFCAL_NUM_LOOPS=1
SELFCAL_INTERVAL=[200,200]
NUM_PIXELS_CONTCUBE=4096
CPUS_PER_CORE_CONTCUBE_IMAGING=8
CLEAN_CONTCUBE_MINORCYCLE_NITER=600
CLEAN_CONTCUBE_GAIN=0.2
CLEAN_CONTCUBE_THRESHOLD_MINORCYCLE=”[40%, 0.5mJy, 0.05mJy]”
CLEAN_CONTCUBE_THRESHOLD_MAJORCYCLE=0.06mJy
CLEAN_CONTCUBE_NUM_MAJORCYCLES=3
TILENCHAN_SL=18
DO_APPLY_CAL_SL=true
DO_CONT_SUB_SL=true
DO_SPECTRAL_IMAGING=true
DO_SPECTRAL_IMSUB=true
NUM_PIXELS_SPECTRAL=1024
CELLSIZE_SPECTRAL=8
SPECTRAL_IMAGE_MAXUV=2000
PRECONDITIONER_SPECTRAL_GAUSS_TAPER=”[20arcsec, 20arcsec, 0deg]”
GRIDDER_SPECTRAL_SNAPSHOT_IMAGING=false
GRIDDER_SPECTRAL_WMAX_NO_SNAPSHOT=30000
GRIDDER_SPECTRAL_NWPLANES=257
CLEAN_SPECTRAL_MINORCYCLE_NITER=800
CLEAN_SPECTRAL_GAIN=0.2
CLEAN_SPECTRAL_THRESHOLD_MINORCYCLE=”[45%, 3.5mJy, 0.5mJy]”
CLEAN_SPECTRAL_THRESHOLD_MAJORCYCLE=0.5mJy
CLEAN_SPECTRAL_NUM_MAJORCYCLES=3
0.23.3 (22 February 2019)¶
A further patch release, with a number of small pipeline fixes, along with several fixes to the processing software.
Processing:
The imager would produce slightly different residual and restored images when different values of nchanpercore were used. This was due to the final gridding cycle not being synchronised correctly. This has been fixed and the images are now indepenent of nchanpercore.
The tree reduction used by imager has been improved to have a smaller memory footprint across the cores.
The selavy component fitting is improved in the way negative components are handled. Unless negative components are explicitly accepted, if a fit results in one or more components being negative then that fit will be rejected.
The primary beam used by linmos now has a FWHM scaling by 1.09 lambda/D, which should be more accurate.
The FITSImage interface (in Code/Base/accessors) will now report a human-readable error message (rather than a number code) when an error occurs.
Pipelines:
CASDA uploads again include catalogues (which were left out due to fixes in 0.23.1).
There are new parameters
CIMAGER_MAXUV
andCCALIBRATOR_MAXUV
that allow the imposition of an upper limit to the uv values in the continuum imaging/self-calibration.Parsets for the imager were erroneously getting a “Cimager.Channels” selection that included the %w wildcard. This will no longer happen (unless cimager is used).
The default python module is now always loaded at the start of slurm scripts, to avoid python conflicts due to a user’s particular environment.
There are stronger checks on the number of cores allocated to spectral-line imaging, ensuring that the number of channels must be an exact multiple of the nchanpercore.
The scaling on the beam-wise noise plots has been fixed, so that the scaled MADFM should be closer to the standard deviation in the absence of signal.
Cube stats are now also generated for continuum-cube residual images.
Several scripts have been tidied up with the aim of avoiding spurious errors (validationScience, for instance).
The ASKAPsoft version was being left off FITS headers. This now reflects the version string from the askapsoft module.
0.23.2 (2 February 2019)¶
A patch release, fixing an issue with imager and a couple of minor pipeline issues:
The imager in spectral-imaging mode was not respecting the clean thresholds correctly. This could lead to over-cleaning, and the insertion of spurious clean components at noise peaks (particularly in continuum-subtracted spectral data).
A change has been made to the module setup, avoiding “module swap” in favour of “module unload / module load” - this addresses an occasional issue seen where the module environment can get corrupted by the swap command.
A fix has been made to the flagging parsets, solving a problem where the autocorrelation flagging and the time-range flagging were assigned to the same rule. If both were used, the time range was only flagged in the autocorrelations. They now appear as separate rules and so will be independent.
0.23.1 (22 January 2019)¶
A patch release, addressing a few bugs in both processing software and pipeline scripts
Pipelines:
Changes have been made to the scripts to make them robust in handling field names that contain spaces. This has also made them more robust to being run in a directory with a path that contains spaces.
An update has been made at Pawsey to the module used for the continuum validation task, and consequently a minor change has been made to the continuum sourcefinding script.
Processing:
Enhancements have been made to the continuum-subtraction task ccontsubtract to speed it up - initial tests indicate a speed-up of 6-8x depending on platform.
The Selavy fitting algorithm now defaults to including a test on the size of the fitted Gaussians. This will prevent spuriously large fits from making it through to the catalogue, which has had detrimental effects in the calibration & continuum-subtraction.
A fix was made to the imager, solving a problem where the spectral-imaging option merged the first channel of its allocation without checking the frequency.
Additionally, the user documentation has updated instructions about how best to set the modules on galaxy so that everything runs smoothly (see Central Processor Platform).
0.23.0 (10 December 2018)¶
A major release, addressing a number of issues with the processing software and the pipeline scripts.
Pipelines:
When multiple raw MSs are provided for a given beam (split up by frequency range), the pipeline is capable of recognising this, merging (after any necessary splitting), and handling all required metadata appropriately. The functionality should be the same no matter the structure of the raw data.
The selfcal job allocation (for the sbatch call) has been altered to request a number of nodes, rather than cores + cores-per-node. This should provide more predictable allocations.
The weights cutoff parameter given to Selavy is now fully consistent with the linmos cutoff.
Fixed a bug that meant the raw data was overwritten when calibration was applied, even when KEEP_RAW_AV_MS=true.
The TELESCOP keyword is now added to the FITS headers.
A bug was fixed that was preventing the full-resolution MSs being included in the CASDA upload.
New parameters SPECTRAL_IMAGE_MAXUV and SPECTRAL_IMAGE_MINUV that allow control over the UV distances passed to the spectral imager.
Various improvements to the gatherStats job, so that it will still run after the killAll script has been called, and that looks for the pipeline-errors directory before trying to use it.
Making the cubeStats script more robust against failures of a single process (so that it doesn’t hang but instead carries on as best it can).
Processing:
Imaging:
Fix a coordinate shift that was seen in spectral imaging, due to a different direction being provided by the advise functionality.
Calibration:
Efficiency improvements to ccalapply to help speed it up
Utilities:
Adjustment of the maximum cache size in mssplit to avoid out-of-memory issues
Trimming down of the pointing table in MSs produced by msconcat, so that very large tables do not result.
Selavy:
The restoring beam is now written into the component maps.
A significant change to the handling of the initial estimates for the Gaussian fits, making it more robust and avoiding downstream WCS errors that were hampering the analysis.
Minor catalogue fixes for component & HI catalogues
Segfaults in selfcal (3145)
0.22.2 (02 October 2018)¶
- Minor change to pipeline scripts:
nChan is now set to CHAN_RANGE_SCIENCE/1934 parameter instead of reading it from the raw measurement sets. Fixes the bug arising when working on subset channel range in the measurement sets.
0.22.1 (25 September 2018)¶
- A patch release to 0.22 to fix a couple of bugs:
Fixed issue with missing reading of visibilities causing zeros after calibration.
Added multi-row processing mode for Amplitude and StokesV flaggers.
0.22.0 (20 September 2018)¶
This release sees a number of changes & improvements to both the processing software and the pipeline scripts.
Pipelines:
There are new diagnostic plots produced, particularly for the spectral & continuum cubes. There is a python script run immediately following the imaging to calculate a range of statistics on a channel-by-channel basis, and this data is plotted on a per-image basis, as well as an overview plot showing the statistics for each beam at once.
The ability to specify the number of cores used for the continuum imaging has been improved, to make it more flexible and work better with slurm.
The behaviour of source-finding in the selfcal has changed. We now fit the full set of Gaussian parameters, and require contiguous pixels for the islands.
Several bugs were fixed:
Some FITS files were not having their header keywords updated correctly. This has now been fixed and streamlined.
The CASDA upload script was erroneously including multiple versions of the taylor.1 images, due to a bug introduced in 0.21.0. It was also dropping a few .fits suffixes in certain situations.
The cmodel-based continuum subtraction script had a bug with an undefined local variable that occured with particular parameter settings.s
The clean-model-based self-calibration script was getting the model image name wrong for Taylor-term images.
There are a number of changes to the default parameters:
The DO_MAKE_THUMBNAILS option is now true by default.
There is a new DO_VALIDATION_SCIENCE (true by default) to run the cube validation.
The snapshot imaging has been turned off by default, as this has proved to be more reliable in terms of image quality. Along with this, the number of w-planes has a default value that changes with the snapshot option: snapshot imaging has 99, non-snapshot imaging has 599.
The number of channels in the MS tile is exposed as a parameter for the bandpass & science datasets, taking the same default value as previously (54).
The “solver.Clean.solutiontype” parameter is exposed as a pipeline parameter for all imaging jobs.
The SB_JIRA_ISSUE is replaced by JIRA_ANNOTATION_PROJECT for schedblock annotations, although this functionality is currently only available to the askapops user.
Applications:
Selavy: - The Selavy HI catalogue now has better defined default values, and the NaN values that were appearing have been fixed (through use of masked arrays when fitting to the moment-0 map). - Selavy was previously occasionally dropping components from the catalogue through the application of acceptance criteria. This is now optional, and off by default. - Selavy was failing to calculate spectral indices in certain cases - this is now fixed. - The deconvolved sizes in the Selavy components catalogue are now calculated with better floating-point arithmetic, to avoid rare cases of NaNs. - The column widths for the VOTable catalogues are more tightly controlled, in line with the CASDA software.
Imaging: - Primary beam factory (used by linmos) able to handle elliptical Gaussian beams. Not fully implemented within linmos yet. - A new gridder AltWProjectVisGridder that allows the dumping of uvgrids to casa images. - Caching in the spectral-line imaging of imager is now done by channel. - The spectral-line imager will now correctly write the beam log when using multiple writers. - An issue with the beam fitting failing for very elongated beams has been remedied.
Casda upload: - The casdaupload utility was leaving out the path to measurement sets when making the XML interface file, even when the absolute path option was being requested. This is now fixed and all artifacts will have their absolute path used in this mode. - Similarly, checksums for the thumbnail images were not being created by casdaupload. This has been remedied.
Other: - The FITS accessor interface now better handles missing header keywords - if they are not present it now logs a warning but doesn’t exit. - Ccalapply has improved handling of flags, allowing write access. - Improvements to the efficiency of mssplit and msmerge. - The user documentation has a detailed tutorial on MS(MFS) imaging.
0.21.2 (31 July 2018)¶
A further patch release for the pipeline, fixing a few issues that have been seen on the Galaxy platform.
The previous fix for the OUTPUT directory is now included correctly in the release.
The fix for imagetype parameter in Selavy parsets generated by the pipeline has been extended to the continuum-subtraction jobs.
The bandpass validation log is copied to the diagnostics directory, as it includes useful information about the state of the dataset.
Errors involving ‘find’ (from the ‘rejuvenate’ function) are no longer reported in the slurm output when the file in question does not exist.
When aoflagger is used for the flagging, the slurm script ensures that the correct programming environment (PrgEnv module) is loaded prior to loading the aoflagger module.
The continuum cube imaging can now use more than one channel per core. This is accessed via the new parameter NCHAN_PER_CORE_CONTCUBE.
Added casa, aoflagger and bptool version reporting to the image headers and the copy of the config file, to enhance the reproducibility of the pipeline processing.
0.21.1 (17 July 2018)¶
A patch release fixing minor issues with the 0.21.0 version of the processing pipeline scripts. Only the scripts and the documentation are changed.
Fixes to the pipeline:
The bandpass validation script will now find the correct files when an OUTPUT directory is used.
Similarly, the statsPlotter script is now more robust against the use of the OUTPUT option.
Parallel processing enabled for the larger ccalapply jobs.
The Channels selection parameter for continuum imaging can be left out when NUM_CPUS_CONTIMG_SCI is provided, with a new parameter CHANNEL_SELECTION_CONTIMG_SCI available to specify a selection.
The snapshot imaging option is turned back on by default for all imaging with the pipeline, following further testing & feedback from commissioning & operations teams.
There is better specification of the imagetype parameter in the Selavy parsets - there were issues when imagetype=casa was used.
0.21.0 (6 July 2018)¶
A large release containing a number of updates to the pipeline scripts and to various aspects of the processing tools.
Pipeline updates:
Ability to use AOflagger instead of cflag.
Ability to use the continuum cubes to measure spectral indices of continuum components (using Selavy).
Fixing a bug where the CleanModel option of continuum-subtraction was using the wrong image name.
Allow self-calibration to use the clean model image as the model for calibration (in the manner of continuum-subtraction).
Improvement of the continuum subtraction Selavy parameterisations, to better model the continuum components. The Selavy parsets are now consistent with those used for the continuum cataloguing.
Collation of pipeline jobs that failed, for analysis by ASKAP Operations, to help identify pipeline or platform issues.
Use of an alternative bandpass smoothing task - smooth_bandpass.py (instead of plot_caltable.py).
Use of an additional bandpass validation script to produce summary diagnostic plots for the bandpass solutions.
Fixed a bug where the bandpass table name was not set correctly when the the DO_FIND_BANDPASS switch was turned off.
Addition of the spectral measurement sets, the continuum-subtraction models/catalogues, and the spectral cube beam logs to the list of artefacts to be sent to CASDA upon pipeline completion.
Added more robustness to the pipeline scripts to allow them to run on other systems, allowing the specification of the module directory and flexibility for running on non-Lustre filesystems.
Changes to some default parameters. Here are the parameters that have changed, with their new values (note that the WMAX and MAXSUPPORT gridding parameters now also adapt their default values according to whether snapshot imaging is turned on or off):
# Image type
IMAGETYPE_CONT=fits
IMAGETYPE_CONTCUBE=fits
IMAGETYPE_SPECTRAL=fits
# Bandpass calibration
DO_APPLY_BANDPASS_1934=true
BANDPASS_SMOOTH_FIT=1
BANDPASS_SMOOTH_THRESHOLD=1.0
# Continuum imaging
NUM_TAYLOR_TERMS=2
CLEAN_MINORCYCLE_NITER=2000
CLEAN_PSFWIDTH=256
CLEAN_THRESHOLD_MINORCYCLE="[20%, 1.8mJy, 0.03mJy]"
CLEAN_NUM_MAJORCYCLES="[5,10,10]"
CLEAN_THRESHOLD_MAJORCYCLE="0.03mJy"
SELFCAL_INTERVAL="[1800,1800,200]"
GRIDDER_SNAPSHOT_IMAGING=false
GRIDDER_WMAX_SNAPSHOT=2600
GRIDDER_MAXSUPPORT_SNAPSHOT=512
GRIDDER_WMAX_NO_SNAPSHOT=26000
GRIDDER_MAXSUPPORT_NO_SNAPSHOT=1024
# Continuum cube imaging
CLEAN_CONTCUBE_ALGORITHM=BasisfunctionMFS
CLEAN_CONTCUBE_PSFWIDTH=256
CLEAN_CONTCUBE_MINORCYCLE_NITER=2000
CLEAN_CONTCUBE_THRESHOLD_MINORCYCLE="[40%, 12.6mJy, 0.5mJy]"
CLEAN_CONTCUBE_THRESHOLD_MAJORCYCLE=0.5mJy
# Spectral imaging
NCHAN_PER_CORE_SL=9
NUM_SPECTRAL_WRITERS=16
ALT_IMAGER_SINGLE_FILE=true
PRECONDITIONER_LIST_SPECTRAL="[Wiener,GaussianTaper]"
PRECONDITIONER_SPECTRAL_GAUSS_TAPER="[30arcsec, 30arcsec, 0deg]"
PRECONDITIONER_SPECTRAL_WIENER_ROBUSTNESS=0.5
CLEAN_SPECTRAL_ALGORITHM=BasisfunctionMFS
CLEAN_SPECTRAL_PSFWIDTH=256
CLEAN_SPECTRAL_SCALES="[0,3,10,30]"
CLEAN_SPECTRAL_THRESHOLD_MINORCYCLE="[50%, 30mJy, 3.5mJy]"
CLEAN_SPECTRAL_MINORCYCLE_NITER=2000
GRIDDER_SPECTRAL_SNAPSHOT_IMAGING=false
GRIDDER_SPECTRAL_WMAX_SNAPSHOT=2600
GRIDDER_SPECTRAL_MAXSUPPORT_SNAPSHOT=512
GRIDDER_SPECTRAL_WMAX_NO_SNAPSHOT=26000
GRIDDER_SPECTRAL_MAXSUPPORT_NO_SNAPSHOT=1024
# Spectral source-finding
SELAVY_SPEC_OPTIMISE_MASK=false
SELAVY_SPEC_VARIABLE_THRESHOLD=true
SELAVY_SPEC_SNR_CUT=8
Processing tasks:
An MPI barrier has been added to the spectral imager to prevent race conditions in the writing.
Better handling of cases in the bandpass calibration that were previously (prior to 0.20.3) causing it to fail with SVD conversion errors.
Selavy will now report the best component fit (assuming it converges in the fitting), regardless of the chi-squared. If poor, a new flag will be set.
If the fit fails to converge, Selavy can reduce the number of Gaussians being fit to try to get a good fit.
A bug in Selavy was fixed to allow the curvature-map method of identifying components to better take into account the weights image associated with the image being searched.
A further bug in Selavy (the extraction code) was fixed to allow its use on images without spectral or Stokes axes.
The SNR image produced by Selavy now has a blank string for the pixel units.
The implementation of the variable threshold calculations in Selavy have been streamlined, to improve the memory usage particularly for large spectral cubes. There is also control over the imagetype for the images written as part of this algorithm.
The memory handling within linmos-mpi has been improved to reduce its footprint, making it better able to mosaic large spectral cubes.
Manager & ingest:
Improvements to the CP manager.
UVW calculations fixed in the course of testing new fringe rotator modes.
ASKAPsoft environment:
Incorporation of python-casacore in the cpapps build (used to create the askapsoft module at Pawsey).
Documentation:
Added a chapter to the user documentation on how to combine multiple epochs for spectral line data.
Added a chapter to the user documentation explaining the best way to do MS/MFS deconvolution in askapsoft
Added a page to the user documentation listing the release notes for each release.
0.20.3 (2 April 2018)¶
A patch release fixing a couple of calibrator issues:
The 0.20 updates to the calibrator to allow interaction with the calibration data service had prevented ccalibrator from writing more than one row to the output calibration table. This fix ensures the table that gets written has all the information when solving for time-dependent gains.
The bandpass calibrator would very occasionally fail with an error along the lines of “ERROR: SVD decomposition failed to converge”. This will now only trigger a WARN in the log file, but will not abort the program. Work is still being done to properly flag channels that suffer this.
And a couple of pipeline issues have been fixed:
The beams that are processed by the pipeline are now limited by the number of beams in the bandpass calibrator scheduling block (in the same way that the science SB is used to limit the number of beams).
Minor issues with copying the continuum validation results have been resolved.
Additionally, casacore (in 3rdParty) is now built with the python bindings, so that libcasa_python will be available.
0.20.2 (27 March 2018)¶
A patch release that fixes a few bugs in the build to do with missing directories:
Modified several build configurations so that missing directories do not make the build fail. Missing directories can be present as a result of a bug in our SVN to BitBucket sync which ignores empty directories (even if there is a .gitxxxx file in it). Subsequently, cloning the git repo causes these directories to be missing which can cause a failed build for some packages. In these cases, the build script has been changed to create the missing directories if they are missing.
Note there are no application code or documentation changes for this release.
0.20.1 (08 March 2018)¶
A patch release that fixes a few bugs in the pipeline:
Adds better robustness to the USE_CLI=false option, for use when the databases at MRO are unavailable.
A scripting error in the self-calibration script (for the Cmodel option).
Fixes to the defineArtifacts script, to better handle FITS extensions.
When the image-based continuum-subtraction option is run, the spectral source-finding job will now search the continuum-subtracted cube. The spectral source-finding will also handle sub-bands correctly.
There have also been fixes to ensure the continuum-subtracted cubes are created in appropriate FITS format and mosaicked correctly.
Copying of continuum validation files to the archive directory has been updated to reflect an improved directory structure.
It also makes a few minor changes to the processing software:
The Wiener preconditioner will now report in the log the amount by which the point-source sensitivity is expected to increase over the theoretical naturally-weighted level.
The casdaupload utility can now produce an XML file with absolute paths to data products, leaving them in place - rather than copying all data products to the upload directory. This is compatible with behaviour introduced in CASDA-1.10.
Ccalapply has a new parameter than can restrict the sizes of chunks presented in single iterations, using new options for the TableDataSource classes.
The component catalogue produced by Selavy had a minor error in the calculation of the error on the integrated flux (where the minor axis should have been used, the major axis was used instead).
Fixed issues with cmodel functional tests, relating to using the correct catalogue columns.
Fixed a failing scimath unit test.
The ingest pipeline now can apply phase gradients in parallel.
0.20.0 (09 February 2018)¶
This release sees the first version of the Calibration Data Service (CDS) and Sky Model Service (SMS) in deployable form. These components are intended to run independently of the ASKAPsoft pipelines. At first, they will require some configuration and data initialisation. Testing and feedback will then drive further development.
The CDS provides an interface to a database containing calibration parameters. The SMS allows access to the Global Sky Model data, primarily for the purpose of constructing local sky models.
Other changes in this release include:
- Pipelines:
Corrected the use of the $ACES environment variable when running the continuum validation script, so that pecularities of the local environment are appropriately dealt with.
Some corrections in pipeline scripts regarding FITS mode processing:
Ensures the continuum linmos image is copied at the field-level mosaicking job.
Ensures the spectral-line selavy job uses the correct file extensions.
Ensures the imcontsub job converts the contsub cube to fits at the end if we are working in FITS mode.
Updates the naming of the contsub cube to ensure consistency (removing .fits from the middle of it).
Improve copying of spectral weights images when running linmos to avoid ambiguities and prevent unnecessary files.
Added a parameter, DO_SOURCE_FINDING_FIELD_MOSAICS, to turn off source finding for individual fields and rely on the source finding for the final mosaic instead. This prevents unnecessary source finding jobs being launched.
Selavy source finding jobs now have scheduling block ID (SBID) passed in parsets.
The casdaupload utility can now handle cubelets (as well as spectra & moment-maps). These are included by the casda script in the pipeline.
TIME selection options in flagging are now exposed in pipeline scripts via TIME_FLAG_SCIENCE, TIME_FLAG_SCIENCE_AV and TIME_FLAG_1934. It is up to the user to provide suitable values.
Pipelines allow processing of scheduling blocks (SB) where the number of measurement sets (MS) is different to the number of beams. This addresses an issue where the SB have recorded 36 MSs but only a subset of them are valid.
The use of dcp for copying MSs from the archive is turned off by default to minimise the load on the hpc-data nodes (the method for doing this is not ideal).
- Processing Software:
Reduction in logging in the imager task.
Modifications to Selavy to include additional information in the headers of the spectra & related images (Object name, date-obs and duration, Project ID and SBID, history comments). This involved improvements to the image interface classes.
Fixed a problem where mslist output was corrupted by long field names.
Shortened objectID strings are now used in catalogues. No longer uses image name, but instead SBID + catalogue/data product type + sequence ID.
0.19.7 (11 January 2018)¶
A patch release that allows the pipelines to run correctly on native slurm, using srun to launch applications rather than aprun. This is timed to be available for the upgrade of the galaxy supercomputer to CLE6.
The release also has a slightly improved build procedure that better handles python dependencies, and updated documentation regarding the ASKAP processing platform at Pawsey.
No functional change is expected for the processing software itself.
0.19.6 (19 November 2017)¶
A patch release for both the processing and pipeline areas. This fixes a few bugs and introduces a few minor features to enhance the processing.
- Pipelines:
Default values of a number of parameters have been updated, particularly for the spectral-line imaging. Importantly, the default imager has been changed for all imaging jobs to be the new imager task.
Fix for the image-based continuum subtraction script. This uses scripts in the ACES repository, which have been recently updated, and this change allows the use of the new interface. Needs to be used with ACES revision number 47195 or later.
The bandpass solutions can now be applied to the calibrator observations themselves, producing calibrated MSs that could be used later for analysis.
The reference antenna for the bandpass calibration can be specified via the new config parameter BANDPASS_REFANTENNA.
Self-calibration with cmodel can now avoid using components below some nominated signal-to-noise level. It can also be forced to use PSF-shaped components for the calibration.
When copying raw per-beam measurement sets, there is now the option to use regular cp, instead of the dcp-over-ssh approach (which requires the ability to ssh to hpc-data).
The first stage of mosaicking now uses the weighttype=Combined option (see below), which should give a better reflection of the data in the event different beams have different weights. Previous behaviour can be used by setting the config parameter LINMOS_SINGLE_FIELD_WEIGHTTYPE=FromPrimaryBeamModel.
The following bugs have been fixed:
RM Synthesis is now turned off if only the Stokes-I continuum cube is being created (which is the default).
When using a component parset for self-calibration, the reference direction could be incorrect (if the full-resolution MS was absent). This has been fixed, by obtaining the direction from the averaged dataset.
The continuum source-finding will now not attempt to measure spectral terms of higher order than the number of terms requested in the imaging (for instance, if nterms=2, the spectral curvature will not be measured). Similarly, in that situation the .taylor.2 images will not be provided as mosaics or as final archived artefacts.
Processing software:
Cflag:
There was a bug where the StokesV flagger would crash with a segmentation fault on occasions where it was presented with a spectrum or time-series that was entirely flagged. It is now more robust against such datasets.
Imager:
The imager is now more robust against small changes in the frequency labels of channels, with an optional tolerance parameter available.
Selavy:
A few bugs were fixed that were preventing Selavy working for spectral-line cubes, where it was trying to read in the entire cube on all processing cores (leading to an out-of-memory error).
Moment-0 maps now have a valid mask applied to them.
Selavy can now measure the spectral index & curvature from a continuum cube, instead of fitting to Taylor-term images.
Duchamp version 1.6.2 has been included in the askapsoft codebase.
The deconvolved position angle of components is now forced to lie between 0 & 2pi, and its error is limited to be no more than 2pi.
Linmos:
Fixed a bug that meant (in some cases) only a single input image was included in the mosaic. Happened when the input images had masks attached to them (for instance, combination of mosaics).
New option of “weighttype=Combined” for linmos-mpi, that uses both the weight images and the primary beam model to create the output weights.
0.19.5 (8 October 2017)¶
A patch release that adds a few new bits of functionality:
The Selavy code has been updated to add to the catalogue specifications for the continuum island & component catalogues:
The component catalogue now has error columns for the deconvolved sizes, as well as for the alpha & beta values.
Additionally, the 3rd flag column now indicates where the alpha & beta values are measured from - true indicates they come from Taylor-term images.
The island catalogue now has:
An error column for the integrated flux density
Columns describing the background level, both the mean background across the island, and the average background noise level.
Statistics for the residual after subtracting the island’s fitted Gaussian components - columns for the max, mean, min, standard deviation and rms.
Columns indicating the solid angle of the island, and of the image restoring beam.
Occasional errors in converting the major/minor axis sizes to the correct units have also been fixed.
- The pipelines have been updated with new functionality and options:
The new ingest mode of recording one measurement set per beam is now able to be processed. The MS metadata is recorded from one of the measurement sets, and the splitting is done from the appropriate beam. For the science dataset, if no selection of channels or scans is required, and there is only a single field in the observation, then copying of the MS is done instead of splitting.
Stokes-V flagging is available for all flagging steps. This is performed in the same job as the dynamic amplitude flagging, and is parameterised by its own parameters - consult the documentation for the full list (essentially the same as FLAG_DYNAMIC parameters with STOKESV replacing DYNAMIC or DYNAMIC_AMPLITUDE).
Selection of specific spectral channels in the flagging tasks is now possible with CHANNEL_FLAG_1934, CHANNEL_FLAG_SCIENCE, and CHANNEL_FLAG_SCIENCE_AV.
A bug that meant the continuum source-finding job would fail to convert higher-order Taylor terms or continuum cubes to FITS format has been fixed.
A fix has been made to the bandpass-smoothing casa script call, adding in a –agg command-line flag to the casa arguments. This allows the plotting to be run correctly on the compute nodes.
Scripting errors in the flagging scripts that showed up when splitting was not being done have been rectified.
0.19.4 (21 September 2017)¶
A patch release covering the pipeline scripts and the processing software. The following bugs are fixed:
The pipeline configuration parameter FOOTPRINT_PA_REFERENCE will now over-ride the value of footprint.rotation in the scheduling block parset. Additionally, the scheduling block summary metadata files (created in the pipeline working directory) are now not regenerated if they already exist.
The metadata collection in the pipeline now does not fail if a FIELD in the measurement set has ‘RA’ in its name.
There was a memory leak in Selavy, causing an error to be thrown when dealing with fitted components, specifically when the numGaussFromGuess flag was set to false and a fit failed. The code now falls back to whatever the initial estimate for components was, even if that has fewer than the maximum number indicated by maxNumGauss.
There was a half-pixel offset enforced in the location of the fitted Gaussian when fitting to the restoring beam when imaging. This was resulting in a slightly incorrect restoring beam.
If there are multiple MSs in the SB directory, one can be processed by giving MS_INPUT_SCIENCE its full path, setting the SB_SCIENCE parameter appropriately, and putting DIR_SB=””.
0.19.3 (4 September 2017)¶
A patch release just covering the pipeline scripts. The following bugs are fixed:
The number of writers used in the spectral-line imaging when the askap_imager is used (DO_ALT_IMAGER=true) is now better described. The input parameter NUM_SPECTRAL_CUBES is now NUM_SPECTRAL_WRITERS, and the pipeline is better able to handle a single output (FITS) cube written by multiple writers.
The running of the validation script after continuum source-finding now has the $ACES environment variable set correctly. The validation script requires it to be set, and when it was not set within a user’s environment the script could crash.
The image-based continuum subtraction script has had two fixes:
The cube name was being incorrectly set when the single-writer FITS option was used
The working directory was the same for all sub-bands for a given beam. This could cause issues with casa’s ipython log file, resulting in jobs crashing with obscure errors.
0.19.2 (24 August 2017)¶
A patch release that fixes bugs in both the pipeline scripts and Selavy, as well as a minor one in casdaupload.
- Pipeline fixes:
The ‘contsub’ spectral cubes were not being mosaicked. This was caused by incorrect handling of the “.fits” suffix (it was being added for CASA images, not FITS image).
It was possible for the pipeline to attempt to flag an averaged MS even if the averaged MS was not being created. The pipeline is now more careful about setting its switches to cover this scenario.
The continuum validation reports are now automatically (by default) copied to a standard location, tagged with the user’s ID and timestamp of pipeline. This can be turned off by setting VALIDATION_ARCHIVE_DIR to “”.
The spectral imaging jobs were capable of asking for more writers than there were cores in the job. The pipeline scripts are now careful to check the number of writers, and ensure it is no more than the number of workers. The default number of writers has been changed to one.
The handling of FITS files by the inter-field mosaicking tasks was error-prone - files would either not be copied (in the case of a single field) or would not be identified correctly (for the spectral-line case).
- Pipeline improvements:
The image size (number of pixels) and cellsize (in arcsec) for the continuum cubes can now be given explicitly, and so be allowed to differ from the continuum images.
Some default cleaning parameters for continuum cube imaging have been changed as well.
- The following bugs in Selavy have been fixed:
There was an issue with the weight-normalisation option in Selavy, where the incorrect normalisation was applied if a subsection (in particular the first subsection) had no valid pixels present (ie. all were masked). The masking is now correctly accounted for.
There were bugs that caused memory errors in the spectral-line (HI) parameterisation of sources. This code has been improved.
The ‘fitResults’ files were reporting the catalogue twice, and producing the same catalogue for all fit types. Additionally, there was the possibility of errors if different fit types yielded different numbers of components for a given island.
Finally, the casdaupload utility would fail if presented with a wildcard that did not resolve to anything. It will now just carry on, ignoring that particular parameter.
0.19.1 (04 August 2017)¶
User documentation changes only. No code changes.
0.19.0 (06 July 2017)¶
New features:
linmos now produces mosaicks with correct masking of pixels in in both CASA and FITS formats.
linmos can also remove the contribution of the primary beam frequency dependence to the Taylor term images. This only applies to Gaussian primary beam models.
Added Selavy support for FITS outputs
Addition of ACES-OPS module to facilitate controlled dependency between ASKAPsoft and ACES Tools.
Parallelised the RM Synthesis module in Selavy.
New Selavy output - a map of the residual emission not covered by the fitted Gaussians in a continuum image.
Developed patch for casacore’s poor handling of the lanczos interpolation method.
Added support for casdaupload to handle spectral-line catalogues.
CASDA related Support for new image types.
Ensure calibration tables are uploaded to CASDA.
Added support for continuum validation script and results including CASDA upload.
Improvements to Selavy spectral-line parameterisation.
Selavy sets spectral index & curvature to a flag-value if not calculated rather than leaving as zero.
Bug fixes:
linmos, reduced memory footprint. A bug was found that was causing a complete image cube to accessed, when only the image shape was required. This has been fixed.
Selavy catalogues occasionally fail CASDA validation due to wide columns - fixed.
Fixed bug where restore.beam.cutoff value not read from parset when present.
Added missing beam log output to new imager.
Improved handling of failed processing and the effect of that on executing final diagnostics/FITSconversion/thumbnails jobs at end of pipeline.
Use number of beams in footprint rather than assume 36.
Minor bug fixes
0.18.3 (23 May 2017)¶
This patch release fixes the following bugs in the pipeline scripts:
Incorrect indexing of some self-calibration array parameters
Better handling of logic in determining the usage of the alternative imager.
Ensuring the image-based continuum-subtracted cubes are converted to FITS and handled by the CASDA upload. Also that this task is able to see cubes directly written to FITS by the spectral imagers.
Fixing handling of directory names so that extracted artefacts are found correctly for FITS conversion.
Removal of extraneous inverted commas in the continuum imaging jobscript.
Additionally, there is a new parameter USE_CLI, which defaults to true but allows the user to turn off use of the online services, should they not be available.
Finally, a number of the default parameters used by the bandpass calibration and the continuum imaging have been updated, following extensive commissioning work with the 12-antenna early science datasets. Here is a list of the changed parameters:
NCYCLES_BANDPASS_CAL=50
NUM_CPUS_CBPCAL=216
BANDPASS_MINUV=200
BANDPASS_SMOOTH_FIT=0
BANDPASS_SMOOTH_THRESHOLD=3.0
NUM_TAYLOR_TERMS=1
NUM_PIXELS_CONT=3200
CELLSIZE_CONT=4
RESTORING_BEAM_CUTOFF_CONT=0.5
GRIDDER_OVERSAMPLE=5
CLEAN_MINORCYCLE_NITER=4000
CLEAN_PSFWIDTH=1600
CLEAN_SCALES="[0]"
CLEAN_THRESHOLD_MINORCYCLE="[40%, 1.8mJy]"
CLEAN_NUM_MAJORCYCLES="[1,8,10]"
CLEAN_THRESHOLD_MAJORCYCLE="[10mJy,4mJy,2mJy]"
PRECONDITIONER_LIST="[Wiener]"
PRECONDITIONER_GAUSS_TAPER="[10arcsec, 10arcsec, 0deg]"
PRECONDITIONER_WIENER_ROBUSTNESS=-0.5
RESTORE_PRECONDITIONER_LIST="[Wiener]"
RESTORE_PRECONDITIONER_GAUSS_TAPER="[10arcsec, 10arcsec, 0deg]"
RESTORE_PRECONDITIONER_WIENER_ROBUSTNESS=-2
SELFCAL_NUM_LOOPS=2
SELFCAL_INTERVAL="[57600,57600,1]"
SELFCAL_SELAVY_THRESHOLD=8
RESTORING_BEAM_CUTOFF_CONTCUBE=0.5
RESTORING_BEAM_CUTOFF_SPECTRAL=0.5
0.18.2 (5 May 2017)¶
This patch release fixes the following bugs in the pipeline scripts:
The ntasks-per-node parameter for the continuum subtraction could still be more than ntasks for certain parameter settings.
When using a subset of the spectral channels, the new imager jobs were not configured properly, with some elements trying to use the full number of channels.
Mosaicking of the image-based-continuum-subtracted cubes was not waiting for the completion of the continuum subtraction jobs, so would invariably fail to run correctly.
The image-based continuum-subtraction jobs are now run from separate directories, so that ipython logs can not conflict.
The spectral source-finding job had an error in the image name in the parset.
Mosaicking of the continuum-cubes now creates separate weights cubes for each type of image product.
Continuum imaging with the new imager has been improved, fixing inconsistencies in the names of images.
The PNG thumbnails were not being propagated to the CASDA directory.
The noise map produced by Selavy is now included in the set of artefacts converted to FITS and sent to CASDA.
Additionally, the ability to impose a position shift to the model used in self-calibration has been added, with the aim of supporting on-going commissioning work.
0.18.1 (13 April 2017)¶
This patch release sees a few bug-fixes to the pipeline scripts:
When re-running the pipeline on already-processed data, where the raw input data no longer exists in the archive directory, the pipeline was previously failing due to it not knowing the name of the MS or the related metadata file. It now has the ability to read MS_INPUT_SCIENCE and MS_INPUT_1934 and determine the metadata file from that. It will also not try to run jobs that depend on the raw data.
The new imager used in spectral-line mode can now be directed to create a single spectral cube, even with multiple writers, via the ALT_IMAGER_SINGLE_FILE and ALT_IMAGER_SINGLE_FILE_CONTCUBE parameters.
There have been changes to the defaults for the number of cores for spectral imaging (from 2000 to 200) and the number of cores per node for continuum imaging (from 16 to 20), based on benchmarking tests.
In addition, the following bugs were fixed:
The ntasks-per-node parameter could sometimes be more than ntasks, causing a slurm failure.
The self-calibration algorithm was not retaining images from the intermediate loops.
The image-based continuum subtraction script was not finding the correct image cube.
0.18.0 (29 March 2017)¶
New features and updates:
Scheduling block state changes, in conjunction with a new TOS release:
The CP manager now monitors the transition from EXECUTING to OBSERVED, and the ICE interfaces have been updated accordingly.
The pipeline will now transition the scheduling block state from OBSERVED to PROCESSING at the beginning of processing. This will only be done for scheduling blocks in the OBSERVED state, and will apply to both the science field and the bandpass calibrator.
Python libraries:
3rdParty python libraries have been updated to current versions. This applies to: numpy, scipy, matplotlib, pywcs, pytz, and APLpy. The current astropy package has been added, and pyfits has been removed. The python scripts in Analysis/evaluation have been updated to be consistent with these new packages.
There is a new script in Analysis/evaluation, makeThumbnailImage.py, that produces grey-scale plots of continuum images, and has the capability to add weights contours and/or continuum components. This script is used by the makeThumbnails script in the pipeline, as well as the new diagnostics script (that produces more complex plots aimed at being aids for quality analysis).
Calibration & Imaging changes:
The residual image is now the residual at the end of the last major cycle. (Previously, it was the residual at the beginning of the last major cycle.)
The residual images now have units of Jy/beam rather than Jy/pixel, and have the restoring beam written to the header.
When the “restore preconditioner” option is used in imaging, the residual and psf.image are also written out for this preconditioner.
Pipeline updates:
There is a new pipeline parameter, CCALIBRATOR_MINUV, that allows the bandpass calibration to exclude baseline below some value.
Minor errors and inconsistencies in some catalogue specifications have been fixed, with the polarisation catalogue being updated to v0.7.
The spectral-line catalogue has been added to the CASDA upload part of the pipeline, and has been renamed to incorporate the image name (in the line of other data products).
There are new pipeline parameters SELFCAL_REF_ANTENNA & SELFCAL_REF_GAINS that allow the self-calibration to use a reference antenna and/or gain solution.
A weights cutoff for Selavy can now be specified via the config file using the new parameters SELAVY_WEIGHTS_CUTOFF & SELAVY_SPEC_WEIGHTS_CUTOFF (rather than using the linmos cutoff value).
The new imager is better integrated into the pipeline, with DO_ALT_IMAGER parameters for CONT, CONTCUBE & SPECTRAL.
It is possible to make use of the direct FITS output in the pipeline, by using “IMAGETYPE_xxx” parameters for CONT, CONTCUBE & SPECTRAL. Note that this is still somewhat of a work-in-progress.
Bug fixes:
Casacore v2 had several patches added that had been left out of the upgrade. Notably a patch allowing the use of the SIGMA_SPECTRUM measurement set column following concatenation of measurement sets.
The mssplit utility has been made more robust with memory allocation when splitting large datasets.
Better checking of the size of SELFCAL- and imaging-related arrays in the pipeline configuration, particularly when not using self-calibration.
[Weights bug in Selavy]
The continuum-subtracted cubes were not being mosaicked by the pipeline.
The pipeline is more robust against errors encountered when obtaining the metadata at the beginning. It can better detect when a corrupted metadata file is present, and re-run the extraction of that metadata.
An error in handling the beam numbering for non-zero beam numbers was identified & fixed.
The pipeline Selavy jobs were using the incorrect weights cutoff, leading to them not searching the full extent of the image.
The use of the PURGE_FULL_MS flag in the pipelines will now not trigger the re-splitting (and subsequent processing) of the full-resolution dataset.
0.17.0 (24 February 2017)¶
New features:
Capability for direct FITS output from imager. The “fits” imagetype is now supported for cimager and imager. This should be considered “beta” as the completeness of the header information for post processing has not been confirmed. This enables the parallel write of FITS cubes which considerably improves the performance of spectral line imaging.
Selavy’s RM Synthesis module can export the Faraday Dispersion Function to an image on disk.
New source-finding capabilities in the processing pipelines, with a spectral-line source-finding task added (using Selavy), and the option of RM Synthesis done in the continuum source-finding.
The full-resolution measurement set can be purged by the pipeline when no longer needed (ie. after the averaging has been done, and if no spectral-line imaging is required). This will help to minimise unncessary disk-space usage.
CASDA upload is now able to handle extracted spectral data products (object spectra and moment maps etc) that are produced by the source-finding tasks.
A few relatively minor additions have been made to the pipeline scripts:
A minimum UV distance can be applied to the bandpass calibration.
The checks done on the self-calibration parameters are less restrictive and less prone to give warning messages.
Mosaicking at the top level (combining FIELDs) is now not done when there is only a single FIELD.
User documentation has been updated to better reflect the current arrangements with Pawsey (e.g. host names and web addresses). It also describes new modules that are available, as well as alternative visualisation options using Pawsey’s zeus cluster.
Bug fixes:
Imaging:
The brightness units in the restored images from the new imager are now correctly assigned (they were ‘Jy/pixel’ and are now ‘Jy/beam’). The beam is also now written correctly.
The beam logs (recording the restoring beam at each channel of an image cube) are now read correctly - previously the comment line at the start was not being ignored.
A number of fixes for the spectral line imaging mode of “imager” have been implemented. These fix issues with zero channels caused by flagging.
Analysis:
The Faraday Dispersion function in Selavy’s RM Synthesis module was being incorrectly normalised. It is now normalised by the model Stokes I flux at the reference frequency.
Pipelines:
When using more than one Taylor term in the imaging, the continuum subtraction with cmodel images was not working correctly, with incomplete subtraction. This was due to a malformed parset generated within the pipeline. This has been fixed, and the continuum subtraction works as expected.
The beam logs are now correctly passed to Selavy for accurate flux correction of extracted spectra.
Job dependencies for the mosaicking and source-finding jobs have been fixed, so that all jobs start when they are intended to. The mosaicking jobs now only start when they are needed, to avoid wasting resources.
The project ID was incorrectly obtained from the schedblock service when there was more than one word in the SB alias.
The SELAVY_POL_WRITE_FDF parameter was incorrectly described in the documentation - it has been renamed SELAVY_POL_WRITE_COMPLEX_FDF.
0.16.1 (16 December 2016)¶
A patch release that is largely bug fixes, with several minor updates to the pipeline scripts.
New features:
The pipelines will now accept a list of beams to be processed, via a comma-separated list of beams and beam ranges - for instance 0,1,4,7-9,16. This should be given with the BEAMLIST configuration parameter. If this is not given, it falls back to using BEAM_MIN & BEAM_MAX as usual.
An additional column is now written to the stats files, showing the starting time of each job.
There is a new parameter FOOTPRINT_PA_REFERENCE that allows a user to specify a reference rotation angle for the beam footprint, should it not be included in the scheduling block parset.
There is a new parameter NCHAN_PER_CORE_SPECTRAL_LINMOS that determines how many cores are used for the spectral-line mosaicking. This helps ensure that the job is sized such that the memory load is spread evenly.
Bug fixes:
Imaging:
Improvements to the new imager to handle writers who do not get work due to the barycentring.
Improvements to the allocation of work within the new imager.
RM Synthesis & Selavy:
The new RM Synthesis module was not correctly respecting the ‘%p’ wildcard in image names, which also affected extraction run from within Selavy. This has been fixed.
Pipelines:
The findBandpass slurm job had a bug that stopped it completing successfully.
A number of bugs were identified with the mosaicking:
The Taylor term parameter was set incorrectly in the continuum mosaicking scripts.
The image name was not being set correctly in the spectral-line mosaicking.
The job dependencies for the spectral-line mosaicking have been fixed so that all spectral imaging jobs are included.
The askapsoft module is now loaded more reliably within the slurm jobs.
The return value of the askapcli tasks is now tested, so that errors (often due to conflicting modules) can be detected and the pipeline aborted.
A certain combination of parameters (IMAGE_AT_BEAM_CENTRES=false and DO_MOSAIC=false) meant that the determination of fields in the observation was not done, so no science processing was done. This has been fixed so that the list of fields is always determined.
A couple of bugs in the source-finding script were fixed, where the image name was incorrectly parsed, and the Taylor 1 & 2 images were not being found.
The footprint position angle for individual fields was incorrectly being added to the default value listed in the scheduling block parset.
To avoid conflicts between source-finding results of different images, the artefacts produced by selavy (catalogues and images) now incorporate the image name in their name. The source-finding jobs are also more explicit in which image they are searching.
Finally, two deprecated scripts have been removed from the pipeline directory.
0.16.0 (28 November 2016)¶
A release with a number of bug fixes, new features, and updates to the pipeline scripts
New features:
Rotation Measure synthesis is now possible within the Selavy source-finder. This extracts Stokes spectra from continuum cubes at the positions of identified continuum components, performs RM Synthesis, and creates a catalogue of polarisation properties for each component. While still requiring some development, most features are available and should permit testing.
The new imager, which was made available in an earlier release, has been added to the askapsoft module at Pawsey.
Bug fixes for processing software:
The bandpass calibrator cbpcalibrator will now not allow through a bandpass table with NaN values in it. If NaNs appear in solving the bandpass, then cbpcalibrator will throw an exception. In the process, the GSL library used in 3rdParty has been updated to v1.16.
The writing of noise maps by Selavy (in the VariableThreshold case) has been streamlined, so that making such maps for large cubes is more tractable.
Pipeline updates:
The driving script for the ASKAP pipeline is now called processASKAP.sh, instead of processBETA.sh. The latter is still available, but gives a warning before callling processASKAP.sh. All interfaces remain the same.
Linear mosaicking has been improved:
It is now available for spectral-line and continuum cubes, in addition to continuum images.
Mosaics are made for each field, and for each tile if the observation was done with the “tilesky” mode.
The continuum mosaicking can also include mosaics of the self-calibration loops.
The pipelines make better use of the online services of ASKAP, to determine things like the footprint (location of beams). This makes calculations more internally self-consistent.
When running self-calibration, some parameters can be given different values for each loop. This includes parameters for the cleaning, the source-finding, and the calibration. More flexibility is also provided for the source-finding within the self-calibration.
Processing of BETA datasets are made possible via an IS_BETA parameter, which avoids using the online system to obtain beam locations, and changes the defaults for the data location.
Smoothing of the bandpass solutions is now possible, using a script in the ACES repository to produce a new calibration table. It also allows plotting of the calibration solutions.
More flexibility is allowed for the number of cores used in the continuum imaging.
A notable bug was fixed that led to incorrect calibration and continuum-subtraction when Taylor-terms were being produced (i.e. nterms>1)
Various other more minor bug fixes, related to logging, stats files, and default values of parameters (for instance, the default for cmodel was to use a flux cutoff that was too high).
0.15.2 (26 October 2016)¶
This is a patch release that fixes several issues:
The parallel linear mosaicking tool linmos-mpi has been patched to correct a bug that was initialising cube slices incorrectly.
Several fixes to the CP manager and the pipeline scripts were made following end-to-end testing with the full ASKAP online system:
The CP manager will send notifications to a nominated JIRA ticket upon SB state changes.
Several fixes were made to the CASDA uploading and polling scripts, to ensure accurate execution. The capability of sending notifications to a JIRA ticket has also been added.
The Project ID is now taken preferentially from the SB, rather than the config file.
The linear mosaicking in the pipelines is now not turned off when only a single beam is processed.
0.15.1 (19 October 2016)¶
This is a patch release that fixes a couple of issues:
The bandpass calibrator cbpcalibrator has had its run-time improved by changing the way the calibration table is written. It is now written in one pass at the completion of the task - this reduces the I/O overhead and greatly reduces the run-time for larger datasets.
The pipeline settings for the flagging have been changed. The default settings now are to have the integrate_spectra option switched on, and the integrate_times and flat amplitude options switched off. This is the same approach as used in 0.14.0-p2 and earlier, and so should avoid the case of most of the dataset being flagged (as was seen with ADE data using the default settings in 0.15.0).
The flagging step for the average dataset now uses a different check-file to the full-size dataset flagging.
0.15.0 (10 October 2016)¶
This release sees a number of bugs fixes and improvements.
Improved the efficiency of the msmerge operation by allowing the writing of arbitrary tile-sizes and the mssplit by forcing bulk read operations from the source measurement set when possible.
To be consistent with changes made to Cimager (ASKAPSDP-1607), Simager has been changed to only access cross-correlations.
Parallel linmos - a new application linmos-mpi with the same interface as linmos has been added. This will distribute the channels of the cube between mpi ranks and process them separately. Writing each channel to the output cube individually. This should allow a full resolution cube to be mosaicked.
Improved Selavy HI emission catalogue, with a more complete set of parameters available. This is now turned on by an input parameter Selavy.HiEmissionCatalogue.
JIRA notification for Scheduling Block status changes.
Pipeline updates:
The bandpass calibration approach has changed slightly. All beams of the calibrator will be processed up to the requested BEAM_MAX - the BEAM_MIN parameter only applies to the science dataset.
There is more flexibility in specifying flagging thresholds for the dynamic flagger. Each instance of the flagging can have different thresholds for the integrateSpectra & integrateTimes options, and both of these are now available for the bandpass calibrator.
When uploading to CASDA and upon successful ingest into CASDA, the SB state can be transitioned through the state model.
Initial support for the new imager.
Modified CBPCALIBRATOR to reference the XX and the YY visibilities independently to the XX and YY of the reference antenna.
Added ability to playback in any number of loops in Correlator and TOS Simulators.
- Bug fixes:
Pipelines:
When components were used in the pipeline for self-calibration or continuum subtraction, the reference direction was not being interpreted correctly, leading to erroneous positions.
The bandpass calibration table was not inheriting the complete path to it - it is now put in a standard location and all scripts correctly point to it.
More robustness added to the source-finding job so that it doesn’t run if the FITS conversion fails.
Documentation fixes to names of the MS utility functions.
Fixing casdaupload to handle images that don’t have associated thumbnails, and to set the correct write permissions of the upload directory.
Selavy’s extraction of moment maps and cubelets was not working correctly when a subsection was given to Selavy. These calculations have also been improved slightly to better handle the spectral increments.
Minor-fixes to new imager to deal with brittle logic in the channel allocations in spectral line mode. My fix for this essentially gives all the workers the same info as the master.
0.14.0-p2 (25 September 2016)¶
A further update only to the pipeline processing:
Changes to the directory structure created by the pipeline. Each field in the MS is given its own directory, within which processing on all beams is done. The bandpass calibrator likewise gets its own directory. All files & job names are now identified by the field and the beam IDs.
Flagging of the science data is now done differently. The MS is first bandpass-calibrated, and then flagged. After averaging, there is the option to run the flagging again on the averged data. The flagging for the bandpass calibrator has not been changed.
The dynamic flagging for the science data also allows the use of both integrateSpectra and integrateTimes, with the former no longer done by default.
Modules are loaded correctly by the scripts and slurm jobs before particular tasks are used, so that the scripts are less reliant on the user’s environment.
Better handling of metadata files, particularly if a previous metadata call had failed.
The FITS conversion and thumbnail tasks correctly interact with the different fields, and the thumbnail images make a better measurement of the image noise, taking into account any masked regions from the associated weights images.
The cleaning parameter Clean.psfwidth is exposed to the configuration file.
Bugs in associating the footprint information with the correct field have been fixed.
If the CASDA-upload script is used to prepare data for deposit, the scheduling block state is transitioned to PENDINGARCHIVE.
0.14.0-p1 (9 September 2016)¶
An update to the pipeline processing only:
Fixing a bug in the handling of multiple FIELDs within a measurement set. These are now correctly given their own directory for the processed data products.
The footprint parameters are now preferentially determined from the scheduling block parset (using the ‘schedblock’ command-line utility). If not present, the scripts fall back to using the config file inputs.
The metadata files (taken from mslist, schedblock and footprint.py) are re-used on subsequent runs of the pipeline, rather than re-running each of these tools.
The default bucketsize for the mssplit jobs has been increased to 1MB, and made configurable by the user. The stripe count for the non-data directories has also been changed to 1.
0.14.0 (11 August 2016)¶
A major release, with several new features and improvements for both the imaging software and the pipeline scripts.
A new imager in under test in this release, currently just called “imager” and it has the following features:
In continuum mode it allows a core to process more than one channel. This has a small cost in memory and a proportional increase in disk access. But allows the continuum imaging to proceed with a much smaller footprint on the cluster. This will allow simultaneous processing of all beams in a coming release.
Spectral line cubes can be made from measurement sets that are from different epochs. The epochs are imaged separately but merged into the same image for minor-cycle solving.
The output spectral line cubes can be in the barycentric frame. This is currently just nearest neighbour indexing. But the possibility of interpolation has not been designed out.
The concept of “multiple writers” has been introduced to improve the disk access pattern for the spectral line mode. This breaks up the cube into frequency bands. These can be recombined post-processing.
If you really want to increase the performance for many major cycles you can also turn on a shared memory option which stores visibility sets in memory throughout processing.
The imager takes the same parset as Cimager - but extra key-value pairs are required to implement the features.
This new imager is still under test and we have not added the hooks into the pipeline yet.
- Other updates to the imaging code include:
Simager is now more robust against completely-flagged channels - such channels will now be set to zero in the output cube, instead of failing the simager job.
The extraction of spectra done by Selavy is now more robust and better able to handle multiple components and distributed processing.
Selavy now accepts a reference direction when providing a components parset - the l & m coordinates are calculated relative to this, rather than the image centre.
The restore solver can now accept its own preconditioner parameters, in addition to the general parameters used by the other solvers. If specified, a second set of restored images will be written with suffix “.alt.restored”.
- The pipeline scripts have seen the following updates:
There is a new option to have a different image centre for each beam, rather than a common pixel grid for all images. This uses the beam centre location taken from the footprint.py utility (an external task in the ACES subversion area).
The self-calibration can now use cmodel to generate a model image, instead of using a components parset.
There are new tasks to:
Apply the gains calibration to the averaged measurement set
Image the averaged measurement set as “continuum cubes”, in multiple polarisations
Apply an image-based continuum-subtraction following the creation of the spectral-line cubes. This makes use of an ACES python script to fit a low-order polynomial to each spectrum in the cube.
The headers of the FITS files created by the pipelines now have a wider range of metadata, including observatory and date-obs keywords, as well as information about the askapsoft & pipeline versions.
The restore preconditioner options mentioned above are available through “RESTORE_PRECONDITIONER_xxx” parameters, for the continuum imaging only (it is not implemented for simager).
Several bugs were fixed:
The continuum subtraction was failing when using components if no sources were found - it now skips the continuum subtraction step.
The askapdata module was, in certain situations, not loaded correctly, leading to somewhat cryptic errors in the imaging.
The parsing of mslist to obtain MS metadata would sometimes fail, depending on the content of the MS. It is now much more robust.
The default for TILENCHAN_SL has been increased to 10, to counter issues with mssplit running slow.
0.13.2 (19 July 2016)¶
This bug-fix version addresses a few issues with the imaging & source-finding code, along with minor updates to the pipeline scripts. The following bugs have been fixed in the processing software:
Caching of the Wiener preconditioner is now done, so that the weights are only calculated once for all solvers and the filters are only calculated once for all major cycles, scales & Taylor-terms. This has the effect of greatly speeding up the imaging, particularly for large image sizes.
The BasisfunctionMFS solver has had the additional convolution with the PSF removed. This fixes a bug where central sources were being cleaned preferentially to sources near the edge of the image. It also improves the resolution and SNR of minor-cycle dirty images.
From the update to casacore-2 in 0.13.0, linmos would fail when mosaicking images without restoring beams. This has been fixed (and behaves as it did prior to 0.13.0).
The size check in Selavy that rejects very large fitted components has been re-instated. This should allow the rejection of spurious large fitted components. The minimum size requirement (which forced sizes to be >60% of the PSF) has been removed.
- And the pipeline has seen these fixes:
The resolution of the input science measurement set, when not given explicitly in the config file, is now done properly in all cases, rather than just for the case of splitting & flagging.
The pipeline now allows clipping in the snapshot option of the gridding - this improves performance at high declinations, where different warping between snapshots could introduce sharp edges to the weights image.
The pipeline also allows the use of a weights cutoff in the Selavy job used in self-calibration, to avoid the presence of these sharp cutoffs seen at high declinations.
0.13.1 (24 June 2016)¶
This bug-fix version primarily addresses issues with the processing pipelines. The following bugs have been fixed:
Non-integer image cell sizes were not being interpreted correctly. These values can now be any decimal value.
A change in the mslist output format with casacore v2 meant that the Cmodel continuum subtraction script was not reading the correct reference frequency. This caused the cmodel job to fail for the case of nterms>1. The parsing code has been fixed.
The archiving scripts had a few changes:
The resolution of filenames & paths has been fixed.
The source-finding is now run on FITS versions of the images
The catalogue keys in the observation.xml are now internally consistent.
The way thumbnail sizes are specified in the pipeline configuration file has changed slightly.
Related to the above changes, the C++ code has had a couple of changes:
casdaupload now correctly puts the thumbnail information in the <image> group in the observation.xml file.
Fixes were made to the Selavy VOTable output to fix formatting errors that were preventing it passing validation upon CASDA ingest.
- Other C++ code changes include:
Fixes to the output files from the crossmatch utility.
Updates to the slice interfaces for compatibility with the TOS.
The documentation has also been updated, with updated descriptions of parameters that have changed as a result of the above, a few typos fixed, and new information about the management of data on Pawsey’s scratch2 filesystem.
0.13.0 (31 May 2016)¶
This version fixes a few issues with the processing pipelines, fixes some bugs with the source-finder and casda upload utility, and moves the underlying code to use version 2 of the casacore package.
- The pipeline scripts have seen the following changes:
The requested times for the slurm jobs are now individually configurable via parameters in the processBETA config file.
The Pawsey account can be explicitly given, allowing the use of the scripts under other accounts on magnus.
The linmos job now properly checks the CLOBBER parameter, and will avoid over-writing mosaicked images if CLOBBER=false.
There is now an archiving option to the pipeline, which includes:
conversion of images to FITS format
creation of PNG ‘thumbnail’ versions of the 2D images
staging of data to a directory for ingest into CASDA
- The processing software had the following changes:
The casacore package has been updated to version 2.0.3, with corresponding changes throughout the ASKAPsoft code tree.
NOTE that this has resulted in the code not building on OS X Mavericks (10.9).
The Selavy sourcefinder had two changes:
Errors on the fitted parameters are now reported in the component catalogue.
A bug that stopped Selavy running the variable-threshold option when the SNR image name was not specified has been fixed.
The casdaupload utility now requires the observation start and end times to be specified if no measurement set is provided.
0.12.2 (24 May 2016)¶
A bug fix release for the processing pipeline. This fixes a problem where the mosaicking task was still assuming beam IDs that had a single integer - ie. it was looking for image.beam0.restored instead of image.beam00.restored.
0.12.1 (18 May 2016)¶
This is a simple patch release that fixes a couple of bugs, one of which affected the performance of both the source-finder and the pipelines.
The measurement of spectral indices for fitted components to continuum Taylor-term images was being done incorrectly, leading to erroneous values for spectral-index and spectral-curvature. This, in turn, could lead to inaccuracies or even failures in the continuum-subtraction task of the pipeline (when the CONTSUB_METHOD=Cmodel option was used). This only affected version 0.12.0 (released on 8 May 2016), and is fully corrected in 0.12.1.
The other bug enforces the total number of channels processed by the pipelines to be an exact multiple of the averaging width (NUM_CHAN_TO_AVERAGE). In previous versions, the pipeline scripts would press on, but this would potentially result in errors in the slurm files and jobs not executing. Now, should NUM_CHAN_TO_AVERAGE not divide evenly into the number of channels requested, the script will exit with an error message before submitting any jobs.
0.12.0 (8 May 2016)¶
This version has a number of changes to the processing applications and the pipeline scripts.
- Bugs that have been fixed in the processing applications include:
The deconvolution major cycles were using out-of-date residual values when logging and testing against the threshold.majorcycle parameter. This is now fixed.
The initialisation of calibrator input now depends more closely on the input parameters nAnt, nBeam & the calibrator model, rather than the first chunk of the data - this allows the shape of the data cube to change throughout the dataset (which will help with data imported from MIRIAD/CASA).
Simager was showing a cross-shaped artefact when Wiener preconditioning was used, even with the preservecf parameter set to true. This parameter is now recognised, and the artefact is no longer seen.
Full polarisation handling is now possible with simager (in the same manner as for cimager).
Simager was crashing when no preconditioner was given - this has been fixed.
The casdaupload task now conforms to the current CASDA requirements of allowing multiple SBIDs, and of reporting the image type.
Selavy’s Gaussian fitting is now more able to fit confused components that are not immediately identified from the initial estimates.
Selavy was also failing when given images of a particular name (short, without a full-stop). This has been fixed.
- The pipeline scripts have had a number of improvements:
They are more robust for processing ADE data, with >9 beams and >6 antennas.
The flagging tasks have been improved, with:
Flagging of autocorrelations an option
The selection flagger (that does antenna-based & autocorrelations) is done first, along with (an optional) flat amplitude threshold.
The dynamic flagging is done as the second pass
There is more user control over these individual elements
New parameters are available in the scripts, to make use of the snapshotimaging.longtrack parameter in the gridding, and normalisegains option in the self-calibration. The latter improves the performance of the self-calibration, approximating phase-only self-calibration.
The slurm jobfiles are now more robust to the user’s environment - if the askapsoft module has not been loaded, it will be in the jobfile, and the user can request a different version.
0.11.2 (28 March 2016)¶
This release is a relatively small bug-fix update, primarily fixing a bug in cimager.
This bug would prevent a parallel job completing in the case of the major cycle threshold being reached prior to the requested maximum number of major cycles.
- Other changes include:
The pipeline scripts have a few minor fixes to the code to improve reliability, and ensure the correct number of cores used for jobs is reported in the statistics files.
The only change to the ingest pipeline (within askapservices) incorporates an extra half-cycle wait following fringe-rotator update.
0.11.1 (8 March 2016)¶
The imaging software now incorporates the preservecf option (released in 0.11.0) into the SphFunc gridder, and introduces a new option to the gridding - snapshotimaging.longtrack - that predicts the best fit W plane used for the snapshot imaging, finding the plane that minimises the future deviation in W. This can have substantial savings in processing time for long tracks.
The pipeline scripts have seen a number of minor improvements and fixes, with improved alternative methods for continuum subtraction, and improved reporting of resource usage (including a record of the number of cores used for each job). The user configuration file is now also copied to a timestamped version for future reference.
The ingest pipeline code has incorporated changes resulting from the recent commissioning activities.
0.11.0 (15 February 2016)¶
A key change made in the processing software relates the preconditioning. There is a new parameter preconditioning.preservecf that should be set to true for the case of using WProject and the Wiener preconditioner. This has fixed a couple of issues - at low (negative) robustness values, the cross-shaped artefact that was sometimes seen has now gone, and the performance should now more closely match that expected from robust weighting for the full range of robustness values.
- Several other bugs were fixed:
Linmos had a bug (that was introduced in version 0.10) where automatically-generated primary beams were being set to the position of the first image.
The multiscale-MFS solver had a small bug that would lead to higher-order terms being preconditioned multiple times.
Cmodel had bugs related to the reading of Selavy catalogues, and correctly representing deconvolved Gaussians. It now works correctly with such data.
Simager would fail were no preconditioners supplied.
Selavy now better handles images that do not have spectral axes (an issue when dealing with images made by packages other than ASKAPsoft).
Additionally, the regridding has been sped up through a patch to the casacore library.
The pipeline scripts also have a new feature, making use of Selavy + Cmodel to better perform the continuum subtraction from spectral-line data. The old approach is still available, but is not the default.
0.10.1 (18 January 2016)¶
Much of this release relates to updates to the ingest pipeline and related tasks, in preparation for getting it running at Pawsey. These are now deployed as their own module, although it is not expected that ACES members will need to use this.
In the science processing area, an important fix was made to the code responsible for uvw rotations. A fault was identified where these were being projected into the wrong frame, which could lead to positional offsets in images made away from the initial phase centre. This fault has been fixed.
Some initial fixes to the preconditioner have been made that may improve images when Wiener filtering with a low or negative robustness parameter. Improvements are only expected when snapshot imaging is not being used. A full fix is being tested and is planned for the next release.
This release also sees the BETA pipeline scripts move into an askapsoft-derived module (although this had previously been announced).
0.9.0 (12 October 2015)¶
There are only a small number of changes to the core processing part of the software that would affect ACES work on galaxy, and these are almost all to do with the source-finder Selavy. The default values of some parameters governing output files have changed, with the preference now to minimise the number of output files. A few corrections have been made to the units of parameters in some of the output catalogues.
0.8.1 (10 September 2015)¶
This release introduces simager, the prototype spectral-line imager - this allows imaging of large spectral cubes through distributed processing, and is capable of creating much larger cubes than cimager. While this is not the final version of the spectral-line imager - the software framework that underpins the imaging code is going through a re-design prior to early science - it does demonstrate the distributed-processing approach that enables large numbers of spectral channels to be processed.
For those wanting to make use of the ACES scripts under subversion, these will be updated shortly to include use of simager.
Other changes to the askapsoft module include minor updates to the CASDA HI catalogue interface from the Selavy sourcefinder, and ADE-related updates to the ingest pipeline and associated tools (which won’t affect work on galaxy).
0.7.3 (21 August 2015)¶
This release has a few relatively small bug fixes that have been resolved in the past week:
a minor fix to cimager that solves a rare problem with the visibility metadata statistics calculations, that would result in cimager failing (this had been seen in processing the basic continuum tutorial data).
correcting the shape (BMAJ/BMIN/BPA) parameters in the Selavy-generated component parset output (that might be used as input to ccalibrator in self-calibration) - they were previously given in arcsec/degrees rather than radians (as required by ccalibrator/csimulator).
aligning the cmodel VOTable inputs with the new Selavy output formats
a fix to the units in one of the Selavy VOTable outputs
0.7.2 (9 August 2015)¶
This release is a bug-fix release aimed at fixing a problem identified in running the basic continuum imaging tutorial. There was an issue with the way the simulated data had been created, which meant that mssplit would fail on those measurement sets. This has been fixed (fixing both mssplit and msmerge), and the tutorial dataset and description have been updated.
If you use mssplit on real BETA data, you will not notice any difference, save for potentially a small performance improvement.
The only other change has been implementation of the CASDA format for absorption-line catalogues, although the implementation of actual absorption-line searching is not complete in Selavy, so this will probably not affect any of you (it has been more to provide early examples for use by the CASDA team).
0.7.0 (3 July 2015)¶
- The key features of the release are:
Mk-II compatible ingest (although not applicable for galaxy processing)
A new task mslist that provides basic information for a measurement set
Phase-only calibration
- Bug:
[ASKAPSDP-1657] - mssplit corrupts POINTING table
[ASKAPSDP-1658] - change actual_pol to expect degrees as the unit
[ASKAPSDP-1660] - Driving to an AzEl position throws an exception in the ingest pipeline.
- Feature:
[ASKAPSDP-1635] - SupportSearcher performance patch
[ASKAPSDP-1650] - Develop utility to extract and print information from a measurement set
[ASKAPSDP-1670] - Develop phase-only calibration option for CImager
- Task:
[ASKAPSDP-1663] - Modify ingest pipeline source task to conform with the ADE correlator ioc changes
0.6.3 (11 May 2015)¶
Changes for this release include bug fixes and improvements to assist the casdaupload tool, and a calibration bug that affected leakage terms. The release notes follow.
- Bug
[ASKAPSDP-1665] - Data format bug in casdaupload
- Feature
[ASKAPSDP-1659] - Update casdaupload utility to conform to new spec
- Task
[ASKAPSDP-1633] - Test ASKAPsoft leakage calibration using BETA observation 619
[ASKAPSDP-1668] - Fix width and precision in CASDA catalogues
0.6.1 (12 March 2015)¶
A bug-fix release adding a couple of elements to 0.6.0:
- Bug
[ASKAPSDP-1657] - mssplit corrupts POINTING table
[ASKAPSDP-1658] - change actual_pol to expect degrees as the unit
0.6.0 (6 March 2015)¶
Some highlight features and bugfixes are:
[ASKAPSDP-1652] - Gridding failing with concatenated MS
[ASKAPSDP-1654] - Selavy’s component parset output gets positions wrong
[ASKAPSDP-1646] - Develop CASDA upload utility
[ASKAPSDP-1649] - Add selection by field name to mssplit
[ASKAPSDP-1653] - Add parset parameter to change the weight cutoff used in linmos
- Bug
[ASKAPSDP-1628] - ASKAPsoft fails to build on Ubuntu 14.04
[ASKAPSDP-1632] - Spurious message: Observation has been aborted before first scan was started
[ASKAPSDP-1642] - Intermittant functest failure in java-logappenders
[ASKAPSDP-1651] - Program version string shows “Unknown” branch name
[ASKAPSDP-1652] - Gridding failing with concatenated MS
[ASKAPSDP-1654] - Selavy’s component parset output gets positions wrong
- Feature
[ASKAPSDP-1615] - Implement Ice monitoring interface in Ingest Pipeline
[ASKAPSDP-1637] - Flag antennas with out-of-range delays
[ASKAPSDP-1638] - Adapt VOTable output of Selavy to match recent CASDA table descriptions
[ASKAPSDP-1646] - Develop CASDA upload utility
[ASKAPSDP-1649] - Add selection by field name to mssplit
[ASKAPSDP-1653] - Add parset parameter to change the weight cutoff used in linmos
- Task
[ASKAPSDP-1624] - Document ASKAPsoft SDP platform dependencies
[ASKAPSDP-1640] - Update user documentation to use /scratch2 filesystem
[ASKAPSDP-1641] - Update Scons dependency to 2.3.4
0.5.1 (9 January 2015)¶
A bug fix release, providing an option to flag antennas with out-of-range delays in the DRx or FR hardware setting.
0.5.0 (15 December 2014)¶
The list of features & bugfixes is below:
- Bug
[ASKAPSDP-1606] - Segmentation fault when using cflag dynamic threshold
[ASKAPSDP-1608] - Calibration fails when flagged visibilities have values of NaN or Inf
[ASKAPSDP-1616] - Row index calculation in Ingest Pipelines MergedSource::addVis() is too slow
[ASKAPSDP-1622] - CP Manager should gracefully handle unavailability of the FCM
- Feature
[ASKAPSDP-1607] - Change the default for data accessor parameter “CorrelationType”
[ASKAPSDP-1610] - Account for averaging when setting noise sigma values in mssplit
[ASKAPSDP-1612] - Add support for SIGMA_SPECTRUM column to Data Accessor
[ASKAPSDP-1623] - Ingest Pipeline: Add support for pausing an observation with scanid -1
- Task
[ASKAPSDP-1603] - Improve scalability of (spectral-line) source-finding
[ASKAPSDP-1611] - Remove 3rdParty/mysql dependency
[ASKAPSDP-1613] - Document cpmanager
[ASKAPSDP-1630] - Update Apache Ant dependency to 1.9.4
0.4.1 (13 November 2014)¶
A minor update, with the following features added:
[ASKAPSDP-1610] - Account for averaging when setting noise sigma values in mssplit
[ASKAPSDP-1612] - Add support for SIGMA_SPECTRUM column to Data Accessor
0.4.0 (22 October 2014)¶
The list of features & bugfixes is below:
- Bug
[ASKAPSDP-1567] - ccalapply running slow
[ASKAPSDP-1570] - AdviseParallel fails when run in parallel with the tangent parameter unset
[ASKAPSDP-1578] - Ingest pipeline fails with exception in FrtHWAndDrx
[ASKAPSDP-1581] - CP manager occasionally fails to mkdir
[ASKAPSDP-1587] - Selavy - remove limits on component ID suffix
[ASKAPSDP-1589] - cimager fails when direction not specified
[ASKAPSDP-1594] - Thresholds in Selavy get too low near the edge caused by low weights
[ASKAPSDP-1596] - cbpcalibrator crashes in parallel mode
[ASKAPSDP-1598] - Typo in VOTable PARAM headers
- Feature
[ASKAPSDP-1390] - Develop ASKAP imaging advise functionality
[ASKAPSDP-1551] - Add time based selection to MSSplit
[ASKAPSDP-1569] - AdviseParallel should distribute statistics back to the workers
[ASKAPSDP-1573] - Add dynamic threshold flagging to cflag
[ASKAPSDP-1580] - Support AZEL coordinate system in ingest pipeline
[ASKAPSDP-1582] - Add timing metrics in ingest pipeline
[ASKAPSDP-1588] - Add ability for Selavy to write out a component parset
[ASKAPSDP-1592] - Obtain linmos feed centres from a reference image
[ASKAPSDP-1599] - Implement Ice monitoring interface in CP Manager
[ASKAPSDP-1600] - Add scan id to vispublisher
- Task
[ASKAPSDP-1583] - Improve performance of Ingest FlagTask
[ASKAPSDP-1584] - FringeRotationTask needs some performance improvements
0.3.0 (28 July 2014)¶
The version 0.3 release of the ASKAPsoft Science Data Processor has been installed as a module to Galaxy. The included features/bugfixes are listed below, and are also listed on Redmine: https://pm.atnf.csiro.au/askap/projects/cmpt/versions/197
Bug #6029: Ingest pipeline zeros flagged visibilities
Bug #6107: Fix the curvature-map option in Selavy’s Gaussian fitting
Bug #6112: Ingest pipeline flags incorrect antenna
Bug #6113: RA & Dec swapped in Ingest Pipeline Monitoring data
Bug #6121: openssl-1.0.1c fails to build on XUbuntu 14.04
Bug #6125: Superfluous loop over w in WProjectVisGridder::initConvolutionFunction
Bug #6126: gridder parameter snapshotimaging.coorddecimation is ignored
Bug #6154: Ingest pipeline should not write SBID in observation column
Bug #6179: SVN 1.7 breaks rbuilds get_svn_revision function
Bug #6183: Selavy - component catalogues for individual fit types are incomplete
Feature #6073: Support of different phase and pointing centres via scheduling blocks
Feature #6075: MSSink should populate POINTING table
Feature #6120: Ingest Pipeline: Get obs data from TOS metadata
Feature #6164: Tool to assist delay calibration
Feature #6180: Add –version cmdline parameter to askap::Application
Task #6176: SDP codebase restructure
Documentation #6106: Create an analysis tutorial
0.2.0 (4 June 2014)¶
- Bug
[ASKAPSDP-1522] - Inappropriate default level of logging in CP applications
[ASKAPSDP-1523] - cpingest: NaNs in visibilities
[ASKAPSDP-1526] - Selavy: source lists differ between serial & distributed processing
[ASKAPSDP-1529] - Problems when running Selavy on FITS file
[ASKAPSDP-1533] - ccalibrator ignores the data for other than the first beam in the antennagain mode
- Feature
[ASKAPSDP-1261] - Integrate CP ingest pipeline with TOS
[ASKAPSDP-1540] - Handle scan id of -2 in ingest pipeline
- Task
[ASKAPSDP-1525] - Update Duchamp to 1.6
[ASKAPSDP-1537] - ASKAPsoft SDP - Cleanup HPC build environment
0.1.0 (31 March 2014)¶
- Feature
[ASKAPSDP-1459] - Develop linmos utility
[ASKAPSDP-1460] - ccalibrator enhancements
- Task
[ASKAPSDP-1521] - Create CP-0.1 release