This page summarises the key changes in each tagged release of ASKAPsoft. This replicates the CHANGES file that is included in the code base.
A pipeline patch, covering the following:
- Introduction of a user-specified set of beam-wise flagging commands. These are provided through a single input file, and can define sets of flagging commands that can differ from beam to beam.
- Introduction of BANDPASS_CAL_SOLVER to set the solver parameter for cbpcalibrator.
- Allowing the re-use of existing directory tar files in the casdaupload job.
- Making those tar files more specific, so they only keep files relevant for the current SBID and set of fields.
A simple patch to the pipeline scripts, allowing control over the version of the python module used in the slurm scripts. This is now governed by the PYTHON_MODULE_VERSION parameter.
A further patch to the 1.0 release, affecting just the pipeline scripts:
- Leakage-calibrated measurement sets are now included in the CASDA upload, in preference to those with just the self-cal gains applied.
- There is now a (user-parameterisable) sleep period inserted before the second and subsequent srun calls in slurm jobs. The length is governed by SLEEP_PERIOD and defaults to 30 (seconds). This aims to help with slurm-related bus errors on the compute nodes.
- The findCubeStatistics jobs are now reported in the stats files alongside the maskBadChannels. Both these have clear descriptions as well.
- Fixed a bug where one couldn’t specify both CIMAGER_MAXUV and CIMAGER_MINUV.
- Fixed an issue where the symbolic links to reportProgress, killAll and jobList were not being made correctly.
- Fixed a bug in the wrapper script used in the rapid-mode version of the apply-leakage-correction job.
A patch to the 1.0 release, affecting both pipeline scripts and applications
Pipelines:
- A change to the leakage calibration scripts so that we do not use srun calls within loops over short-lived jobs. This will reduce the load on the Pawsey slurm controller.
- The finding of leakage parameters now does not involve a split into multiple measurement sets.
- When SUBMIT_JOBS=false, we now do not save a copy of the config file.
Applications:
- Selavy will now only write out polarisation spectra for sources with a polarised flux signal-to-noise above a particular value.
A patch to the 1.0 release, covering both pipeline scripts and one application
Pipelines:
- Better error handling in a number of scripts, so that the return values from slurm jobs accurately reflect what failed (if anything).
- Improved determination of footprint files for observation metadata. These are now fully determined from the scheduling block parameter set, so an observation that was only partially completed will have all planned fields with defined footprints.
Applications:
- The maskBadChannels app now exits if the input stats file is not able to be opened.
A further patch release to both the pipeline scripts and certain applications.
Pipelines:
- Ensuring the BPCAL can run without DO_SCIENCE_FIELD=true - the case of MSs needing to be merged was not previously running correctly.
- The FEED table in the MS created by msconcat (in the split-by-time case) is ensured to be identical to the original.
- Channel masking is enabled in the pipeline - the image cube statistics are used to determine discrepant channels, which are masked in the image cube, along with channels that are completely blank.
- Implementation of the MPI-enabled version of robust_contsub.py, to speed up the image-based continuum subtraction.
- Allow flagging through the pipeline based on UV-range (through cflag’s selection flagger).
Applications:
- Fix to the calculation of fluxes in the polarisation catalogue.
- Improved maskBadChannels to better use the statistics to identify the outlying channels.
- Fix to cflag’s selection flagger, so that it can make use of the uvrange option.
A patch release to the pipeline scripts:
- Better handling of repeated field names in the scheduling block parset.
- Better handling of repeated fields and their corresponding tiles, particularly in the linmos scripts.
- Ability to turn off the leakage calibration jobs when DO_LEAKAGE_CAL_CONT=false.
- Allow use of the parallactic angle rotation in the gridding by extracting the pol_axis angle from the SB parset.
A major release that facilitates derivation and application of on-axis instrumental leakage calibration.
This is also the final release to be made out of the monolithic ASKAPSDP subversion repository. Future releases will be made from our new bitbucket repositories. We have thus designated this version 1.0.0.
Processing Software:
- Ccalibrator now allows derivation of leakages per beam. This is done by adding a new value “antennaleakages” for the parameter “solve” of ccalibrator. It functions in a similar way as “antennagains”.
- The parallactic angle rotation can be incorporated into the gridding. This is either read from the FEED table or provided via a parset parameter.
- Improvements to ccalapply
- Better initial estimates for psf beam-fit
- Detects divergence in BFMFS and stops clean early.
- More robustness in the RM-synthesis module of Selavy to blank channels.
Pipelines:
The bandpass self-calibrated 1934-638 data are further processed (flagged, averaged to 1MHz, flagged again) to allow derivation of on-axis leakage solutions per 1MHz channels for each beam.
The derived leakages are applied on the science data after the continuum selfcal stage. For each beam, the leakage correction is done per 1MHz coarse channel.
The contPol (except for Stokes-I) and the contCube images will now be made using the on-axis leakage calibrated data.
Other minor fixes: - Parset fixes for the contPol imaging - Addition of a “-r” option to reportProgress, so that only the
running jobs are shown
- Tidying up of the job stats reporting & plotting, so that all jobs (including the new leakage calibration ones) are included.
A further pipeline patch, ensuring that the rapid mode functions correctly. The preimaging script was hard-coding the output directory to be the final field in the list of fields, and so data was ending up in the wrong directory, and failing to image.
This has been corrected, along with work to ensure post-selfcal jobs are not run when the DO_CONT_IMAGING flag is turned off.
An important patch to the pipeline scripts.
This fixes an issue with setting the lustre striping, where the value given for the striping had not yet been defined in the script. If lustre pools have been set in the current directory, these are removed if no argument is given, ie.: lfs setstripe -c “” .
The change sets the default parameters before running the initialisation script.
A further patch to both processing and pipeline code.
Processing:
- Further fixes to the PSF-fitting within the imager, to avoid divergence in the case of extended beams.
- A check is now made within the imager to determine whether the clean is diverging. If very large residual values are seen the clean is halted.
- The makeThumbnailImage script has been updated to address an issue making images with weights contours.
Pipelines:
- The rapid-mode now properly checks for the return values of the pre-imaging tasks prior to running the imaging. A failure in one of those tasks will stop the imaging from running.
- The flagSummary tasks are now turned on by default.
- The bandpass plotting now shows the smoothed table when the smoothing option is used.
- The use of nominated askapsoft modules has been made more robust, particularly in the prepareMetadata script.
- The updating of the HISTORY cards in FITS headers now has a check for the existence of any HISTORY cards before checking for repeated entries (this was causing the script to fail).
- The user can now specify the number of cores per node for the continuum source-finding job, via the CPUS_PER_CORE_SELAVY parameter.
- The continuum cube imaging task will now use the correct MS when no selfcal calibration is applied to the data.
- The following jobs are no longer needlessly submitted:
- msconcat for the spectral data (upon re-running the pipeline)
- Stokes V linmos when DO_CONTPOL_IMAGING=false
A patch to provide a fix to Selavy, correcting a problem with the integrated fluxes in the Island catalogue - these were being corrected twice for the beam area, resulting in values that were too small.
There is also a new tool in Synthesis, maskBadChannels, that can mask channels in a spectral cube based on the cube statistics (noise as a function of channel).
There are also minor improvements to the pipeline scripts:
- The HISTORY headers written to the FITS files should no longer have repeated entries.
- The modules loaded while running processASKAP.sh have been updated to avoid python errors when running on Galaxy (should the user not have the python module loaded at time of running processASKAP.sh).
- The findCubeStatistics script has a -n option to allow plotting of the stats without recomputing them.
A patch release largely built around fixing issues with data presented to CASDA for release.
- Ensuring the validation metrics file is correctly named when casdaupload is re-run.
- Getting the correct RA/Dec positions for each beam’s MS for the casda metadata (rather than the pointing centre of the observation).
- Allowing the correct annotation of Jira tickets when uploading to CASDA.
- Ensuring the column widths within the VOTable catalogues produced by Selavy are consistent with CASDA expectations.
There are also several pipeline fixes:
- A new “rapid” mode of the pipeline for wide-area continuum observations of many fields within the one scheduling block. This combines the pre-imaging tasks into the imaging slurm job, to reduce the overall load on the slurm controller.
- New bandpass plotting is provided, along with flag summaries for each MS.
- The reference frequency was not being set correctly for the CleanModel option of continuum subtraction.
- We avoid writing out the schedblock variables to the metadata files, to avoid formatting issues that were causing errors.
- The Stokes value had been set to lower-case in some labels on plots, which led to confusion. It is now back to upper case.
And for the askapsoft processing software:
- The numpy, matplotlib, scipy and astropy 3rdParty modules have been removed - these need to be provided by the external system. This was to avoid segmentation faults occuring in the makeThumbnails jobs for the larger continuum images. This has resulted in changes to the modules loaded by the pipeline scripts as well (so that those python packages are provided by system modules).
- The PSF fitting in the imager has been tweaked to avoid rare cases of divergence.
A further small patch release, addressing a few critical issues:
- The determination of beam locations in the pipeline is now done in a more robust way, that is insensitive to the specifics of the parset formatting (making use of the LOFAR getparsetvalue command).
- The handling of the continuum validation directory in the CASDA upload is fixed, so that the filename is recognised correctly. It has also been shortened, to avoid problems with Windows filesystems.
- The code to invert the sign of the Stokes V images in the pipeline has been removed. The images are now presented as-is, and work will be done to ensure the correct values are presented by the imager.
- Selavy has had improvements made to its memory handling, to avoid out-of-memory errors due to memory leaks.
A small release providing a patch to the processing software and a new option to the pipeline scripts.
Processing software:
0.24.2 had an issue with the application of self-cal gains. If integrations were substantially flagged, they would have a null row in the calibration table, and this could result in errors during the application, whether this was done from within the imaging or by ccalapply.
The new code is much more robust against flagged data, and completely flagged integrations will be left out of the table. This allows the application of the solutions to proceed without error.
Pipelines:
- There is a new option to produce MFS images in different polarisations, following the application of the self-cal gains to the averaged dataset. The default set of polarisations is “I,V” (although I is done in the self-cal procedure and not repeated), and there are new parameters to govern the deconvolution for these images - consult the documentation for coverage and default values.
- This also includes a script to invert the sign of the pixel values in the Stokes V images, to make the values accord with the definition of Stokes V from IAU/IEEE. This can be turned off by setting INVERT_SIGN_STOKES_V_IMAGE=false.
- For consistency, there is a similar one for the continuum cubes: INVERT_SIGN_STOKES_V_CONTCUBE. These are kept separate to allow them to be switched on independently, in case (for instance) the memory requirements of the cubes mean you want to do it separately.
- Additionally, the pipeline scripts now allow the averaged dataset to be flagged when the full-spectral-resolution dataset is not.
- There are checks made at the start of the spectral splitting, to ensure we don’t do it when the concatenated version of the MS is present (similar to the change made in 0.24.2).
- The dependencies for the continuum source-finding job are fixed so that the single-field cont-cube mosaicking job is included.
- The mosaicking now appropriately handles the continuum images when nterms=1.
This is largely a patch release, with a few minor additional features.
Pipelines:
- Changes to the distribution of ranks within the imaging job, to smooth out the memory usage a little.
- Use the bptool module to identify antennas to flag based on the bandpass calibration table. Triggered by the new parameter DO_PREFLAG_SCIENCE. The bptool module is now assumed to be in the same place as other ASKAP modules.
- Checkfiles will now indicate the time-window where appropriate, to better distinguish work that has been done on datasets.
- Checks for spectral-line parameters are now only done when DO_SPECTRAL_PROCESSING is set to true as well as DO_SPECTRAL_IMAGING.
- Time-split data will now no longer be regenerated if the merged dataset is present.
- The second mosaicking job is now only done if there is more than one field. There are also better error checking done for the copying of the mosaic in the case of a single field.
- The altrestored mosaicking jobs are not launched when this mode is not used.
- When restarting a partially failed imaging pipeline, the selfcal gains table is also searched for to see if a given beam needs re-doing (previously it was only looking for the image, which meant a partially-completed selfcal job could look like it was complete).
- The gatherStats slurm job now has a longer time request, to prevent it timing out for SBs with large numbers of fields.
- Additional checks are made when restarting selfcal, so that the gains table needs to be present (as well as the image) for the selfcal to be skipped.
- When measuring the spectral indices from the continuum cubes, the pipeline scripts will now use the reference-beam’s beamLog to correctly scale the spectra prior to fitting.
- When archiving measurement sets, the metadata directory is now copied into the MS prior to it being tarred, where it is called ASKAP_METADATA.
- New or changed pipeline default parameters:
- DO_PREFLAG_SCIENCE (true)
- GRIDDER_SHARECF and GRIDDER_SPECTRAL_SHARECF, to allow performance enhancements to be realised in the pipeline.
- TILENCHAN_AV, TILENCHAN_SL, TILE_NCHAN_SCIENCE & TILE_NCHAN_1934 all now default to 1.
- SELAVY_NSUBY is now 6
- ARCHIVE_SPECTRAL_MS is now false
- MULTI_JOB_SELFCAL is now false
- SELAVY_SPECTRAL_INDEX_THRESHOLD and SELAVY_SPECTRAL_INDEX_THRESHOLD_SNR, for the new Selavy threshold parameters (see below).
- PURGE_FULL_MS_AFTER_COPY to remove the full spectral MS after copying a nominated channel range.
Processing:
- Selavy:
- Images created by Selavy, such as noise maps, will now have the same image mask as the input image.
- There are new parameters “Selavy.spectralTerms.threshold” and “Selavy.spectralTerms.thresholdSNR” to provide a minimum flux or signal-to-noise ratio for the spectral index/curvature to be reported in the component catalogue (components fainter than this threshold will not have a spectral index/curvature value).
- A fix to the computation of spectral indices from extracted spectra - this was failing due to uninitialised memory.
- MFS imaging:
- The higher-order Taylor-term residual images are now appropriately scaled to take into account the coupling with the Taylor-0 residual. The behaviour is now consistent with CASA. For reliable spectral indices, it is recommended to measure from cleaned sources.
- The linear-mosaicking will now correct higher-order Taylor-term images for primary beam variation with frequency, through the use of the removebeam parameter.
- General imaging:
- There are numerous performance enhancements to imaging and linear-mosaicking, resulting in faster processing times.
- The WProject gridder will now discard data above the wmax limit, rather than throwing an exception.
- The WProject gridder now has the option of using a static cache of convolution functions, rather than re-generating for every instance of a gridder.
- CASDA upload:
- The number of time steps for each scan in the measurement set are now reported to CASDA via the observation.xml file.
- Each scan also has a field-of-view reported. It is currently the FWHM of the beam (1.09 lambda/D), evaluated at the centre frequency.
A patch release, focusing only on fixes for pipeline issues:
- Default values for GRIDDER_WMAX_NO_SNAPSHOT and GRIDDER_SPECTRAL_WMAX_NO_SNAPSHOT have been changed to 35000
- For the CASDA upload script, certain evaluation files have had the pipeline timestamp added to their filename
- A few bugs were fixed:
- The list of MSs given to the upload script were erroneously the final time-window MS (when that mode is used).
- The continuum validation script was mistakenly making all files group-writeable (rather than just the copied validation directory).
- The logic for checking the FREQ_FRAME parameter was incorrect and should now do the right thing.
The user documentation has also been updated, with a missing table added back in.
A major release, with an improved framework for the pipeline scripts to handle large datasets efficiently, along with several performance improvements to the processing software.
Processing software:
- Spectral imaging:
- There is a change to the behaviour of the Channels keyword - now the syntax is [number,startchannel], where the channels are those in the input measurement set.
- Introduction of more frequency frames. You can now specify one of three reference frames topocentric (default) barycentric and the local kinematic standard of rest. The new keyword is: Cimager.freqframe=topo | bary | lsrk
- You can now specify the output channels via their frequencies if you choose. This is a very new feature but simple to test. Note that, as before, there is no interpolation in this mode. It is simply a “nearest” neighbour scheme. The syntax is: Cimager.Frequencies=[number,start,width], with the start and width parameters are in Hz.
- The spectral line imager should be more robust to missing channels either via flagging or frequency conversion.
- The restoring beam was often not being written to the header of spectral cubes, particularly when more than one writer was used.
- The beamlog (the list of restoring beams per channel) for cubes was not being written correctly. It now will always be written, and will have every channel present, even those with no good data (their beams will be zero-sized).
- The bandpass calibration had reverted to the issue solved in 0.22.0, where a failed fit to a single channel/beam would crash the job. This is now robust against such failures again.
- The mssplit tool has had a memory overflow fixed, so that bucketsizes larger than 4GB can be used.
- The linmos-mpi task would fail with an error when the first image being processed was not the largest.
Pipelines:
- The pre-imaging tasks can now run more efficiently by dividing the dataset into time segments and processing each segment separately. The length of the time segment is configurable. The time-splitting applies to the application of bandpass solutions, flagging, averaging, and, for the spectral data, continuum-subtraction.
- Additional continuum-imaging parameters are made selcal-loop-dependent: CLEAN_ALGORITHM, CLEAN_MINORCYCLE_NITER, CLEAN_GAIN, CLEAN_PSFWIDTH, CLEAN_SCALES, and CLEAN_THRESHOLD_MINORCYCLE. The last two parameters can have vectors for individual loops, and so this necessitates a new format, whereindividual loops are separated by ‘ ; ‘.
- Overall control over the spectral processing is now provided by DO_SPECTRAL_PROCESSING. This defaults to false, meaning only the continuum processing will be done. Turning this to true will result in application of the selfcal gains, continuum subtraction, spectral imaging and image-based continuum subtraction being done - each of these are turned on by default.
- Elevation-based flagging for the science observation is able to be configured through the pipeline parameters.
- There are parameters to specify a given range of times to be used from the bandpass calibration observation - useful if you wish to use part of a long observation.
- The arguments to the bandpass smoothing tools can be provided as a text string instead of the specific pipeline parameters - this will allow continued development of these tools without needing to keep the pipeline up-to-date with possible parameter inputs.
- The new spectral imaging features (see above) are exposed through pipeline parameters FREQ_FRAME_SL and OUTPUT_CHANNELS_SL. The DO_BARY parameter has been deprecated.
- The following bugs have been fixed
- The continuum subtraction selavy jobs were using the wrong nsubx/nsuby parameters.
- The pipeline scripts now check for the correct loading of the askapsoft & askappipeline modules, and exit if these are not loaded correctly.
- Wildcards used in identifying polarisation data products are now correctly applied.
- There are new parameters used in the pipeline (with their defaults):
- DO_SPLIT_TIMEWISE=true
- DO_SPLIT_TIMEWISE_SERIAL=true
- SPLIT_INTERVAL_MINUTES=60
- MULTI_JOB_SELFCAL=true
- FAT_NODE_CONT_IMG=true
- TILENCHAN_AV=18
- SPLIT_TIME_START_1934=”“
- SPLIT_TIME_END_1934=”“
- BANDPASS_SMOOTH_ARG_STRING=”“
- BANDPASS_SMOOTH_F54=”“
- ELEVATION_FLAG_SCIENCE_LOW=”“
- ELEVATION_FLAG_SCIENCE_HIGH=”“
- OUTPUT_CHANNELS_SL=”“
- FREQ_FRAME_SL=bary
- DO_SPECTRAL_PROCESSING=true
- There are new default values for some pipeline parameters:
- JOB_TIME_DEFAULT=”24:00:00”
- CPUS_PER_CORE_CONT_IMAGING=6
- FAT_NODE_CONT_IMG=true
- NUM_PIXELS_CONT=6144
- CELLSIZE_CONT=2
- NCHAN_PER_CORE=12
- NCHAN_PER_CORE_CONTCUBE=3
- NCHAN_PER_CORE_SL=64
- NUM_SPECTRAL_WRITERS=”“
- NUM_SPECTRAL_WRITERS_CONTCUBE=”“
- GRIDDER_WMAX_NO_SNAPSHOT=30000
- GRIDDER_NWPLANES_NO_SNAPSHOT=257
- CLEAN_MINORCYCLE_NITER=”[400,800]”
- CLEAN_GAIN=0.2
- CLEAN_SCALES=”[0,3,10]”
- CLEAN_THRESHOLD_MINORCYCLE=”[30%, 0.5mJy, 0.03mJy]”
- CLEAN_NUM_MAJORCYCLES=”[5,10]”
- CLEAN_THRESHOLD_MAJORCYCLE=”0.035mJy”
- SELFCAL_NUM_LOOPS=1
- SELFCAL_INTERVAL=[200,200]
- NUM_PIXELS_CONTCUBE=4096
- CPUS_PER_CORE_CONTCUBE_IMAGING=8
- CLEAN_CONTCUBE_MINORCYCLE_NITER=600
- CLEAN_CONTCUBE_GAIN=0.2
- CLEAN_CONTCUBE_THRESHOLD_MINORCYCLE=”[40%, 0.5mJy, 0.05mJy]”
- CLEAN_CONTCUBE_THRESHOLD_MAJORCYCLE=0.06mJy
- CLEAN_CONTCUBE_NUM_MAJORCYCLES=3
- TILENCHAN_SL=18
- DO_APPLY_CAL_SL=true
- DO_CONT_SUB_SL=true
- DO_SPECTRAL_IMAGING=true
- DO_SPECTRAL_IMSUB=true
- NUM_PIXELS_SPECTRAL=1024
- CELLSIZE_SPECTRAL=8
- SPECTRAL_IMAGE_MAXUV=2000
- PRECONDITIONER_SPECTRAL_GAUSS_TAPER=”[20arcsec, 20arcsec, 0deg]”
- GRIDDER_SPECTRAL_SNAPSHOT_IMAGING=false
- GRIDDER_SPECTRAL_WMAX_NO_SNAPSHOT=30000
- GRIDDER_SPECTRAL_NWPLANES=257
- CLEAN_SPECTRAL_MINORCYCLE_NITER=800
- CLEAN_SPECTRAL_GAIN=0.2
- CLEAN_SPECTRAL_THRESHOLD_MINORCYCLE=”[45%, 3.5mJy, 0.5mJy]”
- CLEAN_SPECTRAL_THRESHOLD_MAJORCYCLE=0.5mJy
- CLEAN_SPECTRAL_NUM_MAJORCYCLES=3
A further patch release, with a number of small pipeline fixes, along with several fixes to the processing software.
Processing:
- The imager would produce slightly different residual and restored images when different values of nchanpercore were used. This was due to the final gridding cycle not being synchronised correctly. This has been fixed and the images are now indepenent of nchanpercore.
- The tree reduction used by imager has been improved to have a smaller memory footprint across the cores.
- The selavy component fitting is improved in the way negative components are handled. Unless negative components are explicitly accepted, if a fit results in one or more components being negative then that fit will be rejected.
- The primary beam used by linmos now has a FWHM scaling by 1.09 lambda/D, which should be more accurate.
- The FITSImage interface (in Code/Base/accessors) will now report a human-readable error message (rather than a number code) when an error occurs.
Pipelines:
- CASDA uploads again include catalogues (which were left out due to fixes in 0.23.1).
- There are new parameters CIMAGER_MAXUV and CCALIBRATOR_MAXUV that allow the imposition of an upper limit to the uv values in the continuum imaging/self-calibration.
- Parsets for the imager were erroneously getting a “Cimager.Channels” selection that included the %w wildcard. This will no longer happen (unless cimager is used).
- The default python module is now always loaded at the start of slurm scripts, to avoid python conflicts due to a user’s particular environment.
- There are stronger checks on the number of cores allocated to spectral-line imaging, ensuring that the number of channels must be an exact multiple of the nchanpercore.
- The scaling on the beam-wise noise plots has been fixed, so that the scaled MADFM should be closer to the standard deviation in the absence of signal.
- Cube stats are now also generated for continuum-cube residual images.
- Several scripts have been tidied up with the aim of avoiding spurious errors (validationScience, for instance).
- The ASKAPsoft version was being left off FITS headers. This now reflects the version string from the askapsoft module.
A patch release, fixing an issue with imager and a couple of minor pipeline issues:
- The imager in spectral-imaging mode was not respecting the clean thresholds correctly. This could lead to over-cleaning, and the insertion of spurious clean components at noise peaks (particularly in continuum-subtracted spectral data).
- A change has been made to the module setup, avoiding “module swap” in favour of “module unload / module load” - this addresses an occasional issue seen where the module environment can get corrupted by the swap command.
- A fix has been made to the flagging parsets, solving a problem where the autocorrelation flagging and the time-range flagging were assigned to the same rule. If both were used, the time range was only flagged in the autocorrelations. They now appear as separate rules and so will be independent.
A patch release, addressing a few bugs in both processing software and pipeline scripts
Pipelines:
- Changes have been made to the scripts to make them robust in handling field names that contain spaces. This has also made them more robust to being run in a directory with a path that contains spaces.
- An update has been made at Pawsey to the module used for the continuum validation task, and consequently a minor change has been made to the continuum sourcefinding script.
Processing:
- Enhancements have been made to the continuum-subtraction task ccontsubtract to speed it up - initial tests indicate a speed-up of 6-8x depending on platform.
- The Selavy fitting algorithm now defaults to including a test on the size of the fitted Gaussians. This will prevent spuriously large fits from making it through to the catalogue, which has had detrimental effects in the calibration & continuum-subtraction.
- A fix was made to the imager, solving a problem where the spectral-imaging option merged the first channel of its allocation without checking the frequency.
Additionally, the user documentation has updated instructions about how best to set the modules on galaxy so that everything runs smoothly (see Central Processor Platform).
A major release, addressing a number of issues with the processing software and the pipeline scripts.
Pipelines:
- When multiple raw MSs are provided for a given beam (split up by frequency range), the pipeline is capable of recognising this, merging (after any necessary splitting), and handling all required metadata appropriately. The functionality should be the same no matter the structure of the raw data.
- The selfcal job allocation (for the sbatch call) has been altered to request a number of nodes, rather than cores + cores-per-node. This should provide more predictable allocations.
- The weights cutoff parameter given to Selavy is now fully consistent with the linmos cutoff.
- Fixed a bug that meant the raw data was overwritten when calibration was applied, even when KEEP_RAW_AV_MS=true.
- The TELESCOP keyword is now added to the FITS headers.
- A bug was fixed that was preventing the full-resolution MSs being included in the CASDA upload.
- New parameters SPECTRAL_IMAGE_MAXUV and SPECTRAL_IMAGE_MINUV that allow control over the UV distances passed to the spectral imager.
- Various improvements to the gatherStats job, so that it will still run after the killAll script has been called, and that looks for the pipeline-errors directory before trying to use it.
- Making the cubeStats script more robust against failures of a single process (so that it doesn’t hang but instead carries on as best it can).
Processing:
- Imaging:
- Fix a coordinate shift that was seen in spectral imaging, due to a different direction being provided by the advise functionality.
- Calibration:
- Efficiency improvements to ccalapply to help speed it up
- Utilities:
- Adjustment of the maximum cache size in mssplit to avoid out-of-memory issues
- Trimming down of the pointing table in MSs produced by msconcat, so that very large tables do not result.
- Selavy:
- The restoring beam is now written into the component maps.
- A significant change to the handling of the initial estimates for the Gaussian fits, making it more robust and avoiding downstream WCS errors that were hampering the analysis.
- Minor catalogue fixes for component & HI catalogues
- Segfaults in selfcal (3145)
This release sees a number of changes & improvements to both the processing software and the pipeline scripts.
Pipelines:
- There are new diagnostic plots produced, particularly for the spectral & continuum cubes. There is a python script run immediately following the imaging to calculate a range of statistics on a channel-by-channel basis, and this data is plotted on a per-image basis, as well as an overview plot showing the statistics for each beam at once.
- The ability to specify the number of cores used for the continuum imaging has been improved, to make it more flexible and work better with slurm.
- The behaviour of source-finding in the selfcal has changed. We now fit the full set of Gaussian parameters, and require contiguous pixels for the islands.
- Several bugs were fixed:
- Some FITS files were not having their header keywords updated correctly. This has now been fixed and streamlined.
- The CASDA upload script was erroneously including multiple versions of the taylor.1 images, due to a bug introduced in 0.21.0. It was also dropping a few .fits suffixes in certain situations.
- The cmodel-based continuum subtraction script had a bug with an undefined local variable that occured with particular parameter settings.s
- The clean-model-based self-calibration script was getting the model image name wrong for Taylor-term images.
- There are a number of changes to the default parameters:
- The DO_MAKE_THUMBNAILS option is now true by default.
- There is a new DO_VALIDATION_SCIENCE (true by default) to run the cube validation.
- The snapshot imaging has been turned off by default, as this has proved to be more reliable in terms of image quality. Along with this, the number of w-planes has a default value that changes with the snapshot option: snapshot imaging has 99, non-snapshot imaging has 599.
- The number of channels in the MS tile is exposed as a parameter for the bandpass & science datasets, taking the same default value as previously (54).
- The “solver.Clean.solutiontype” parameter is exposed as a pipeline parameter for all imaging jobs.
- The SB_JIRA_ISSUE is replaced by JIRA_ANNOTATION_PROJECT for schedblock annotations, although this functionality is currently only available to the askapops user.
Applications:
- Selavy: - The Selavy HI catalogue now has better defined default values, and the NaN values that were appearing have been fixed (through use of masked arrays when fitting to the moment-0 map). - Selavy was previously occasionally dropping components from the catalogue through the application of acceptance criteria. This is now optional, and off by default. - Selavy was failing to calculate spectral indices in certain cases - this is now fixed. - The deconvolved sizes in the Selavy components catalogue are now calculated with better floating-point arithmetic, to avoid rare cases of NaNs. - The column widths for the VOTable catalogues are more tightly controlled, in line with the CASDA software.
- Imaging: - Primary beam factory (used by linmos) able to handle elliptical Gaussian beams. Not fully implemented within linmos yet. - A new gridder AltWProjectVisGridder that allows the dumping of uvgrids to casa images. - Caching in the spectral-line imaging of imager is now done by channel. - The spectral-line imager will now correctly write the beam log when using multiple writers. - An issue with the beam fitting failing for very elongated beams has been remedied.
- Casda upload: - The casdaupload utility was leaving out the path to measurement sets when making the XML interface file, even when the absolute path option was being requested. This is now fixed and all artifacts will have their absolute path used in this mode. - Similarly, checksums for the thumbnail images were not being created by casdaupload. This has been remedied.
- Other: - The FITS accessor interface now better handles missing header keywords - if they are not present it now logs a warning but doesn’t exit. - Ccalapply has improved handling of flags, allowing write access. - Improvements to the efficiency of mssplit and msmerge. - The user documentation has a detailed tutorial on MS(MFS) imaging.
A further patch release for the pipeline, fixing a few issues that have been seen on the Galaxy platform.
- The previous fix for the OUTPUT directory is now included correctly in the release.
- The fix for imagetype parameter in Selavy parsets generated by the pipeline has been extended to the continuum-subtraction jobs.
- The bandpass validation log is copied to the diagnostics directory, as it includes useful information about the state of the dataset.
- Errors involving ‘find’ (from the ‘rejuvenate’ function) are no longer reported in the slurm output when the file in question does not exist.
- When aoflagger is used for the flagging, the slurm script ensures that the correct programming environment (PrgEnv module) is loaded prior to loading the aoflagger module.
- The continuum cube imaging can now use more than one channel per core. This is accessed via the new parameter NCHAN_PER_CORE_CONTCUBE.
- Added casa, aoflagger and bptool version reporting to the image headers and the copy of the config file, to enhance the reproducibility of the pipeline processing.
A patch release fixing minor issues with the 0.21.0 version of the processing pipeline scripts. Only the scripts and the documentation are changed.
Fixes to the pipeline:
- The bandpass validation script will now find the correct files when an OUTPUT directory is used.
- Similarly, the statsPlotter script is now more robust against the use of the OUTPUT option.
- Parallel processing enabled for the larger ccalapply jobs.
- The Channels selection parameter for continuum imaging can be left out when NUM_CPUS_CONTIMG_SCI is provided, with a new parameter CHANNEL_SELECTION_CONTIMG_SCI available to specify a selection.
- The snapshot imaging option is turned back on by default for all imaging with the pipeline, following further testing & feedback from commissioning & operations teams.
- There is better specification of the imagetype parameter in the Selavy parsets - there were issues when imagetype=casa was used.
A large release containing a number of updates to the pipeline scripts and to various aspects of the processing tools.
Pipeline updates:
- Ability to use AOflagger instead of cflag.
- Ability to use the continuum cubes to measure spectral indices of continuum components (using Selavy).
- Fixing a bug where the CleanModel option of continuum-subtraction was using the wrong image name.
- Allow self-calibration to use the clean model image as the model for calibration (in the manner of continuum-subtraction).
- Improvement of the continuum subtraction Selavy parameterisations, to better model the continuum components. The Selavy parsets are now consistent with those used for the continuum cataloguing.
- Collation of pipeline jobs that failed, for analysis by ASKAP Operations, to help identify pipeline or platform issues.
- Use of an alternative bandpass smoothing task - smooth_bandpass.py (instead of plot_caltable.py).
- Use of an additional bandpass validation script to produce summary diagnostic plots for the bandpass solutions.
- Fixed a bug where the bandpass table name was not set correctly when the the DO_FIND_BANDPASS switch was turned off.
- Addition of the spectral measurement sets, the continuum-subtraction models/catalogues, and the spectral cube beam logs to the list of artefacts to be sent to CASDA upon pipeline completion.
- Added more robustness to the pipeline scripts to allow them to run on other systems, allowing the specification of the module directory and flexibility for running on non-Lustre filesystems.
- Changes to some default parameters. Here are the parameters that have changed, with their new values (note that the WMAX and MAXSUPPORT gridding parameters now also adapt their default values according to whether snapshot imaging is turned on or off):
# Image type
IMAGETYPE_CONT=fits
IMAGETYPE_CONTCUBE=fits
IMAGETYPE_SPECTRAL=fits
# Bandpass calibration
DO_APPLY_BANDPASS_1934=true
BANDPASS_SMOOTH_FIT=1
BANDPASS_SMOOTH_THRESHOLD=1.0
# Continuum imaging
NUM_TAYLOR_TERMS=2
CLEAN_MINORCYCLE_NITER=2000
CLEAN_PSFWIDTH=256
CLEAN_THRESHOLD_MINORCYCLE="[20%, 1.8mJy, 0.03mJy]"
CLEAN_NUM_MAJORCYCLES="[5,10,10]"
CLEAN_THRESHOLD_MAJORCYCLE="0.03mJy"
SELFCAL_INTERVAL="[1800,1800,200]"
GRIDDER_SNAPSHOT_IMAGING=false
GRIDDER_WMAX_SNAPSHOT=2600
GRIDDER_MAXSUPPORT_SNAPSHOT=512
GRIDDER_WMAX_NO_SNAPSHOT=26000
GRIDDER_MAXSUPPORT_NO_SNAPSHOT=1024
# Continuum cube imaging
CLEAN_CONTCUBE_ALGORITHM=BasisfunctionMFS
CLEAN_CONTCUBE_PSFWIDTH=256
CLEAN_CONTCUBE_MINORCYCLE_NITER=2000
CLEAN_CONTCUBE_THRESHOLD_MINORCYCLE="[40%, 12.6mJy, 0.5mJy]"
CLEAN_CONTCUBE_THRESHOLD_MAJORCYCLE=0.5mJy
# Spectral imaging
NCHAN_PER_CORE_SL=9
NUM_SPECTRAL_WRITERS=16
ALT_IMAGER_SINGLE_FILE=true
PRECONDITIONER_LIST_SPECTRAL="[Wiener,GaussianTaper]"
PRECONDITIONER_SPECTRAL_GAUSS_TAPER="[30arcsec, 30arcsec, 0deg]"
PRECONDITIONER_SPECTRAL_WIENER_ROBUSTNESS=0.5
CLEAN_SPECTRAL_ALGORITHM=BasisfunctionMFS
CLEAN_SPECTRAL_PSFWIDTH=256
CLEAN_SPECTRAL_SCALES="[0,3,10,30]"
CLEAN_SPECTRAL_THRESHOLD_MINORCYCLE="[50%, 30mJy, 3.5mJy]"
CLEAN_SPECTRAL_MINORCYCLE_NITER=2000
GRIDDER_SPECTRAL_SNAPSHOT_IMAGING=false
GRIDDER_SPECTRAL_WMAX_SNAPSHOT=2600
GRIDDER_SPECTRAL_MAXSUPPORT_SNAPSHOT=512
GRIDDER_SPECTRAL_WMAX_NO_SNAPSHOT=26000
GRIDDER_SPECTRAL_MAXSUPPORT_NO_SNAPSHOT=1024
# Spectral source-finding
SELAVY_SPEC_OPTIMISE_MASK=false
SELAVY_SPEC_VARIABLE_THRESHOLD=true
SELAVY_SPEC_SNR_CUT=8
Processing tasks:
- An MPI barrier has been added to the spectral imager to prevent race conditions in the writing.
- Better handling of cases in the bandpass calibration that were previously (prior to 0.20.3) causing it to fail with SVD conversion errors.
- Selavy will now report the best component fit (assuming it converges in the fitting), regardless of the chi-squared. If poor, a new flag will be set.
- If the fit fails to converge, Selavy can reduce the number of Gaussians being fit to try to get a good fit.
- A bug in Selavy was fixed to allow the curvature-map method of identifying components to better take into account the weights image associated with the image being searched.
- A further bug in Selavy (the extraction code) was fixed to allow its use on images without spectral or Stokes axes.
- The SNR image produced by Selavy now has a blank string for the pixel units.
- The implementation of the variable threshold calculations in Selavy have been streamlined, to improve the memory usage particularly for large spectral cubes. There is also control over the imagetype for the images written as part of this algorithm.
- The memory handling within linmos-mpi has been improved to reduce its footprint, making it better able to mosaic large spectral cubes.
Manager & ingest:
- Improvements to the CP manager.
- UVW calculations fixed in the course of testing new fringe rotator modes.
ASKAPsoft environment:
- Incorporation of python-casacore in the cpapps build (used to create the askapsoft module at Pawsey).
Documentation:
- Added a chapter to the user documentation on how to combine multiple epochs for spectral line data.
- Added a chapter to the user documentation explaining the best way to do MS/MFS deconvolution in askapsoft
- Added a page to the user documentation listing the release notes for each release.
A patch release fixing a couple of calibrator issues:
- The 0.20 updates to the calibrator to allow interaction with the calibration data service had prevented ccalibrator from writing more than one row to the output calibration table. This fix ensures the table that gets written has all the information when solving for time-dependent gains.
- The bandpass calibrator would very occasionally fail with an error along the lines of “ERROR: SVD decomposition failed to converge”. This will now only trigger a WARN in the log file, but will not abort the program. Work is still being done to properly flag channels that suffer this.
And a couple of pipeline issues have been fixed:
- The beams that are processed by the pipeline are now limited by the number of beams in the bandpass calibrator scheduling block (in the same way that the science SB is used to limit the number of beams).
- Minor issues with copying the continuum validation results have been resolved.
Additionally, casacore (in 3rdParty) is now built with the python bindings, so that libcasa_python will be available.
A patch release that fixes a few bugs in the build to do with missing directories:
- Modified several build configurations so that missing directories do not make the build fail. Missing directories can be present as a result of a bug in our SVN to BitBucket sync which ignores empty directories (even if there is a .gitxxxx file in it). Subsequently, cloning the git repo causes these directories to be missing which can cause a failed build for some packages. In these cases, the build script has been changed to create the missing directories if they are missing.
- Note there are no application code or documentation changes for this release.
A patch release that fixes a few bugs in the pipeline:
- Adds better robustness to the USE_CLI=false option, for use when the databases at MRO are unavailable.
- A scripting error in the self-calibration script (for the Cmodel option).
- Fixes to the defineArtifacts script, to better handle FITS extensions.
- When the image-based continuum-subtraction option is run, the spectral source-finding job will now search the continuum-subtracted cube. The spectral source-finding will also handle sub-bands correctly.
- There have also been fixes to ensure the continuum-subtracted cubes are created in appropriate FITS format and mosaicked correctly.
- Copying of continuum validation files to the archive directory has been updated to reflect an improved directory structure.
It also makes a few minor changes to the processing software:
- The Wiener preconditioner will now report in the log the amount by which the point-source sensitivity is expected to increase over the theoretical naturally-weighted level.
- The casdaupload utility can now produce an XML file with absolute paths to data products, leaving them in place - rather than copying all data products to the upload directory. This is compatible with behaviour introduced in CASDA-1.10.
- Ccalapply has a new parameter than can restrict the sizes of chunks presented in single iterations, using new options for the TableDataSource classes.
- The component catalogue produced by Selavy had a minor error in the calculation of the error on the integrated flux (where the minor axis should have been used, the major axis was used instead).
- Fixed issues with cmodel functional tests, relating to using the correct catalogue columns.
- Fixed a failing scimath unit test.
- The ingest pipeline now can apply phase gradients in parallel.
This release sees the first version of the Calibration Data Service (CDS) and Sky Model Service (SMS) in deployable form. These components are intended to run independently of the ASKAPsoft pipelines. At first, they will require some configuration and data initialisation. Testing and feedback will then drive further development.
The CDS provides an interface to a database containing calibration parameters. The SMS allows access to the Global Sky Model data, primarily for the purpose of constructing local sky models.
Other changes in this release include:
A patch release that allows the pipelines to run correctly on native slurm, using srun to launch applications rather than aprun. This is timed to be available for the upgrade of the galaxy supercomputer to CLE6.
The release also has a slightly improved build procedure that better handles python dependencies, and updated documentation regarding the ASKAP processing platform at Pawsey.
No functional change is expected for the processing software itself.
A patch release for both the processing and pipeline areas. This fixes a few bugs and introduces a few minor features to enhance the processing.
Processing software:
- Cflag:
- There was a bug where the StokesV flagger would crash with a segmentation fault on occasions where it was presented with a spectrum or time-series that was entirely flagged. It is now more robust against such datasets.
- Imager:
- The imager is now more robust against small changes in the frequency labels of channels, with an optional tolerance parameter available.
- Selavy:
- A few bugs were fixed that were preventing Selavy working for spectral-line cubes, where it was trying to read in the entire cube on all processing cores (leading to an out-of-memory error).
- Moment-0 maps now have a valid mask applied to them.
- Selavy can now measure the spectral index & curvature from a continuum cube, instead of fitting to Taylor-term images.
- Duchamp version 1.6.2 has been included in the askapsoft codebase.
- The deconvolved position angle of components is now forced to lie between 0 & 2pi, and its error is limited to be no more than 2pi.
- Linmos:
- Fixed a bug that meant (in some cases) only a single input image was included in the mosaic. Happened when the input images had masks attached to them (for instance, combination of mosaics).
- New option of “weighttype=Combined” for linmos-mpi, that uses both the weight images and the primary beam model to create the output weights.
A patch release that adds a few new bits of functionality:
The Selavy code has been updated to add to the catalogue specifications for the continuum island & component catalogues:
- The component catalogue now has error columns for the deconvolved sizes, as well as for the alpha & beta values.
- Additionally, the 3rd flag column now indicates where the alpha & beta values are measured from - true indicates they come from Taylor-term images.
- The island catalogue now has:
- An error column for the integrated flux density
- Columns describing the background level, both the mean background across the island, and the average background noise level.
- Statistics for the residual after subtracting the island’s fitted Gaussian components - columns for the max, mean, min, standard deviation and rms.
- Columns indicating the solid angle of the island, and of the image restoring beam.
- Occasional errors in converting the major/minor axis sizes to the correct units have also been fixed.
A patch release covering the pipeline scripts and the processing software. The following bugs are fixed:
- The pipeline configuration parameter FOOTPRINT_PA_REFERENCE will now over-ride the value of footprint.rotation in the scheduling block parset. Additionally, the scheduling block summary metadata files (created in the pipeline working directory) are now not regenerated if they already exist.
- The metadata collection in the pipeline now does not fail if a FIELD in the measurement set has ‘RA’ in its name.
- There was a memory leak in Selavy, causing an error to be thrown when dealing with fitted components, specifically when the numGaussFromGuess flag was set to false and a fit failed. The code now falls back to whatever the initial estimate for components was, even if that has fewer than the maximum number indicated by maxNumGauss.
- There was a half-pixel offset enforced in the location of the fitted Gaussian when fitting to the restoring beam when imaging. This was resulting in a slightly incorrect restoring beam.
- If there are multiple MSs in the SB directory, one can be processed by giving MS_INPUT_SCIENCE its full path, setting the SB_SCIENCE parameter appropriately, and putting DIR_SB=””.
A patch release just covering the pipeline scripts. The following bugs are fixed:
- The number of writers used in the spectral-line imaging when the askap_imager is used (DO_ALT_IMAGER=true) is now better described. The input parameter NUM_SPECTRAL_CUBES is now NUM_SPECTRAL_WRITERS, and the pipeline is better able to handle a single output (FITS) cube written by multiple writers.
- The running of the validation script after continuum source-finding now has the $ACES environment variable set correctly. The validation script requires it to be set, and when it was not set within a user’s environment the script could crash.
- The image-based continuum subtraction script has had two fixes:
- The cube name was being incorrectly set when the single-writer FITS option was used
- The working directory was the same for all sub-bands for a given beam. This could cause issues with casa’s ipython log file, resulting in jobs crashing with obscure errors.
A patch release that fixes bugs in both the pipeline scripts and Selavy, as well as a minor one in casdaupload.
Finally, the casdaupload utility would fail if presented with a wildcard that did not resolve to anything. It will now just carry on, ignoring that particular parameter.
User documentation changes only. No code changes.
New features:
- linmos now produces mosaicks with correct masking of pixels in in both CASA and FITS formats.
- linmos can also remove the contribution of the primary beam frequency dependence to the Taylor term images. This only applies to Gaussian primary beam models.
- Added Selavy support for FITS outputs
- Addition of ACES-OPS module to facilitate controlled dependency between ASKAPsoft and ACES Tools.
- Parallelised the RM Synthesis module in Selavy.
- New Selavy output - a map of the residual emission not covered by the fitted Gaussians in a continuum image.
- Developed patch for casacore’s poor handling of the lanczos interpolation method.
- Added support for casdaupload to handle spectral-line catalogues.
- CASDA related Support for new image types.
- Ensure calibration tables are uploaded to CASDA.
- Added support for continuum validation script and results including CASDA upload.
- Improvements to Selavy spectral-line parameterisation.
- Selavy sets spectral index & curvature to a flag-value if not calculated rather than leaving as zero.
Bug fixes:
- linmos, reduced memory footprint. A bug was found that was causing a complete image cube to accessed, when only the image shape was required. This has been fixed.
- Selavy catalogues occasionally fail CASDA validation due to wide columns - fixed.
- Fixed bug where restore.beam.cutoff value not read from parset when present.
- Added missing beam log output to new imager.
- Improved handling of failed processing and the effect of that on executing final diagnostics/FITSconversion/thumbnails jobs at end of pipeline.
- Use number of beams in footprint rather than assume 36.
- Minor bug fixes
This patch release fixes the following bugs in the pipeline scripts:
- Incorrect indexing of some self-calibration array parameters
- Better handling of logic in determining the usage of the alternative imager.
- Ensuring the image-based continuum-subtracted cubes are converted to FITS and handled by the CASDA upload. Also that this task is able to see cubes directly written to FITS by the spectral imagers.
- Fixing handling of directory names so that extracted artefacts are found correctly for FITS conversion.
- Removal of extraneous inverted commas in the continuum imaging jobscript.
Additionally, there is a new parameter USE_CLI, which defaults to true but allows the user to turn off use of the online services, should they not be available.
Finally, a number of the default parameters used by the bandpass calibration and the continuum imaging have been updated, following extensive commissioning work with the 12-antenna early science datasets. Here is a list of the changed parameters:
NCYCLES_BANDPASS_CAL=50
NUM_CPUS_CBPCAL=216
BANDPASS_MINUV=200
BANDPASS_SMOOTH_FIT=0
BANDPASS_SMOOTH_THRESHOLD=3.0
NUM_TAYLOR_TERMS=1
NUM_PIXELS_CONT=3200
CELLSIZE_CONT=4
RESTORING_BEAM_CUTOFF_CONT=0.5
GRIDDER_OVERSAMPLE=5
CLEAN_MINORCYCLE_NITER=4000
CLEAN_PSFWIDTH=1600
CLEAN_SCALES="[0]"
CLEAN_THRESHOLD_MINORCYCLE="[40%, 1.8mJy]"
CLEAN_NUM_MAJORCYCLES="[1,8,10]"
CLEAN_THRESHOLD_MAJORCYCLE="[10mJy,4mJy,2mJy]"
PRECONDITIONER_LIST="[Wiener]"
PRECONDITIONER_GAUSS_TAPER="[10arcsec, 10arcsec, 0deg]"
PRECONDITIONER_WIENER_ROBUSTNESS=-0.5
RESTORE_PRECONDITIONER_LIST="[Wiener]"
RESTORE_PRECONDITIONER_GAUSS_TAPER="[10arcsec, 10arcsec, 0deg]"
RESTORE_PRECONDITIONER_WIENER_ROBUSTNESS=-2
SELFCAL_NUM_LOOPS=2
SELFCAL_INTERVAL="[57600,57600,1]"
SELFCAL_SELAVY_THRESHOLD=8
RESTORING_BEAM_CUTOFF_CONTCUBE=0.5
RESTORING_BEAM_CUTOFF_SPECTRAL=0.5
This patch release fixes the following bugs in the pipeline scripts:
- The ntasks-per-node parameter for the continuum subtraction could still be more than ntasks for certain parameter settings.
- When using a subset of the spectral channels, the new imager jobs were not configured properly, with some elements trying to use the full number of channels.
- Mosaicking of the image-based-continuum-subtracted cubes was not waiting for the completion of the continuum subtraction jobs, so would invariably fail to run correctly.
- The image-based continuum-subtraction jobs are now run from separate directories, so that ipython logs can not conflict.
- The spectral source-finding job had an error in the image name in the parset.
- Mosaicking of the continuum-cubes now creates separate weights cubes for each type of image product.
- Continuum imaging with the new imager has been improved, fixing inconsistencies in the names of images.
- The PNG thumbnails were not being propagated to the CASDA directory.
The noise map produced by Selavy is now included in the set of artefacts converted to FITS and sent to CASDA.
Additionally, the ability to impose a position shift to the model used in self-calibration has been added, with the aim of supporting on-going commissioning work.
This patch release sees a few bug-fixes to the pipeline scripts:
- When re-running the pipeline on already-processed data, where the raw input data no longer exists in the archive directory, the pipeline was previously failing due to it not knowing the name of the MS or the related metadata file. It now has the ability to read MS_INPUT_SCIENCE and MS_INPUT_1934 and determine the metadata file from that. It will also not try to run jobs that depend on the raw data.
- The new imager used in spectral-line mode can now be directed to create a single spectral cube, even with multiple writers, via the ALT_IMAGER_SINGLE_FILE and ALT_IMAGER_SINGLE_FILE_CONTCUBE parameters.
- There have been changes to the defaults for the number of cores for spectral imaging (from 2000 to 200) and the number of cores per node for continuum imaging (from 16 to 20), based on benchmarking tests.
- In addition, the following bugs were fixed:
- The ntasks-per-node parameter could sometimes be more than ntasks, causing a slurm failure.
- The self-calibration algorithm was not retaining images from the intermediate loops.
- The image-based continuum subtraction script was not finding the correct image cube.
New features and updates:
- Scheduling block state changes, in conjunction with a new TOS release:
- The CP manager now monitors the transition from EXECUTING to OBSERVED, and the ICE interfaces have been updated accordingly.
- The pipeline will now transition the scheduling block state from OBSERVED to PROCESSING at the beginning of processing. This will only be done for scheduling blocks in the OBSERVED state, and will apply to both the science field and the bandpass calibrator.
- Python libraries:
- 3rdParty python libraries have been updated to current versions. This applies to: numpy, scipy, matplotlib, pywcs, pytz, and APLpy. The current astropy package has been added, and pyfits has been removed. The python scripts in Analysis/evaluation have been updated to be consistent with these new packages.
- There is a new script in Analysis/evaluation, makeThumbnailImage.py, that produces grey-scale plots of continuum images, and has the capability to add weights contours and/or continuum components. This script is used by the makeThumbnails script in the pipeline, as well as the new diagnostics script (that produces more complex plots aimed at being aids for quality analysis).
- Calibration & Imaging changes:
- The residual image is now the residual at the end of the last major cycle. (Previously, it was the residual at the beginning of the last major cycle.)
- The residual images now have units of Jy/beam rather than Jy/pixel, and have the restoring beam written to the header.
- When the “restore preconditioner” option is used in imaging, the residual and psf.image are also written out for this preconditioner.
- Pipeline updates:
- There is a new pipeline parameter, CCALIBRATOR_MINUV, that allows the bandpass calibration to exclude baseline below some value.
- Minor errors and inconsistencies in some catalogue specifications have been fixed, with the polarisation catalogue being updated to v0.7.
- The spectral-line catalogue has been added to the CASDA upload part of the pipeline, and has been renamed to incorporate the image name (in the line of other data products).
- There are new pipeline parameters SELFCAL_REF_ANTENNA & SELFCAL_REF_GAINS that allow the self-calibration to use a reference antenna and/or gain solution.
- A weights cutoff for Selavy can now be specified via the config file using the new parameters SELAVY_WEIGHTS_CUTOFF & SELAVY_SPEC_WEIGHTS_CUTOFF (rather than using the linmos cutoff value).
- The new imager is better integrated into the pipeline, with DO_ALT_IMAGER parameters for CONT, CONTCUBE & SPECTRAL.
- It is possible to make use of the direct FITS output in the pipeline, by using “IMAGETYPE_xxx” parameters for CONT, CONTCUBE & SPECTRAL. Note that this is still somewhat of a work-in-progress.
Bug fixes:
- Casacore v2 had several patches added that had been left out of the upgrade. Notably a patch allowing the use of the SIGMA_SPECTRUM measurement set column following concatenation of measurement sets.
- The mssplit utility has been made more robust with memory allocation when splitting large datasets.
- Better checking of the size of SELFCAL- and imaging-related arrays in the pipeline configuration, particularly when not using self-calibration.
- [Weights bug in Selavy]
- The continuum-subtracted cubes were not being mosaicked by the pipeline.
- The pipeline is more robust against errors encountered when obtaining the metadata at the beginning. It can better detect when a corrupted metadata file is present, and re-run the extraction of that metadata.
- An error in handling the beam numbering for non-zero beam numbers was identified & fixed.
- The pipeline Selavy jobs were using the incorrect weights cutoff, leading to them not searching the full extent of the image.
- The use of the PURGE_FULL_MS flag in the pipelines will now not trigger the re-splitting (and subsequent processing) of the full-resolution dataset.
New features:
- Capability for direct FITS output from imager. The “fits” imagetype is now supported for cimager and imager. This should be considered “beta” as the completeness of the header information for post processing has not been confirmed. This enables the parallel write of FITS cubes which considerably improves the performance of spectral line imaging.
- Selavy’s RM Synthesis module can export the Faraday Dispersion Function to an image on disk.
- New source-finding capabilities in the processing pipelines, with a spectral-line source-finding task added (using Selavy), and the option of RM Synthesis done in the continuum source-finding.
- The full-resolution measurement set can be purged by the pipeline when no longer needed (ie. after the averaging has been done, and if no spectral-line imaging is required). This will help to minimise unncessary disk-space usage.
- CASDA upload is now able to handle extracted spectral data products (object spectra and moment maps etc) that are produced by the source-finding tasks.
- A few relatively minor additions have been made to the pipeline scripts:
- A minimum UV distance can be applied to the bandpass calibration.
- The checks done on the self-calibration parameters are less restrictive and less prone to give warning messages.
- Mosaicking at the top level (combining FIELDs) is now not done when there is only a single FIELD.
- User documentation has been updated to better reflect the current arrangements with Pawsey (e.g. host names and web addresses). It also describes new modules that are available, as well as alternative visualisation options using Pawsey’s zeus cluster.
Bug fixes:
- Imaging:
- The brightness units in the restored images from the new imager are now correctly assigned (they were ‘Jy/pixel’ and are now ‘Jy/beam’). The beam is also now written correctly.
- The beam logs (recording the restoring beam at each channel of an image cube) are now read correctly - previously the comment line at the start was not being ignored.
- A number of fixes for the spectral line imaging mode of “imager” have been implemented. These fix issues with zero channels caused by flagging.
Analysis:
- The Faraday Dispersion function in Selavy’s RM Synthesis module was being incorrectly normalised. It is now normalised by the model Stokes I flux at the reference frequency.
- Pipelines:
- When using more than one Taylor term in the imaging, the continuum subtraction with cmodel images was not working correctly, with incomplete subtraction. This was due to a malformed parset generated within the pipeline. This has been fixed, and the continuum subtraction works as expected.
- The beam logs are now correctly passed to Selavy for accurate flux correction of extracted spectra.
- Job dependencies for the mosaicking and source-finding jobs have been fixed, so that all jobs start when they are intended to. The mosaicking jobs now only start when they are needed, to avoid wasting resources.
- The project ID was incorrectly obtained from the schedblock service when there was more than one word in the SB alias.
- The SELAVY_POL_WRITE_FDF parameter was incorrectly described in the documentation - it has been renamed SELAVY_POL_WRITE_COMPLEX_FDF.
A patch release that is largely bug fixes, with several minor updates to the pipeline scripts.
New features:
- The pipelines will now accept a list of beams to be processed, via a comma-separated list of beams and beam ranges - for instance 0,1,4,7-9,16. This should be given with the BEAMLIST configuration parameter. If this is not given, it falls back to using BEAM_MIN & BEAM_MAX as usual.
- An additional column is now written to the stats files, showing the starting time of each job.
- There is a new parameter FOOTPRINT_PA_REFERENCE that allows a user to specify a reference rotation angle for the beam footprint, should it not be included in the scheduling block parset.
- There is a new parameter NCHAN_PER_CORE_SPECTRAL_LINMOS that determines how many cores are used for the spectral-line mosaicking. This helps ensure that the job is sized such that the memory load is spread evenly.
Bug fixes:
- Imaging:
- Improvements to the new imager to handle writers who do not get work due to the barycentring.
- Improvements to the allocation of work within the new imager.
- RM Synthesis & Selavy:
- The new RM Synthesis module was not correctly respecting the ‘%p’ wildcard in image names, which also affected extraction run from within Selavy. This has been fixed.
- Pipelines:
- The findBandpass slurm job had a bug that stopped it completing successfully.
- A number of bugs were identified with the mosaicking:
- The Taylor term parameter was set incorrectly in the continuum mosaicking scripts.
- The image name was not being set correctly in the spectral-line mosaicking.
- The job dependencies for the spectral-line mosaicking have been fixed so that all spectral imaging jobs are included.
- The askapsoft module is now loaded more reliably within the slurm jobs.
- The return value of the askapcli tasks is now tested, so that errors (often due to conflicting modules) can be detected and the pipeline aborted.
- A certain combination of parameters (IMAGE_AT_BEAM_CENTRES=false and DO_MOSAIC=false) meant that the determination of fields in the observation was not done, so no science processing was done. This has been fixed so that the list of fields is always determined.
- A couple of bugs in the source-finding script were fixed, where the image name was incorrectly parsed, and the Taylor 1 & 2 images were not being found.
- The footprint position angle for individual fields was incorrectly being added to the default value listed in the scheduling block parset.
- To avoid conflicts between source-finding results of different images, the artefacts produced by selavy (catalogues and images) now incorporate the image name in their name. The source-finding jobs are also more explicit in which image they are searching.
- Finally, two deprecated scripts have been removed from the pipeline directory.
A release with a number of bug fixes, new features, and updates to the pipeline scripts
New features:
- Rotation Measure synthesis is now possible within the Selavy source-finder. This extracts Stokes spectra from continuum cubes at the positions of identified continuum components, performs RM Synthesis, and creates a catalogue of polarisation properties for each component. While still requiring some development, most features are available and should permit testing.
- The new imager, which was made available in an earlier release, has been added to the askapsoft module at Pawsey.
Bug fixes for processing software:
- The bandpass calibrator cbpcalibrator will now not allow through a bandpass table with NaN values in it. If NaNs appear in solving the bandpass, then cbpcalibrator will throw an exception. In the process, the GSL library used in 3rdParty has been updated to v1.16.
- The writing of noise maps by Selavy (in the VariableThreshold case) has been streamlined, so that making such maps for large cubes is more tractable.
Pipeline updates:
- The driving script for the ASKAP pipeline is now called processASKAP.sh, instead of processBETA.sh. The latter is still available, but gives a warning before callling processASKAP.sh. All interfaces remain the same.
- Linear mosaicking has been improved:
- It is now available for spectral-line and continuum cubes, in addition to continuum images.
- Mosaics are made for each field, and for each tile if the observation was done with the “tilesky” mode.
- The continuum mosaicking can also include mosaics of the self-calibration loops.
- The pipelines make better use of the online services of ASKAP, to determine things like the footprint (location of beams). This makes calculations more internally self-consistent.
- When running self-calibration, some parameters can be given different values for each loop. This includes parameters for the cleaning, the source-finding, and the calibration. More flexibility is also provided for the source-finding within the self-calibration.
- Processing of BETA datasets are made possible via an IS_BETA parameter, which avoids using the online system to obtain beam locations, and changes the defaults for the data location.
- Smoothing of the bandpass solutions is now possible, using a script in the ACES repository to produce a new calibration table. It also allows plotting of the calibration solutions.
- More flexibility is allowed for the number of cores used in the continuum imaging.
- A notable bug was fixed that led to incorrect calibration and continuum-subtraction when Taylor-terms were being produced (i.e. nterms>1)
- Various other more minor bug fixes, related to logging, stats files, and default values of parameters (for instance, the default for cmodel was to use a flux cutoff that was too high).
This is a patch release that fixes several issues:
- The parallel linear mosaicking tool linmos-mpi has been patched to correct a bug that was initialising cube slices incorrectly.
- Several fixes to the CP manager and the pipeline scripts were made following end-to-end testing with the full ASKAP online system:
- The CP manager will send notifications to a nominated JIRA ticket upon SB state changes.
- Several fixes were made to the CASDA uploading and polling scripts, to ensure accurate execution. The capability of sending notifications to a JIRA ticket has also been added.
- The Project ID is now taken preferentially from the SB, rather than the config file.
- The linear mosaicking in the pipelines is now not turned off when only a single beam is processed.
This is a patch release that fixes a couple of issues:
- The bandpass calibrator cbpcalibrator has had its run-time improved by changing the way the calibration table is written. It is now written in one pass at the completion of the task - this reduces the I/O overhead and greatly reduces the run-time for larger datasets.
- The pipeline settings for the flagging have been changed. The default settings now are to have the integrate_spectra option switched on, and the integrate_times and flat amplitude options switched off. This is the same approach as used in 0.14.0-p2 and earlier, and so should avoid the case of most of the dataset being flagged (as was seen with ADE data using the default settings in 0.15.0).
- The flagging step for the average dataset now uses a different check-file to the full-size dataset flagging.
This release sees a number of bugs fixes and improvements.
A further update only to the pipeline processing:
- Changes to the directory structure created by the pipeline. Each field in the MS is given its own directory, within which processing on all beams is done. The bandpass calibrator likewise gets its own directory. All files & job names are now identified by the field and the beam IDs.
- Flagging of the science data is now done differently. The MS is first bandpass-calibrated, and then flagged. After averaging, there is the option to run the flagging again on the averged data. The flagging for the bandpass calibrator has not been changed.
- The dynamic flagging for the science data also allows the use of both integrateSpectra and integrateTimes, with the former no longer done by default.
- Modules are loaded correctly by the scripts and slurm jobs before particular tasks are used, so that the scripts are less reliant on the user’s environment.
- Better handling of metadata files, particularly if a previous metadata call had failed.
- The FITS conversion and thumbnail tasks correctly interact with the different fields, and the thumbnail images make a better measurement of the image noise, taking into account any masked regions from the associated weights images.
- The cleaning parameter Clean.psfwidth is exposed to the configuration file.
- Bugs in associating the footprint information with the correct field have been fixed.
- If the CASDA-upload script is used to prepare data for deposit, the scheduling block state is transitioned to PENDINGARCHIVE.
An update to the pipeline processing only:
- Fixing a bug in the handling of multiple FIELDs within a measurement set. These are now correctly given their own directory for the processed data products.
- The footprint parameters are now preferentially determined from the scheduling block parset (using the ‘schedblock’ command-line utility). If not present, the scripts fall back to using the config file inputs.
- The metadata files (taken from mslist, schedblock and footprint.py) are re-used on subsequent runs of the pipeline, rather than re-running each of these tools.
- The default bucketsize for the mssplit jobs has been increased to 1MB, and made configurable by the user. The stripe count for the non-data directories has also been changed to 1.
A major release, with several new features and improvements for both the imaging software and the pipeline scripts.
A new imager in under test in this release, currently just called “imager” and it has the following features:
- In continuum mode it allows a core to process more than one channel. This has a small cost in memory and a proportional increase in disk access. But allows the continuum imaging to proceed with a much smaller footprint on the cluster. This will allow simultaneous processing of all beams in a coming release.
- Spectral line cubes can be made from measurement sets that are from different epochs. The epochs are imaged separately but merged into the same image for minor-cycle solving.
- The output spectral line cubes can be in the barycentric frame. This is currently just nearest neighbour indexing. But the possibility of interpolation has not been designed out.
- The concept of “multiple writers” has been introduced to improve the disk access pattern for the spectral line mode. This breaks up the cube into frequency bands. These can be recombined post-processing.
- If you really want to increase the performance for many major cycles you can also turn on a shared memory option which stores visibility sets in memory throughout processing.
- The imager takes the same parset as Cimager - but extra key-value pairs are required to implement the features.
This new imager is still under test and we have not added the hooks into the pipeline yet.
- The headers of the FITS files created by the pipelines now have a wider range of metadata, including observatory and date-obs keywords, as well as information about the askapsoft & pipeline versions.
- The restore preconditioner options mentioned above are available through “RESTORE_PRECONDITIONER_xxx” parameters, for the continuum imaging only (it is not implemented for simager).
- Several bugs were fixed:
- The continuum subtraction was failing when using components if no sources were found - it now skips the continuum subtraction step.
- The askapdata module was, in certain situations, not loaded correctly, leading to somewhat cryptic errors in the imaging.
- The parsing of mslist to obtain MS metadata would sometimes fail, depending on the content of the MS. It is now much more robust.
- The default for TILENCHAN_SL has been increased to 10, to counter issues with mssplit running slow.
This bug-fix version addresses a few issues with the imaging & source-finding code, along with minor updates to the pipeline scripts. The following bugs have been fixed in the processing software:
- Caching of the Wiener preconditioner is now done, so that the weights are only calculated once for all solvers and the filters are only calculated once for all major cycles, scales & Taylor-terms. This has the effect of greatly speeding up the imaging, particularly for large image sizes.
- The BasisfunctionMFS solver has had the additional convolution with the PSF removed. This fixes a bug where central sources were being cleaned preferentially to sources near the edge of the image. It also improves the resolution and SNR of minor-cycle dirty images.
- From the update to casacore-2 in 0.13.0, linmos would fail when mosaicking images without restoring beams. This has been fixed (and behaves as it did prior to 0.13.0).
- The size check in Selavy that rejects very large fitted components has been re-instated. This should allow the rejection of spurious large fitted components. The minimum size requirement (which forced sizes to be >60% of the PSF) has been removed.
This bug-fix version primarily addresses issues with the processing pipelines. The following bugs have been fixed:
- Non-integer image cell sizes were not being interpreted correctly. These values can now be any decimal value.
- A change in the mslist output format with casacore v2 meant that the Cmodel continuum subtraction script was not reading the correct reference frequency. This caused the cmodel job to fail for the case of nterms>1. The parsing code has been fixed.
- The archiving scripts had a few changes:
- The resolution of filenames & paths has been fixed.
- The source-finding is now run on FITS versions of the images
- The catalogue keys in the observation.xml are now internally consistent.
- The way thumbnail sizes are specified in the pipeline configuration file has changed slightly.
Related to the above changes, the C++ code has had a couple of changes:
- casdaupload now correctly puts the thumbnail information in the <image> group in the observation.xml file.
- Fixes were made to the Selavy VOTable output to fix formatting errors that were preventing it passing validation upon CASDA ingest.
The documentation has also been updated, with updated descriptions of parameters that have changed as a result of the above, a few typos fixed, and new information about the management of data on Pawsey’s scratch2 filesystem.
This version fixes a few issues with the processing pipelines, fixes some bugs with the source-finder and casda upload utility, and moves the underlying code to use version 2 of the casacore package.
A bug fix release for the processing pipeline. This fixes a problem where the mosaicking task was still assuming beam IDs that had a single integer - ie. it was looking for image.beam0.restored instead of image.beam00.restored.
This is a simple patch release that fixes a couple of bugs, one of which affected the performance of both the source-finder and the pipelines.
The measurement of spectral indices for fitted components to continuum Taylor-term images was being done incorrectly, leading to erroneous values for spectral-index and spectral-curvature. This, in turn, could lead to inaccuracies or even failures in the continuum-subtraction task of the pipeline (when the CONTSUB_METHOD=Cmodel option was used). This only affected version 0.12.0 (released on 8 May 2016), and is fully corrected in 0.12.1.
The other bug enforces the total number of channels processed by the pipelines to be an exact multiple of the averaging width (NUM_CHAN_TO_AVERAGE). In previous versions, the pipeline scripts would press on, but this would potentially result in errors in the slurm files and jobs not executing. Now, should NUM_CHAN_TO_AVERAGE not divide evenly into the number of channels requested, the script will exit with an error message before submitting any jobs.
This version has a number of changes to the processing applications and the pipeline scripts.
This release is a relatively small bug-fix update, primarily fixing a bug in cimager.
This bug would prevent a parallel job completing in the case of the major cycle threshold being reached prior to the requested maximum number of major cycles.
The imaging software now incorporates the preservecf option (released in 0.11.0) into the SphFunc gridder, and introduces a new option to the gridding - snapshotimaging.longtrack - that predicts the best fit W plane used for the snapshot imaging, finding the plane that minimises the future deviation in W. This can have substantial savings in processing time for long tracks.
The pipeline scripts have seen a number of minor improvements and fixes, with improved alternative methods for continuum subtraction, and improved reporting of resource usage (including a record of the number of cores used for each job). The user configuration file is now also copied to a timestamped version for future reference.
The ingest pipeline code has incorporated changes resulting from the recent commissioning activities.
A key change made in the processing software relates the preconditioning. There is a new parameter preconditioning.preservecf that should be set to true for the case of using WProject and the Wiener preconditioner. This has fixed a couple of issues - at low (negative) robustness values, the cross-shaped artefact that was sometimes seen has now gone, and the performance should now more closely match that expected from robust weighting for the full range of robustness values.
Additionally, the regridding has been sped up through a patch to the casacore library.
The pipeline scripts also have a new feature, making use of Selavy + Cmodel to better perform the continuum subtraction from spectral-line data. The old approach is still available, but is not the default.
Much of this release relates to updates to the ingest pipeline and related tasks, in preparation for getting it running at Pawsey. These are now deployed as their own module, although it is not expected that ACES members will need to use this.
In the science processing area, an important fix was made to the code responsible for uvw rotations. A fault was identified where these were being projected into the wrong frame, which could lead to positional offsets in images made away from the initial phase centre. This fault has been fixed.
Some initial fixes to the preconditioner have been made that may improve images when Wiener filtering with a low or negative robustness parameter. Improvements are only expected when snapshot imaging is not being used. A full fix is being tested and is planned for the next release.
This release also sees the BETA pipeline scripts move into an askapsoft-derived module (although this had previously been announced).
There are only a small number of changes to the core processing part of the software that would affect ACES work on galaxy, and these are almost all to do with the source-finder Selavy. The default values of some parameters governing output files have changed, with the preference now to minimise the number of output files. A few corrections have been made to the units of parameters in some of the output catalogues.
This release introduces simager, the prototype spectral-line imager - this allows imaging of large spectral cubes through distributed processing, and is capable of creating much larger cubes than cimager. While this is not the final version of the spectral-line imager - the software framework that underpins the imaging code is going through a re-design prior to early science - it does demonstrate the distributed-processing approach that enables large numbers of spectral channels to be processed.
For those wanting to make use of the ACES scripts under subversion, these will be updated shortly to include use of simager.
Other changes to the askapsoft module include minor updates to the CASDA HI catalogue interface from the Selavy sourcefinder, and ADE-related updates to the ingest pipeline and associated tools (which won’t affect work on galaxy).
This release has a few relatively small bug fixes that have been resolved in the past week:
- a minor fix to cimager that solves a rare problem with the visibility metadata statistics calculations, that would result in cimager failing (this had been seen in processing the basic continuum tutorial data).
- correcting the shape (BMAJ/BMIN/BPA) parameters in the Selavy-generated component parset output (that might be used as input to ccalibrator in self-calibration) - they were previously given in arcsec/degrees rather than radians (as required by ccalibrator/csimulator).
- aligning the cmodel VOTable inputs with the new Selavy output formats
- a fix to the units in one of the Selavy VOTable outputs
This release is a bug-fix release aimed at fixing a problem identified in running the basic continuum imaging tutorial. There was an issue with the way the simulated data had been created, which meant that mssplit would fail on those measurement sets. This has been fixed (fixing both mssplit and msmerge), and the tutorial dataset and description have been updated.
If you use mssplit on real BETA data, you will not notice any difference, save for potentially a small performance improvement.
The only other change has been implementation of the CASDA format for absorption-line catalogues, although the implementation of actual absorption-line searching is not complete in Selavy, so this will probably not affect any of you (it has been more to provide early examples for use by the CASDA team).
Changes for this release include bug fixes and improvements to assist the casdaupload tool, and a calibration bug that affected leakage terms. The release notes follow.
A bug-fix release adding a couple of elements to 0.6.0:
Some highlight features and bugfixes are:
- [ASKAPSDP-1652] - Gridding failing with concatenated MS
- [ASKAPSDP-1654] - Selavy’s component parset output gets positions wrong
- [ASKAPSDP-1646] - Develop CASDA upload utility
- [ASKAPSDP-1649] - Add selection by field name to mssplit
- [ASKAPSDP-1653] - Add parset parameter to change the weight cutoff used in linmos
A bug fix release, providing an option to flag antennas with out-of-range delays in the DRx or FR hardware setting.
The list of features & bugfixes is below:
A minor update, with the following features added:
- [ASKAPSDP-1610] - Account for averaging when setting noise sigma values in mssplit
- [ASKAPSDP-1612] - Add support for SIGMA_SPECTRUM column to Data Accessor
The list of features & bugfixes is below:
The version 0.3 release of the ASKAPsoft Science Data Processor has been installed as a module to Galaxy. The included features/bugfixes are listed below, and are also listed on Redmine: https://pm.atnf.csiro.au/askap/projects/cmpt/versions/197
- Bug #6029: Ingest pipeline zeros flagged visibilities
- Bug #6107: Fix the curvature-map option in Selavy’s Gaussian fitting
- Bug #6112: Ingest pipeline flags incorrect antenna
- Bug #6113: RA & Dec swapped in Ingest Pipeline Monitoring data
- Bug #6121: openssl-1.0.1c fails to build on XUbuntu 14.04
- Bug #6125: Superfluous loop over w in WProjectVisGridder::initConvolutionFunction
- Bug #6126: gridder parameter snapshotimaging.coorddecimation is ignored
- Bug #6154: Ingest pipeline should not write SBID in observation column
- Bug #6179: SVN 1.7 breaks rbuilds get_svn_revision function
- Bug #6183: Selavy - component catalogues for individual fit types are incomplete
- Feature #6073: Support of different phase and pointing centres via scheduling blocks
- Feature #6075: MSSink should populate POINTING table
- Feature #6120: Ingest Pipeline: Get obs data from TOS metadata
- Feature #6164: Tool to assist delay calibration
- Feature #6180: Add –version cmdline parameter to askap::Application
- Task #6176: SDP codebase restructure
- Documentation #6106: Create an analysis tutorial