[sofia] Release of SoFiA 1.3.0
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Paolo Serra <paolo.serra_at_email.protected>
Date: Mon, 15 Apr 2019 09:06:56 +0200
Dear colleague,
The SoFiA team is pleased to announce the release of SoFiA 1.3.0, the latest stable version of the Source Finding Application, now available at https://github.com/SoFiA-Admin/SoFiA/releases/tag/v1.3.0 <https://github.com/SoFiA-Admin/SoFiA/releases/tag/v1.3.0> .
This is the 11ᵗʰ stable release of SoFiA. It introduces a few new features and comes with a large number of improvements and bug fixes.
The most notable new features include the introduction of source quality flags that indicate if a source might be truncated for a number of reasons, new control parameters to automatically remove detections at the linking stage (e.g. based on a maximum size threshold), and an experimental feature to automatically identify and flag lines of sight that are dominated by artefacts such as residual continuum emission.
In addition to these new features, several changes have been made to improve the user experience. The most important one of these changes affects the way in which source bounding boxes are defined in the catalogue. Originally, SoFiA would define bounding boxes as [min, max + 1] which would make it straightforward to use them in Python array slicing commands. However, as this is presumably not what the typical user would naïvely expect, the definition was changed to the more logical [min, max].
Users who have processed bounding boxes in their own scripts in the past will need to update their code to account for the new definition.
Another significant change occurred as a result of our attempts to make reliability filtering more robust. Originally, reliability filtering would by default be carried out in [n_pix, snr_max, snr_sum] parameter space. However, it turned out that n_pix and snr_sum were highly correlated, effectively reducing the parameter space from 3-D to 2-D. To reduce the degree of correlation, we have replaced n_pix with snr_mean which does not strongly correlate with either of the other two parameters. In addition, we no longer apply a fixed flux threshold to discard detections believed to be unreliable due to their low signal-to-noise ratio. Instead, the user can now supply a threshold in snr_sum / sqrt(n_pix), which is a much better proxy of integrated signal-to-noise ratio, to discard low-SNR detections.
A more general overview of the most notable changes and bug fixes is given below.
New features
Quality flags: SoFiA now provides a flag that indicates if a source touches the boundaries of the cube, blanked pixels or neighbouring sources. (67881fb <https://github.com/SoFiA-Admin/SoFiA/commit/67881fb32ec394a1e08ff8fb00194ed7bc3fc79c>)
Inverted cube: An option to invert the input data cube was added to facilitate the search for absorption features. (e554505 <https://github.com/SoFiA-Admin/SoFiA/commit/e554505664b15ab32aa81139a748b3d0c74b520b>)
Source list: The user can now provide a source list with any input mask to direct SoFiA to only retain certain sources from the input mask and discard all other ones. (18fe693 <https://github.com/SoFiA-Admin/SoFiA/commit/18fe69305a15a56822d2dfd0d95862d2341f2252>)
Memory tracking: A new option will direct SoFiA to regularly report on its peak memory usage throughout the pipeline run. (18fe693 <https://github.com/SoFiA-Admin/SoFiA/commit/18fe69305a15a56822d2dfd0d95862d2341f2252>)
Flagging of artefacts: A new experimental feature for automated flagging of artefacts such as residual continuum emission has been added. (9f6bdda <https://github.com/SoFiA-Admin/SoFiA/commit/9f6bdda977368346f84e0195b61c43f2ddc9384e>, fdce62e <https://github.com/SoFiA-Admin/SoFiA/commit/fdce62e5e8ffe455ccb9b29cb4ea029e7a8d61f3>)
New linker options: The linker now offers additional filtering options to discard detections, e.g. maximum size thresholds. Some of these have been added to the GUI as well. (2c80693 <https://github.com/SoFiA-Admin/SoFiA/commit/2c80693199525590f9280c5191e0c69e9490a8c5>, 2d692df <https://github.com/SoFiA-Admin/SoFiA/commit/2d692dfc1c9ba3dacc3b5ab83cb9aea73798c802>)
Bug-fixes
Improved mask handling by fixing a few bugs and implementing additional sanity checks. (21e4a3b <https://github.com/SoFiA-Admin/SoFiA/commit/21e4a3b046833e6dbae4c9bcea8a5c1f6c42c323>)
A bug in the source names generated by SoFiA was fixed whereby the coordinate part of the name was not correctly rounded. (7a3121b <https://github.com/SoFiA-Admin/SoFiA/commit/7a3121bed8805a4fb35e243800c0363426225dfd>)
Fixed a bug related to the import of compressed FITS files. (83c6d8b <https://github.com/SoFiA-Admin/SoFiA/commit/83c6d8b35b511cb83059688a9f8610664bec63ba>)
Source bounding boxes and other parameters were not correctly updated after mask dilation and parameterisation, causing a few issues. (4ef5789 <https://github.com/SoFiA-Admin/SoFiA/commit/4ef57895d35e3d09a17828cad1118ab98de5eb2f>, 0ed685b <https://github.com/SoFiA-Admin/SoFiA/commit/0ed685b9847f87f5151d2028258a97efd9656680>)
Added several sanity checks to ensure that the reliability module doesn’t simply crash on faulty user input. (2dfeaa5 <https://github.com/SoFiA-Admin/SoFiA/commit/2dfeaa53de5b92999108a20174aaa8b167b51b3d>)
A NumPy compatibility issue with NumPy 1.10 or higher has been fixed. (bce9ffe <https://github.com/SoFiA-Admin/SoFiA/commit/bce9ffe4c34d7aacfd1fab03ecb7274bdd7fd326>, ed14b0a <https://github.com/SoFiA-Admin/SoFiA/commit/ed14b0a5d5e08bc77cb99f5b583462e4fac200b3>)
Other changes
Mask dilation: Originally, masks were dilated by a fixed amount along the spectral axis, whereas mask dilation is now based on flux conversion in all three dimensions. (157de7b <https://github.com/SoFiA-Admin/SoFiA/commit/157de7b16cba17e9ebd485a8ad09d370935754f3>)
Reliability SNR threshold: Instead of a fixed cut in parameter space, SoFiA now applies a threshold in snr_sum / sqrt(n_pix) to filter out sources deemed unreliable due to low SNR. (590699f <https://github.com/SoFiA-Admin/SoFiA/commit/590699f2870c495966f98fcfb2cbb7d1251ce695>, efe3a0e <https://github.com/SoFiA-Admin/SoFiA/commit/efe3a0e84cbc4ceb45dd651ec28c735e890d8459>)
Bounding boxes: Source bounding boxes are now reported as [min, max] rather than the original [min, max + 1]. The former behaviour was more suitable for use in Python array slicing calls, but the new behaviour is probably what most users would expect. (2c80693 <https://github.com/SoFiA-Admin/SoFiA/commit/2c80693199525590f9280c5191e0c69e9490a8c5>)
Reliability parameter space: The default of reliability.parSpace was changed to replace n_pix with snr_mean to ensure that the parameters used in calculating reliability are only weakly correlated. (15ff158 <https://github.com/SoFiA-Admin/SoFiA/commit/15ff158cfd1852638eab65143391dcafac29369a>)
Graphical user interface: The GUI has been continuously updated and provides a lot of additional control parameters and an up-to-date user manual.
Known issues
A few of the source parameters returned by the linker are currently believed to be incorrect for reasons still to be investigated (see issue #125 <https://github.com/SoFiA-Admin/SoFiA/issues/125>). However, this should not affect the average SoFiA user and would only become an issue if the default parameter space for reliability calculation was changed. We therefore recommend to leave the value of the reliability.parSpace parameter at its default for the time being.
The CNHI finder does not appear to be deterministic, and the number of sources detected by the algorithm varies when run repeatedly with the same settings. The cause and impact of this are under investigation.
Busy Function fitting is currently broken. We instead advise to fit the Busy Function to the source spectra produced by SoFiA using separate fitting software such as BusyFit (https://github.com/SoFiA-Admin/BusyFit <https://github.com/SoFiA-Admin/BusyFit>).
On behalf of the SoFiA team,
Paolo Serra
Received on 2019-04-15 17:07:17
Date: Mon, 15 Apr 2019 09:06:56 +0200
Dear colleague,
The SoFiA team is pleased to announce the release of SoFiA 1.3.0, the latest stable version of the Source Finding Application, now available at https://github.com/SoFiA-Admin/SoFiA/releases/tag/v1.3.0 <https://github.com/SoFiA-Admin/SoFiA/releases/tag/v1.3.0> .
This is the 11ᵗʰ stable release of SoFiA. It introduces a few new features and comes with a large number of improvements and bug fixes.
The most notable new features include the introduction of source quality flags that indicate if a source might be truncated for a number of reasons, new control parameters to automatically remove detections at the linking stage (e.g. based on a maximum size threshold), and an experimental feature to automatically identify and flag lines of sight that are dominated by artefacts such as residual continuum emission.
In addition to these new features, several changes have been made to improve the user experience. The most important one of these changes affects the way in which source bounding boxes are defined in the catalogue. Originally, SoFiA would define bounding boxes as [min, max + 1] which would make it straightforward to use them in Python array slicing commands. However, as this is presumably not what the typical user would naïvely expect, the definition was changed to the more logical [min, max].
Users who have processed bounding boxes in their own scripts in the past will need to update their code to account for the new definition.
Another significant change occurred as a result of our attempts to make reliability filtering more robust. Originally, reliability filtering would by default be carried out in [n_pix, snr_max, snr_sum] parameter space. However, it turned out that n_pix and snr_sum were highly correlated, effectively reducing the parameter space from 3-D to 2-D. To reduce the degree of correlation, we have replaced n_pix with snr_mean which does not strongly correlate with either of the other two parameters. In addition, we no longer apply a fixed flux threshold to discard detections believed to be unreliable due to their low signal-to-noise ratio. Instead, the user can now supply a threshold in snr_sum / sqrt(n_pix), which is a much better proxy of integrated signal-to-noise ratio, to discard low-SNR detections.
A more general overview of the most notable changes and bug fixes is given below.
New features
Quality flags: SoFiA now provides a flag that indicates if a source touches the boundaries of the cube, blanked pixels or neighbouring sources. (67881fb <https://github.com/SoFiA-Admin/SoFiA/commit/67881fb32ec394a1e08ff8fb00194ed7bc3fc79c>)
Inverted cube: An option to invert the input data cube was added to facilitate the search for absorption features. (e554505 <https://github.com/SoFiA-Admin/SoFiA/commit/e554505664b15ab32aa81139a748b3d0c74b520b>)
Source list: The user can now provide a source list with any input mask to direct SoFiA to only retain certain sources from the input mask and discard all other ones. (18fe693 <https://github.com/SoFiA-Admin/SoFiA/commit/18fe69305a15a56822d2dfd0d95862d2341f2252>)
Memory tracking: A new option will direct SoFiA to regularly report on its peak memory usage throughout the pipeline run. (18fe693 <https://github.com/SoFiA-Admin/SoFiA/commit/18fe69305a15a56822d2dfd0d95862d2341f2252>)
Flagging of artefacts: A new experimental feature for automated flagging of artefacts such as residual continuum emission has been added. (9f6bdda <https://github.com/SoFiA-Admin/SoFiA/commit/9f6bdda977368346f84e0195b61c43f2ddc9384e>, fdce62e <https://github.com/SoFiA-Admin/SoFiA/commit/fdce62e5e8ffe455ccb9b29cb4ea029e7a8d61f3>)
New linker options: The linker now offers additional filtering options to discard detections, e.g. maximum size thresholds. Some of these have been added to the GUI as well. (2c80693 <https://github.com/SoFiA-Admin/SoFiA/commit/2c80693199525590f9280c5191e0c69e9490a8c5>, 2d692df <https://github.com/SoFiA-Admin/SoFiA/commit/2d692dfc1c9ba3dacc3b5ab83cb9aea73798c802>)
Bug-fixes
Improved mask handling by fixing a few bugs and implementing additional sanity checks. (21e4a3b <https://github.com/SoFiA-Admin/SoFiA/commit/21e4a3b046833e6dbae4c9bcea8a5c1f6c42c323>)
A bug in the source names generated by SoFiA was fixed whereby the coordinate part of the name was not correctly rounded. (7a3121b <https://github.com/SoFiA-Admin/SoFiA/commit/7a3121bed8805a4fb35e243800c0363426225dfd>)
Fixed a bug related to the import of compressed FITS files. (83c6d8b <https://github.com/SoFiA-Admin/SoFiA/commit/83c6d8b35b511cb83059688a9f8610664bec63ba>)
Source bounding boxes and other parameters were not correctly updated after mask dilation and parameterisation, causing a few issues. (4ef5789 <https://github.com/SoFiA-Admin/SoFiA/commit/4ef57895d35e3d09a17828cad1118ab98de5eb2f>, 0ed685b <https://github.com/SoFiA-Admin/SoFiA/commit/0ed685b9847f87f5151d2028258a97efd9656680>)
Added several sanity checks to ensure that the reliability module doesn’t simply crash on faulty user input. (2dfeaa5 <https://github.com/SoFiA-Admin/SoFiA/commit/2dfeaa53de5b92999108a20174aaa8b167b51b3d>)
A NumPy compatibility issue with NumPy 1.10 or higher has been fixed. (bce9ffe <https://github.com/SoFiA-Admin/SoFiA/commit/bce9ffe4c34d7aacfd1fab03ecb7274bdd7fd326>, ed14b0a <https://github.com/SoFiA-Admin/SoFiA/commit/ed14b0a5d5e08bc77cb99f5b583462e4fac200b3>)
Other changes
Mask dilation: Originally, masks were dilated by a fixed amount along the spectral axis, whereas mask dilation is now based on flux conversion in all three dimensions. (157de7b <https://github.com/SoFiA-Admin/SoFiA/commit/157de7b16cba17e9ebd485a8ad09d370935754f3>)
Reliability SNR threshold: Instead of a fixed cut in parameter space, SoFiA now applies a threshold in snr_sum / sqrt(n_pix) to filter out sources deemed unreliable due to low SNR. (590699f <https://github.com/SoFiA-Admin/SoFiA/commit/590699f2870c495966f98fcfb2cbb7d1251ce695>, efe3a0e <https://github.com/SoFiA-Admin/SoFiA/commit/efe3a0e84cbc4ceb45dd651ec28c735e890d8459>)
Bounding boxes: Source bounding boxes are now reported as [min, max] rather than the original [min, max + 1]. The former behaviour was more suitable for use in Python array slicing calls, but the new behaviour is probably what most users would expect. (2c80693 <https://github.com/SoFiA-Admin/SoFiA/commit/2c80693199525590f9280c5191e0c69e9490a8c5>)
Reliability parameter space: The default of reliability.parSpace was changed to replace n_pix with snr_mean to ensure that the parameters used in calculating reliability are only weakly correlated. (15ff158 <https://github.com/SoFiA-Admin/SoFiA/commit/15ff158cfd1852638eab65143391dcafac29369a>)
Graphical user interface: The GUI has been continuously updated and provides a lot of additional control parameters and an up-to-date user manual.
Known issues
A few of the source parameters returned by the linker are currently believed to be incorrect for reasons still to be investigated (see issue #125 <https://github.com/SoFiA-Admin/SoFiA/issues/125>). However, this should not affect the average SoFiA user and would only become an issue if the default parameter space for reliability calculation was changed. We therefore recommend to leave the value of the reliability.parSpace parameter at its default for the time being.
The CNHI finder does not appear to be deterministic, and the number of sources detected by the algorithm varies when run repeatedly with the same settings. The cause and impact of this are under investigation.
Busy Function fitting is currently broken. We instead advise to fit the Busy Function to the source spectra produced by SoFiA using separate fitting software such as BusyFit (https://github.com/SoFiA-Admin/BusyFit <https://github.com/SoFiA-Admin/BusyFit>).
On behalf of the SoFiA team,
Paolo Serra
Received on 2019-04-15 17:07:17