User Parameters - Bandpass Calibration¶
These parameters govern all processing used for the calibrator
observation. The requested measurement set is split up by beam and
scan, assuming that a given beam points at 1934-638 in the
correspondingly-numbered scan. While it is possible to use the
BEAM_MIN
and BEAM_MAX
parameters to specify a given range of
beams to process for the science field, only the BEAM_MAX
parameter is applied to the bandpass calibrator processing. All beams
up to BEAM_MAX
are split & flagged, and have their bandpass solved
for. This is due to the particular requirements of
cbpcalibrator.
The bandpass is initially checked to ensure it matches the science
observation. Currently, this is just done by looking at the beam
weights ID, to ensure that the two observations are done with
consistent weights (and so one could expect that the bandpass solution
would transfer to the science observation). This check can be
overridden (if you do want to try a bandpass observation with
different beam weights) by setting
OVERRIDE_WEIGHTS_CHECK=true
. The weights ID is determined from the
weights.id
parameter in the SB parset (or, for older datasets, the
beam_weights
parameter).
If the bandpass calibration observation is long, and you just wish to
use a portion of it for bandpass calibration, then you can specify a
time range through SPLIT_TIME_START_1934
and
SPLIT_TIME_END_1934
. If either is not given, the start or end time
are taken to be the start or end of the observation respectively.
The default behaviour is to flag the data with cflag / cflagger (Flagging Utility). In this case, the MS is flagged in two passes. First, a combination of selection rules (allowing flagging of channels, time ranges, antennas & baselines, and autocorrelations) and (optionally) a simple flat amplitude threshold are applied. Then a sequence of Stokes-V flagging and dynamic flagging of amplitudes is done, integrating over individual spectra. Each of these steps is selectable via input parameters.
There is an option to use the AOFlagger tool (written by Andre
Offringa) to do the flagging. This can be turned on by
FLAG_WITH_AOFLAGGER
, or FLAG_1934_WITH_AOFLAGGER
(to just do
it for the bandpass calibrator). You can provide a strategy file via
AOFLAGGER_STRATEGY
or AOFLAGGER_STRATEGY_1934
, with access to
some of the aoflagger parameters provided - see the table below.
There is also the ability to flag out the HI absorption from Miky Way
gas that is seen in the PKS B1934-638 spectrum. These channels will be
interpolated over by the bandpass smoothing (although if no smoothing
is done they will remain flagged and thus the corresponding channels
in the science data will be flagged). This option is toggled by
BANDPASS_FLAG_HI_ABSORPTION
, and the velocity range defaults to
+/- 15 km/s LSRK, but can be adjusted with VEL_RANGE_HI_ABSORPTION
.
Once all this flagging is completed, there is the option (controlled
by DO_ADDITIONAL_FLAGGING_1934
) to completely flag out any
channels that have a flagging fraction above some specified value
(defaulting to 60%). This ensures those channels are not dominated by
a small number of integrations that may have just avoided being
flagged.
Then the bandpass table is calculated with
cbpcalibrator, which requires MSs for all beams to be
given. This is a parallel job, the size of which is configurable
through NUM_CORES_CBPCAL
.
If a second processing job is run with a higher BEAM_MAX
(and
hence wanting to use beams not included in a previous bandpass
solution), the bandpass table is deleted and re-made once the new
beams are split and flagged.
The cbpcalibrator job can make use of one of two scripts developed in
the commissioning team. The first, plot_caltable.py
(from ACES, now kept in the legacy/ directory with the pipeline code) both plots
the bandpass solutions (as a function of antenna, beam and
polarisation), and smooths the bandpass table so that outlying
points are interpolated over. This produces a second table (with
“.smooth” appended to the name), which will then be applied to the
science data instead of the original. There are a number of parameters
that may be used to tweak the smoothing - this is intended to be an
interim solution until this functionality is implemented in
ASKAPsoft.
The second script, smooth_bandpass.py
, does the smoothing of the
bandpass via polynomial and/or harmonic fitting, in a way that is robust to portions of
the spectra being flagged.
This latter script will also properly take into account the location
of the edges between beam-forming intervals. The smoothing will only
occur within a beam-forming interval, and be discontinuous at the
edges. By default, the intervals are determined by taking the edge
frequencies recorded in the SB obs variables, but different intervals
may be used by providing a window list file
(BANDPASS_SMOOTH_WIN_LIST
) or other window specifications
(BANDPASS_SMOOTH_F54
or BANDPASS_SMOOTH_N_WIN
). These
parameters will also need to be provided for SBs prior to 26120, as
the weights information was not available in the SB obs variables
then.
If either of these options are used, a bandpass validation script is run, producing plots that can describe the quality of the bandpass solutions.
Finally, the bandpass solutions can be applied back to the 1934 datasets themselves. This will permit possible diagnostic analysis of the quality of the bandpass solution.
Additionally, we now have the option to derive instrumental on-axis
polarisation leakages from the 1934-638 data. The bandpass calibrated
1934-638 measurement sets are further flagged, spectrally-averaged to 1MHz
resolution, flagged again, and then leakages derived. To turn this on,
use DO_LEAKAGE_CAL_CONT=true
. The derived solutions can then be applied
to continuum averaged, and self-calibrated science data
using DO_APPLY_LEAKAGE=true
. The derivation and application of leakage
calibration is done indepdendently on each continuum channel. With release
1.1.0 of the pipeline and ASKAPsoft, users will be able to write the solution
spectra of all beams into a single table using WRITE_BPLEAKAGE=true
. This
option will reduce I/O resulting from the channel-wise splitting of the msdata.
For these standard casa tables, the pipleine will generate plots of the leakage
spectrum of all antenna/beams.
Version 1.4 of the pipeline also allows the option to obtain smooth representation of the leakage spectra. For details, see the section on leakage smoothing parameters below.
The bandpass table is also used to provide flagging directives that
may be applied to the science data. This process, known as
“pre-flagging” identifies antenna/beam combinations that have clearly
discrepant or missing values in their bandpass solutions. It can also
identify channels that have large residuals from the smooth fit (such
as may be due to RFI). These will be applied at the flagging stage
within the science-field processing. The only relevant parameter here
is the threshold for identifying discrepant channels -
PREFLAG_BADCHAN_DETECTION_THRESH
.
Variable |
Default |
Parset equivalent |
Description |
---|---|---|---|
|
true |
none |
Whether to split a given beam/scan from the input 1934 MS |
|
|
none |
Time request for splitting the calibrator MS |
|
true |
none |
Whether to flag the splitted-out 1934 MS |
|
|
none |
Time request for flagging the calibrator MS |
|
true |
none |
Whether to fit for the bandpass using all 1934-638 MSs |
|
|
none |
Time request for finding the bandpass solution |
|
false |
none |
Whether to ignore the fact that the weights ID for the bandpass and science observations may differ, and continue on with the pipeline processing. |
Preparing the calibrator datasets |
|||
|
1934.SB%s.%b.ms |
none |
Base name for the 1934 measurement sets after splitting. The wildcard %b will be replaced with the string “beamBB”, where BB is the (zero-based) beam number, and the %s will be replaced by the calibration scheduling block ID. |
|
|
Channel range for splitting (1-based!). This range also
defines the internal variable |
|
|
true |
none |
Whether to flag out a nominated range of channels in the
bandpass spectra to cover the known Milky Way HI
absorption. The range of channels is given by the
parameter |
|
15 |
none |
The maximum absolute velocity (in km/s, LSRK frame) to be flagged in the PKS B1934-638 spectrum. Flagging will be done from -vel to + vel. |
|
true |
none |
Whether to do the dynamic flagging, after the rule-based and simple flat-amplitude flagging is done. |
|
4.0 |
|
Dynamic threshold applied to amplitudes when flagging 1934 data [sigma] |
|
true |
amplitude_flagger.integrateSpectra (cflag / cflagger (Flagging Utility)) |
Whether to flag channels in the time-averaged spectra during the dynamic flagging task. |
|
4.0 |
amplitude_flagger.integrateSpectra.threshold (cflag / cflagger (Flagging Utility)) |
Dynamic threshold applied to amplitudes when flagging 1934 data in integrateSpectra mode [sigma] |
|
false |
amplitude_flagger.integrateTimes (cflag / cflagger (Flagging Utility)) |
Whether to flag time samples in the spectrally averaged time-series during the dynamic flagging task. |
|
4.0 |
amplitude_flagger.integrateTimes.threshold (cflag / cflagger (Flagging Utility)) |
Dynamic threshold applied to amplitudes when flagging 1934 data in integrateTimes mode [sigma] |
|
true |
none |
Whether to do Stokes-V flagging, after the rule-based and simple flat-amplitude flagging is done. |
|
0.0 |
|
Absolute threshold applied to amplitudes when flagging Stokes-V in 1934 data [Jy]. Disabled if set to zero. |
|
true |
stokesv_flagger.useRobustStatistics (cflag / cflagger (Flagging Utility)) |
Whether to use robust statistics (median and inter-quartile range) in computing the Stokes-V statistics. |
|
4.0 |
|
Threshold applied to amplitudes when flagging Stokes-V in 1934 data [sigma] |
|
true |
stokesv_flagger.integrateSpectra (cflag / cflagger (Flagging Utility)) |
Whether to flag channels in the time-averaged spectra during the Stokes-V flagging task. |
|
4.0 |
stokesv_flagger.integrateSpectra.threshold (cflag / cflagger (Flagging Utility)) |
Threshold applied to amplitudes when flagging Stokes-V in 1934 data in integrateSpectra mode [sigma] |
|
false |
stokesv_flagger.integrateTimes (cflag / cflagger (Flagging Utility)) |
Whether to flag time samples in the spectrally averaged time-series during the Stokes-V flagging task. |
|
4.0 |
stokesv_flagger.integrateTimes.threshold (cflag / cflagger (Flagging Utility)) |
Threshold applied to amplitudes when flagging Stokes-V in 1934 data in integrateTimes mode [sigma] |
|
false |
none |
Whether to apply a simple (“flat”) amplitude threshold to the 1934 data. |
|
0.2 |
amplitude_flagger.high (cflag / cflagger (Flagging Utility)) |
Simple amplitude threshold applied when flagging 1934
data.
If set to blank ( |
|
|
amplitude_flagger.low (cflag / cflagger (Flagging Utility)) |
Lower threshold for the simple amplitude flagging. If set
to blank ( |
|
|
selection_flagger.<rule>.antenna (cflag / cflagger (Flagging Utility)) |
Allows flagging of antennas or baselines. For example, to
flag out the 1-3 baseline, set this to |
|
|
selection_flagger.<rule>.spw (cflag / cflagger (Flagging Utility)) |
Allows flagging of a specified range of channels. For
example, to flag out the first 100 channnels, use
|
|
|
selection_flagger.<rule>.timerange (cflag / cflagger (Flagging Utility)) |
Allows flagging of a specified time range(s). The string
given is passed directly to the |
|
|
selection_flagger.<rule>.uvrange (cflag / cflagger (Flagging Utility)) |
Allows flagging of a specified UV range(s). The string
given is passed directly to the |
|
false |
selection_flagger.<rule>.autocorr |
If true, then autocorrelations will be flagged. |
|
|
timebegin (mssplit (Measurement Splitting/Averaging Utility)) |
The starting time for data to be used from the 1934 SB. If blank, the start of the observation will be used. The formatting must conform to requirements of mssplit (Measurement Splitting/Averaging Utility). |
|
|
The starting time for data to be used from the 1934 SB. If blank, the start of the observation will be used. The formatting must conform to requirements of mssplit (Measurement Splitting/Averaging Utility). |
|
|
true |
none |
Whether to run an additional flagging pass over the data, flagging any channels that have more than a given percentage flagged. |
|
60.0 |
none |
Flagging fraction above which a channel is flagged by the additional flagging pass. |
Using AOFlagger for flagging |
|||
|
false |
none |
Use AOFlagger for all flagging tasks in the pipeline. This overrides the individual task level switches. |
|
false |
none |
Use AOFlagger for the flagging of the bandpass calibrator. This allows differentiation between the different flagging tasks in the pipeline. |
|
|
none |
The strategy file to use for all AOFlagger tasks in the pipeline. Giving this a value will apply this one strategy file to all flagging jobs. The strategy file needs to be provided by the user. |
|
|
none |
The strategy file to be used for the bandpass
calibrator. This will be overridden by
|
|
true |
none |
Verbose output for AOFlagger |
|
auto |
none |
Read mode for AOflagger. This can take the value of one of “auto”, “direct”, “indirect”, or “memory”. These trigger the following respective command-line options for AOflagger: “-auto-read-mode”, “-direct-read”, “-indirect-read”, “-memory-read”. |
|
false |
none |
When true, the command-line argument “-uvw” is added to the AOFlagger command. This reads uvw values (some exotic strategies require these). |
Solving for the bandpass |
|||
|
|
sources.field1.direction (cbpcalibrator) |
Location of 1934-638, formatted for use in cbpcalibrator. |
|
calparameters.1934_bp.SB%s.tab |
calibaccess.table (cbpcalibrator and ccalapply (Calibration Applicator)) |
Name of the CASA table used for the bandpass calibration parameters. If no leading directory is given, the table will be put in the BPCAL directory. Otherwise, the table is left where it is (this allows the user to specify a previously-created table for use with the science field). The %s will be replaced by the calibration scheduling block ID. |
|
false |
calibrate.scalenoise (ccalapply (Calibration Applicator)) |
Whether the noise estimate will be scaled in accordance with the applied calibrator factor to achieve proper weighting. |
|
50 |
ncycles (cbpcalibrator) |
Number of cycles used in cbpcalibrator. |
|
216 |
none |
The number of cores allocated to the cbpcalibrator job. The job will use all 20 cores on each node (the memory footprint is small enough to allow this). |
|
200 |
MinUV (Data Selection) |
Minimum UV distance [m] applied to data prior to solving for the bandpass (used to exclude the short baselines). |
|
SVD |
solver (Calibration solvers) |
Selection of solver - either “SVD” or “LSQR” |
|
1000 |
solver (Calibration solvers) |
Damping parameter for “LSQR” solver. Ignored for SVD. |
|
1 |
refantenna (cbpcalibrator) |
Antenna number to be used as reference in the bandpass
calibration. Ignored if negative, or if provided as a
blank string ( |
Smoothing and plotting the bandpass |
|||
|
true |
none |
Whether to produce a smoothed version of the bandpass table, which will be applied to the science data. |
|
smooth_bandpass |
none |
Which tool to use. Possible values here are “smooth_bandpass” (the default) or “plot_caltable” Relevant parameters for each tool follow. |
|
|
none |
Integer value which if >0, the fitting is done within the
beam-forming intervals assumed to be a multiple of 54
channels. For eg., for 54-channel beamforming interval
set |
|
20 |
none |
The threshold, in multiples of ‘sigma’, that determines whether a given channel is flagged by the preflagger. |
smooth_bandpass |
Options for the script “smooth_bandpass.py” |
||
|
|
none |
The polynomial order for the fit - the value for the
|
|
|
none |
The harmonic order for the fit - the value for the
|
|
|
none |
The number of windows to divide the spectrum into for the
moving fit - the value for the |
|
|
none |
ASCII filename (with path) to manually specify the window
properties. Uncommented row nos. specify the window nos.,
and must have 2 colummns. The first column must specify
|
|
|
none |
The width (in channels) of the Gaussian Taper function to
remove high-frequency components - the value for the
|
|
|
none |
The number of iterations for Fourier-interpolation across
flagged points - the value for the |
|
|
none |
Provision to manually pass arguments to the bandpass smoothing tool of choice. If not empty, this string will override all parameters of the bandpass smoothing tool. |
plot_caltable |
Options for the script “plot_caltable.py” |
||
|
true |
none |
Whether to produce plots of the bandpass |
|
true |
none |
Whether to smooth the amplitudes (if false, smoothing is done on the real and imaginary values). |
|
true |
none |
If true, only smooth/interpolate over outlier points (based on the inter-quartile range). |
|
1 |
none |
The order of the polynomial (if >=0) or the window size (if <0) used in the smoothing. |
|
1.0 |
none |
The threshold level used for fitting to the bandpass. |
Applying the bandpass solution |
|||
|
true |
none |
Whether to apply the bandpass solution to the 1934 datasets. |
|
true |
none |
If true, the 1934 MSs will be copied prior to having the bandpass solution applied. This means you will have copies of both the raw and calibrated datasets. |
|
false |
usemedian (mssplit (Measurement Splitting/Averaging Utility)) |
If true, the channels are combined with median rather than average to form the continuum dataset. |
|
|
selection_flagger.<rule>.timerange (cflag / cflagger (Flagging Utility)) |
Allows flagging of a specified time range(s). The string
given is passed directly to the |
|
|
selection_flagger.<rule>.timerange (cflag / cflagger (Flagging Utility)) |
Allows flagging of a specified UV range(s). The string
given is passed directly to the |
|
0.0 |
|
Absolute threshold applied to amplitudes when flagging Stokes-V in averaged 1934 data [Jy]. Disabled if set to zero. |
Deriving leakage solutions |
|||
|
true |
none |
Whether to derive leakage solutions from the bandpass calibrated 1934 datasets. |
|
calparameters.1934_bpleakage.SB%s.tab |
calibaccess.table (cbpcalibrator and ccalapply (Calibration Applicator)) |
Name of the CASA table used for the leakage calibration
parameters. If no leading directory is given, the table
will be put in the BPCAL directory. Otherwise, the table
is left where it is (this allows the user to specify a
previously-created table for use with the science
field). The %s will be replaced by the calibration
scheduling block ID.
NB: This parameter is relevant for the case
|
|
true |
none |
If |
|
SVD |
solver (Calibration solvers) |
Selection of solver - either “SVD” or “LSQR”
NB: With |
|
|
solve (Calibration solvers) |
Use one of |
|
3e5 |
solver (Calibration solvers) |
Damping parameter for “LSQR” solver. Ignored for SVD. |
|
|
refantenna (cbpcalibrator) |
Antenna number to be used as reference in the leakage
calibration. Ignored if negative, or if provided as a
blank string ( |
|
false |
none |
Whether to derive leakage solutions for full-spectral-resolution data based on the leakage table derived for continuum-resolution. |
|
calparameters.1934_bpleakage_spec.SB%s.tab |
calibaccess.table (cbpcalibrator and ccalapply (Calibration Applicator)) |
Name of the CASA table used for the leakage calibration parameters. If no leading directory is given, the table will be put in the BPCAL directory. Otherwise, the table is left where it is (this allows the user to specify a previously-created table for use with the science field). The %s will be replaced by the calibration scheduling block ID. |
Smoothing the leakages |
|||
|
|
none |
The polynomial order for the fit - the value for the
|
|
|
none |
The harmonic order for the fit - the value for the
|
|
|
none |
The number of windows to divide the spectrum into for the
moving fit - the value for the |
|
|
none |
The width (in channels) of the Gaussian Taper function to
remove high-frequency components - the value for the
|
|
|
none |
The number of iterations for Fourier-interpolation across
flagged points - the value for the |
|
|
none |
Provision to manually pass arguments to the leakage smoothing tool. If not empty, this string will override all parameters of the leakage smoothing tool. |