User Tools

Site Tools


How to achieve very high spectral resolution

DiFX can support a wide range of spectral resolutions. Achieving the highest resolutions can be tricky for a number of reasons, many with adverse performance side effects:

  • Typically transform sizes get large
  • Lots of memory is needed / cache memory becomes less effective
  • Bandwidth.time product becomes small (as low as 1), meaning less reduction of data
    • Output data could be larger than input data!
  • Individual FFTs become long (in time), butting up against
  • Due to timekeeping mechanisms in mpifxcorr, an FFT cannot span more than 2^31-1 nanoseconds
    • A hair over 2 seconds is the longest allowed transform
    • 0.5 Hz is thus the finest frequency resolution allowed

Some general tips to consider:

  • Start with small input bandwidths (i.e., plan the experiment properly!)
  • Use factors of 10 in FFT spectral resolution, rather than only factors of 2
    • E.g., set spectral resolution (specRes and fftSpecRes) to 0.001 (MHz) rather than 0.0009765625 (MHz)
  • Make sure integration time (tInt) multiplied by specRes is not less than 0.0000001
  • Make use of zoom band to only save part of the spectrum, if consistent with the science
  • To minimize memory usage, numBufferedFFTs should probably be set to 1 (may greatly hurt performance)

Some example .v2d file content leading to 0.5 Hz resolution:

# Might need to increase the largest read size to accomodate long FFTs
maxReadSize = 40000000
# Correlate a 1 kHz portion of the input band centered on 4101.3 MHz
ANTENNA FD { toneSelection=none filelist=fd.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
ANTENNA KP { toneSelection=none filelist=kp.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
ANTENNA NL { toneSelection=none filelist=nl.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
ANTENNA OV { toneSelection=none filelist=ov.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
ANTENNA PT { toneSelection=none filelist=pt.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
SETUP default
  tInt = 2 # this is the shortest integration allowed for half Hz
  fftSpecRes = 0.0000005
  specRes    = 0.0000005
  doPolar = False
  numBufferedFFTs = 1 # When 1 FFT per integration, this is the only legal value
  maxNSBetweenACAvg = 2000000000 # change to be integration time; avoid warnings

Note that the above 5 station correlation of a pair of 4 MHz channels requires 12 GB of RAM for each core process.

difx/highspecres.txt · Last modified: 2021/08/12 18:24 by cormac