User Tools

Site Tools


difx:diskspace

How much disk space do I need?

There are many parameters that go into setting up a correlation, and some combinations can generate ridiculously large amounts of output data. Assuming all scans are set up the same, then a per-scan formula for space needed, X (in bytes) is:

  X = M * (B / R) * T * K * P * V
  
  T : number of integrations
  N : stations with good data
  K : number of baselines: N*(N-1)/2 or N*(N+1)/2 if cross-pol autocorrelations are requested
  R : (output) spectral resolution
  B : total bandwidth (adding up all channels)
  P : number of polarization products (1, 2 or 4)
  V : bytes per visibility: DiFX uses a complex32 type, //i.e.// 2 32-bit floats = 8 (bytes)
  M : a multiplier for whatever else you do with the basic scan (multiple phase centers and output products)

M includes overheads of various sorts (headers, &c) of a few percent, and there might be data losses (again at most of few percent). And depending on the processing you plan, you then may need to multiply this further for additional copies of the data packaged in multiple ways. (E.g. FITS and HOPS both have the same number of result bits as the original SWIN output…but the packaging is different.)

Some experiments have phase cal data that scales with N * T and the details of the tone setup, but that is usually a minor consideration.

And there are other ways to make this less simple.

difx/diskspace.txt · Last modified: 2021/12/22 01:21 by geoffcrew