ATNF VLBI Wiki

A synopsis of the main developments between DiFX-2.0 and the current release can be found here

This release will stem from the 2.8 branch which will originate from trunk in mid Aug 2022. A release planning page can be found here

• Python 3
• Early support for vex2
• Improved support for more than 2 bits per sample
• HOPS 3.24 (minor improvements)

This DiFX release is based on DiFX-2.6.2, extended by minimal code changes that enable Outputbands support. Includes an updated HOPS and PolConvert. Release tagged on 02 May 2022, and essentially frozen since 30 Jun 2022. The next step on this path will preferably be 2.8.2, but a 2.7.2 may be necessary to fix known bugs with 2.7.1.

• HOPS 3.23 vendor branch from r10257
• PolConvert updated to v1.8.4-like release
• m6support version 0.36 from r10260
• vex2difx outputband support (v2d SETUP section 'addOutputBand = freq@<skyfreq_USB_MHz>/bw@<bandwidth_MHz>')
• mpifxcorr outputband support (new .input file TARGET FREQ keyword under BASELINE)
• difx2fits/difio and difx2mark4 support for for .input file TARGET FREQ
• drivepolconvert.py detection of outputbands if present

• HOPS: same as for 3.23 in DiFX-2.6.3
• m6support contains support for NOEMA-style threads (threads comingled within write blocks)
• difxcalc ocean loading files updated for EHTC and GMVA
• PolConvert fix for NaN/Inf in visibility channels related to indexing error

• This release is primarily targeted for GMVA and EHTC correlation needs
• Certain DiFX-2.6 and Trunk features are purposely not yet included in this release. These will likely be added in DiFX-2.7.2, and include e.g. fixed LSB Complex data correlation, difx2fits additions to permit mixed-polarization output data especially to PIMA, PolConvert support for multi-datastream PCal records.

• Polconvert still has some issues to address
• vdifuse was modified to support Noema (and was broken in the process for some other cases)

This tag provides new features and some minor bug fixes

• HOPS updated to v 3.23
• mpifxcorr: new '–vgoscomplex' command line flag, flips sideband of *all* Complex VDIF stations by complex conjugating the unpacked Complex VDIF data
• …

• hops: minor bug fixes and enhancements (normalization fixes for notches and passband usage)
• Vex parser compile issues
• Handling of complex voltage data, Lower sideband specifically.

• …

This tag provides new features and some important bug fixes

• parseDiFX package added
• HOPS updated to v 3.21
• PolConvert updated to v 1.7.8
• Access to CDDIS servers changed to FTP-SSL (geteop.pl)

• mpifxcorr:
  • Fix correlation of Complex LSB data, restore fringes. Note: DiFX 2.5.x and 2.6.1 treated Complex LSB as if Complex USB, while Trunk prior to r9647 05aug2020 treated LSB nearly correctly except for a off-by-one channel bug.
• difx2mark4:
  • Fix seg-fault in createType3s.c when a station has only a single entry in the PCAL file
  • Remove unneeded debugging statement (calling d2m4_pcal_dump_record() )
  • Update createType3s.c to add support for DiFX PCAL files generated from station data where each data-stream thread resides in a separate file (multi-datastream support). This separates the code reading the PCAL files from the code filling the type-3 records so that tone records from multiple data streams can be merged before populating the type-309s.
  • Update createType3s.c to remove support for DiFX version-0 PCAL files
  • Add support for 10 MHz p-cal tone spacing (needed by VGOS at Yebes)
  • Significantly increase hardcoded array sizes (difx2mark4.h: NVRMAX 8M, MAX_FPPAIRS 10k, MAX_DFRQ 800) as required for EHT2018.
  • Fix a bounds check, permit tabs in VEX file
• difx2fits:
  • Fix FITS PH table having missing or superfluous pcal records when one correlates multi-datastream antennas, or not all recorded frequencies, or multiple zooms per recorded frequency
• mark6gather:
  • Fix poor weights in native Mark6 correlation for VDIF frame sizes not equal to 5032 bytes
• difxio:
  • Fix PCal tone frequency rounding bug on some platforms
  • Cope with recorded bands that lack PCal tones, e.g., 200 MHz PCal spacing of KVN with say 32 MHz recorded bands
• calc11:
  • Dave Gordon provided ocean loading params at EHT stations
  • Increased the number of field rows supported in .calc files
• vex2difx:
  • Fix internal merge of SamplingType (real, complex) when info found in VEX and/or v2d file
• other:
  • Minor changes to oms2v2d and vexpeek
  • More IPP versions supported
  • Minor issues with vis2screen fixed
  • Fixed build failure with gcc defaults
  • Python3 support in many/most places

• difx2mark4
  • Some LSB-LSB baselines do not get converted in mixed-sideband correlation setups (DiFX 2.6.1, 2.6.2); if affected, use difx2mark4 2.5.3 with –override-version. A bugfix is pending for DiFX 2.6.3 later this year.
  • Performance regression with p-cal files, conversion of p-cal data may take noticeably longer than before.
• calcserver and difxcalc11
  • With the latest versions of gfortran (10.1 or newer) you will need to uncomment the line with -fallow-argument-mismatch line in the environment setup in order to compile. Users who do this should be alert to possible issues.

Sections below summarize the major changes to the DiFX code since the DiFX 2.5 series. Please look at individual module ChangeLog files for the nitty gritty.

• Improved VDIF support
  • Increased robustness in processing VDIF data with many gaps
  • Improvements in processing VDIF with frame sizes very different from 5000 bytes
  • New in-line reordering functionality via vdifreader…() functions; allows operation on more highly skewed VDIF files
• mpifxcorr input, calc, threads, pulsar files are now only read by the head node, negating the need for these files to be visible to all nodes.
• mpifxcorr can be provided a new stop time via a DifxParameter message; results in clean shutdown at that time.
• mpifxcorr can extract pulse cals with tone spacing smaller than 1 MHz
• Support for Intel Performance Primitives version > 9 (specfically IPP 2018 and 2019)
  • These newer IPP versions are more readily available than earlier versions
• Improved support for Mark6 playback
  • Mark6 activity messages in difxmessage
  • Support in genmachines with updated mk5daemon
  • Support playback of Mark5B data on Mark6
  • New and improved mark6 utilities
• difx2fits: populate antenna diameters and mount types for antennas known to the difxio antenna database
• difx2fits: in verbose mode, explain why files are being split
• difx2fits: new options for merging correlator jobs run with different clock models
• vex2difx: new parameter “exhaustiveAutocorrs” can be used to generate cross-hand autocorrelations even when the two polarizations for an antenna come from different datastreams
• difx2mark4: support multiple bandwidths in one pass
• hops: to rev 3.19 (see notes on 2.5.3 below for details on several new and useful features)
• polconvert: to rev 1.7.5 (see notes on 2.5.3 below for details)

• mpifxcorr: Retry on NFS open errors of kind: “EAGAIN Resource temporarily unavailable”
• mpifxcorr: Fix weight issue when the parameter nBufferedFFTs > 1
• startdifx/genmachines: Fixes for cases when multiple input files are provided
• python 2 scripts now explicitly call python2
• vex2difx: allow up to 32 IFs (was 4) and warn when this is exceeded
• vex2difx: support units in the clock rate (e.g., usec/sec); in general support time in the numerators.
• Sun RPC is on its way out; support for “tirpc” added to calcif2 and calcserver

• Moved “mark6gather” functions from vdifio to mark6sg; this changes the order of dependencies!
• Various changes made between DiFX 2.5 and 2.6 are not API-compatible. Please don't mix packages from these two releases. If you have non-DiFX software that links against the DiFX libraries, be sure to recompile them. A small number of changes may result in need to restructure such code.
• There is some suspicion that correlation of very narrow bandwidth VDIF modes on Mark6 media can result in premature termination of datastreams.
• The .threads file must now exist; previously (before the change to only have manager read these files), a missing .threads file would cause each core process instance to have a single thread.
• “difx_monitor” won't compile with IPP >= 9

This is a minor bugfix release, primarily for (EU)VGOS.

• vex2difx
  • fix 'freqClockOffs=f1,f2,…' and 'loOffsets=f1,f2,…' so they do not expect more values to be provided than actual recorded frequencies present
  • removed obsolete warning about 10 MHz PCal not being supported
• genmachines corrected to support December i.e. doy>=335
• genmachines with support for Mark6 with multiple expansion chassis
• updates to PolConvert scripting

This is a bugfix release for VGOS. It also adds some feature enhancements from the DiFX 2.6 series.

• HOPS updated to 3.22
• Vex2difx and difxio
  • permit H and V polarization labels (but need trunk difx2fits if these are to be propagated into “enhanced” FITS-IDI)
  • support the v2d parameter 'exhaustiveAutocorrs'
  • support Mark6 MSNs i.e. MSNs that contain '%'
• Difx2mark4
  • support PCal data from multi-datastream correlation
  • option '-e <expt nr>' additionally propagates the experiment number into the generated root files
  • backported the options -w for mixed-bandwidth data, and -g for filtering freq groups
• Genmachines updated to support Mark6 host auto-detection
• Includes copies of more recent utilities: packHops.py, distFourfit.py, fplot2pdf, plotResiduals.py

• Mark6 native playback fixed to support other than VLBA VDIF recording
• Vdifio
  • fix excessively long station name printout in printVDIFheader
  • fix to mk6gather to not crop a scan
  • resync code extended with further VDIF frame sizes that are common in VGOS
• Mpifxcorr complex data (VDIFC) PCal extraction fix for bands with other than 16 tones
• Mpifxcorr fix for IVS-related 5/10MHz pcal issue specific to those channels where first pcal is at baseband 2.01 MHz
• Difx2mark4
  • fix count of total bands and polarizations for multi datastream datasets, solving an issue in HOPS post-processing
  • fix parallactic angle calculation error in first t303 record
  • fix to permit Tab characters in VEX
• Mpifxcorr no longer segfaults after “exiting gracefully” message upon missing data files
• Tkinter add-on package renamed to kinter_difx to avoid name collision
• install-difx now works under Python 3 and supports the option –withmark6meta
• startdifx no longer piles up difxlog processes

• Complex VDIF handling of 2.5.3 was deliberately not fixed in this release. Labeling VDIFC data as LSB is “fine” i.e. gets correlated as USB despite LSB label. One should just not correlate any actual VDIFC LSB recordings - this produces no fringes. The bug is retained for VGOS to have all-LSB(-labeled) data products, and avoid postprocessing issues with mixed USB and LSB data products.

This release is created to support EHTC and GMVA processing during the transition to 2.6.

• genmachines
  • r8264 genmachines and mark6 datastream updates
  • r8357 add mark6 activity message to mark6 datastream
  • r8409 allow multiple nodes to serve as datastream nodes for FILE based-data in the same location
• hops 3.19 new features
  • increased the number of allowed frequency “notches” to ridiculous levels
  • an “ad hoc” data flagging capability to allow improved time / channel data selection for fringing
  • a capability to dump all the information on the fringe plots into ascii files for “roll your own” plotting
  • removed obsolete max_parity
  • introduced min_weight (to discard APs with very little correlated data in support)
  • vex2xml, a program that converts VEX (v1.5) into XML to allow easy parsing via standard XML parsers.
  • added type_222 to save control file contents, enabled by keyword gen_cf_record.
• polconvert to v1.7.5 (mostly minor bug fixes and robustifications)

• Fixes for the HOPS 'rootid' rollover. The new rootcode is a conventional base-36 timestamp in seconds from the start of the new epoch (zzzzzz in the old epoch). This will last until 2087.
• Major fixes/improvements to PolConvert for use with ALMA by the EHTC and GMVA

• Initial support for correlating Mark6. This is still much a work in progress.
• Multiple datastreams per antenna supported via vex2difx
• New delay model program: difxcalc11. No longer requires calcserver.
• Support for more than 6 days of EOP values.
• “Union mode” in difx2fits allows merging of correlation output that uses different setups. Some restrictions apply. Designed for GMVA and RadioAstron use.
• Improved VDIF support: wider range of bits/threads, support for multi-channel, multi-thread VDIF, support for complex multi-thread VDIF
• Support for new VDIF Extended Data Version 4 which is useful for multiplexed VDIF data. See: http://vlbi.org/vdif/docs/edv4description.pdf
• Python bindings for vdifio and mark5access
• mpifxcorr: per-thread weights implemented
• Automatic selection of arraystride by mpifxcorr if set to zero; this is done per-datastream. Very useful for correlation of ALMA data or others with non-standard sample rates.
• Automatic selection of xmacstride by mpifxcorr if set to zero
• Automatic selection of guardns by mpifxcorr if set to zero
• mpifxcorr can now operate with unicast messages instead of multicast. Useful in some situations where multicast is not supported.
• New “dirlist” module/file directory listing format. * mk5cp append mode to resume interrupted copy
• ALMA support in HOPS: non-power-of-two FFTs, up to 64 freq. channels, full linear/circular/mixed polarization support
• HOPS improvemetns for VGOS through improved manual phase cal support
• New package: polconvert. Used to post-correlation convert from linear to circular polariations
• New package: autozoom. Helps a user develop .v2d file content when setting up complicated zoom band configurations.
• New package: datasim: generate baseband data suitable for simulated correlation
• Improved error reporting in many places

• fix for incorrect reporting of memory use by mpifxcorr (needed longer int sizes)
• dataweights would sometimes be incorrect after abrupt ending of data from a datastream.
• FITS-IDI files produced by difx2fits more standards compliant; fix problem that caused AIPS task VBGLU to fail.
• Several segfaults across a number of programs/utils now are caught and provide useful feedback.

• Various changes made between DiFX 2.4 and 2.5 are not API-compatible. Please don't mix packages from these two releases. If you have non-DiFX software that links against the DiFX libraries, be sure to recompile them. A small number of changes may result in need to restructure such code.

• A “jitter” of 0.5 ns when using 2Gbps Mark5B format was fixed. A fix was back-ported to DiFX 2.3.
  1. A similar jitter was corrected for high frame rate VDIF (problem identified by the VLITE project)
• Fix case of intermittant fringes that was due to incorrect assumption about the sizeof(unsigned long):
  1. 32 bits on a 32-bit system vs. 64 bits on a 64-bit system
  2. Some other variable types were changed for long term type safety
• Fix off-by-one in correlation using LSB and zoom bands together.

• Support a FAKE correlation mode for multi-threaded VDIF.
• The mpifxcorr produced PCAL files have had a format change that allows unambiguous interpretation across all use cases.
• Add network support (TCP, UDP and Raw Ethernet) for multi-threaded VDIF.
  1. TCP and UDP variants not tested yet
  2. Raw Ethernet variant is used for the VLITE project
• Support updated Mark5 module directories.
• Better checking that Mark5 data being processed matches what is expected.
• Improved Mark5B decoding:
  1. Mark5B data streams are now filtered for extra or missing data
  2. Packets with invalid bit (actually the TVG bit) set replace missing data
  3. This means any valid Mark5B data with the TVG bit set will not correlate
• Information about each Mark5 unit used in “native mode” is emitted at start of jobs so it can be logged.
• Ultra-low frame rate VDIF data was affected by allowing a long “sort window” in the VDIF multiplexer. This has been reduced to 32 frames and seems to work fine for all bandwidths now.

• Slightly improved compliance with the FITS-IDI convention:
  1. Invalid Tsys values become NaN, not 999.
  2. Populate DELTAT keyword in ModelComps table.

• Support for X/Y polarization correlation. Many fundamental issues with linear polarization remain though:
  1. This does not support in a meaningful way Linear*Circular correlations
  2. There is a terminology gap in many bits of software and file formats that confuses X/Y with H/V polarization bases
  3. The intent of this support is for short baselines (VLITE)

• Support for “d2k” mode in Mark5B format (swapped sign and mag bits).
• fixmark5b() function fixed for case that fill pattern is seen at the 1 second transition.
• Make use of the TVG bit as an “invalid frame” indicator for Mark5B data.
• m5bstate: support complex sampled data

• New utility mk5putdir: reads a binary file and replaces a Mark5 directory with it.
• mk5dir: when reading the directories, saves a copy of the binary representation in case it is needed later (perhaps via mk5putdir)
• Reworked mark5 module directory support, including support for many new variants of the directory format.
• mk5erase will save a “conditioning report” to $MARK5_CONDITION_PATH if that environment variable is set.

• Fairly large change to the API. Please read the ChangeLog for details.

• Respect the record enable bit in the SCHED block. If that value is 0 no correlation will be attempted for that antenna.
• Bug fixes preventing some LSB/zoom bands from being correlated.
• Complex data type and # of bits are now read from vex file.

• mpifxcorr: Datasteam buffer send size now calculated correctly for complex sampled data
• mpifxcorr: Avoid very rare bug where combination of geometric delay and data commencing mid-subint meant one invalid FFT might be computed
• mpifxcorr: multicast weights are now computed correctly for mixed-sideband correlation
• mpifxcorr: fixed bug where some autocorrelations were not saved in a mixed-sideband correlation
• mpifxcorr: fixed bug where send size could be computed incorrectly by 1-2 bytes for Mark4/VLBA/Mark5B/VDIF formats, potentially resulting in very small amounts of data loss

• mpifxcorr: LO offsets are now corrected in the time domain when fringe rotation is also done in the time domain (the usual mode), allowing considerably larger LO offsets without decorrelation
• mpifxcorr: Working polarization dependent delay and phase offsets
• mpifxcorr: Experimental linear2circular conversion
• mpifxcorr: Complex Double sideband (RDBE/Xcube) sampling support
• mpifxcorr: new file/mk5 based vdif/mark5b datastream (faster and more robust)
• mark5access: implement work-around for buggy kernel-driver combinations; Mark5 read sizes >20 MB now allowed
• utilities: some new command line tools for mk5b and vdof files (vsum, mk5bsum, vmux, mk5bfix)
• all packages: mk5 support for mark5a/b with sdk9
• new options for passing calibration (Walter B: memo forthcoming)

• Hops updated to version 3.9

• calcif2: fix azimuth polynomial generation in case of wrap
• fix for FITS file generation for mixed sideband correlation
• difx2fits now uses appropriate gain tables for S and X band in S/X experiments (Thanks to James Miller-Jones for reporting)
• difx2fits: correct pcal, weather, tsys and flag data for observations crossing new year
• vex2difx: check for numerical precision error in source RA/dec coordinates to avoid creating two copies of a single source
• fixed scaling of autocorrelations for LBA format data
• 0.5 ns wobble in delays for 2 Gbps Mark5B data fixed
• Fix bug preventing subintegrations longer than 1 second. Now 2 seconds is allowed (this limit comes from signed integer number of nanoseconds).
• Weights corrected in cases where two setups differening only by pcal setup were correlated against each other
• Quashed data and weight echos that would occur for about 1 integration at the beginning of each scan for datstreams that ran out of data before end of job.
• The multicast (diagnostic) weights were low or zero in case of frequency selection (zoom band or freqId selection). Fixed.
• Mutex locking conflict for very short jobs fixed.
• MODE errors no longer generated if a datastream runs out of data a long time before the job finishes.
• For module-based playback of VDIF data, a rounding error could lead to wrong scan selection. Fixed.
• difx2mark4 now correctly handles band-matching and zoom bands - several bugfixes were made.

• calcif2: ability to estimate delay polynomial interpolation errors
• support for a “label” identifier for a local version of DiFX that will help discriminate exact version used.
• faster Mark5 directory reading, for VDIF the threads are saved in the .dir file
• Faster VDIF corner turning through customized bit shifting functions
• mpifxcorr can now be built without Intel Integrated Performance Primitives, though resulting in a slower correlator.
• vdifio: several new VDIF manipulation and processing utilities added: vmux, vsum, vdifd, vdifspec, vdiffold, vdifbstate. Also bug fixes in multi2singlethreadVDIF to cope with large amounts of packet loss.
• difx2fits allows more phase cal tones
• vex2difx allows different start and stop times per antenna
• difxfilterbank now uses plplot instead of pgplot

• VDIF correlation fails when there is a gap in the baseband data (usually vex2difx's singleScan=true will overcome this)
• VDIF corner turning is still a bit slow and will limit performance on Mark5 units

• mpifxcorr: fix a scaling issue with autocorrelations of LBA-format data in mpifxcorr. Does not affect Mark4, VLBA, Mark5B or any other format.

With DiFX 2.1 comes a new version numbering scheme, designed to allow small updates “in-place”, reducing the lag between getting important bugfixes into production correlation. The collection of libraries, utilities and applications comprising the DiFX system will be given a 2-part version number with a major release. This version number (currently DiFX-2.1) refers to a particular set of versioned components. Incremental improvements to components can now be made along the way. If a minor and necessary change for a given component becomes available, this can be committed to the component within the DiFX-2.1 tag. When this occurs, both the component itself and the “whole DiFX” version acquire a third digit, indicating a revision number. This digit would then be incremented with each additional incremental revision. These minor releases should be restricted to bug fixes or new features that are both significant and deemed very unlikely to affect other portions of code. Importantly, binary compatibility of components should not change within a major (2-part version) DiFX release.

• mpifxcorr: LBA format data now scaled roughly correctly (removing the need for large ACCOR corrections).
• mpifxcorr: There was a bug when xmaclength was > nfftchan for pulsar processing. This has been corrected.
• mpifxcorr: guardns was incorrectly (overzealously) calculated.
• mpifxcorr: Mk5DataStream::calculateControlParams: bufferindex>=bufferbytes bug fixed.
• mpifxcorr: Low weight reads could result in uninitialized memory; fixed.
• mpifxcorr: Streamstor XLRRead() bug work-around installed several places (read at position 0 before reading at position > 0). This is thought not to be needed with Conduant SDK 9.2 but the work-around has no performance impact.
• mpifxcorr: Fix to pulse calibration data ordering for LSB or reordered channels.
• mpifxcorr: Pulse Cal amplitude now divided by pulse cal averaging time in seconds.
• mpifxcorr: Pulse cal system would cause crash if no tones in narrow channel. Fixed.
• vex2difx: Zoom band support across mixed bandwidths (see caveat below).
• mpifxcorr: Fix for spurious weights at end of jobs (untested…)
• mpifxcorr: Mixed 1 and 2 bit data are handled more cleanly
• mpifxcorr: now terminates correctly for all short jobs. Previously it hung for jobs with a number of subints between nCores and 4 x nCores
• mpifxcorr: Correctly scale cross-correlation amplitudes for pulsar binning when using Tsys>0 (accounts for varying # samples per bin c.f. nominal)
• mpifxcorr: Lower side-band pulse cal tones had sign error. Fixed.

• mpifxcorr: during Mark5-based correlation, emit multicast message containing drive statistics after each scan
• mpifxcorr: Support for non power-of-2 FFT lengths
• mpifxcorr: Multithread VDIF support enabled for the data sources FILE and MODULE, including stripping of non-VDIF packets
• mpifxcorr/startdifx: Restart of correlation is now possible
• mpifxcorr/vex2difx: new FAKE datastream type for performance testing
• New utilities:
  • mk5map (limited functionality)
  • fileto5c
  • record5c
  • getsmart: during Mark5-based correlation, allows easy access to SMART data
• mk5daemon: VSIS interface added
• mk5daemon: Remote running of vex2difx enabled
• New features added to existing utilities:
  • mk5cp: copy without reference to a module directory
  • mk5cp: ability to send data over ssh connection
  • vsn: get SMART data from disk drives
• e-Control source code analysis (Martin Ettl, Wettzell)
• difx2fits: -0 option to write minimal number of visibilities to FITS
• difx2fits: write new RAOBS, DECOBS columns in source table
• difx2fits: no longer copies autocorrelations to output FITS files except for phase centre 0 and pulsar bin 0 unless the -a option is used.
• vex2difx: tweakIntTime has been re-enabled
• diffDiFX.py can now cope with two files that don't have exactly the same visibilities (i.e., some visibilities are missing from one file)
• plotDiFX.py and plotDynamicSpectrum.py now have better plotting and more options
• Espresso, a lightweight system for managing disk-based correlation, has been added to the DiFX repository.
• vex2difx: Option to correlate only one polarization has been added.
• mk5dir can now produce .dir file information for VDIF formatted data.
• Add NRAO's sniffer plotters to the repository.

• vex2difx: To support zoom bands across mixed bandwidths (and other unnusual modes), “number of channels” becomes a poorly defined quantity. There is a general change to use spectral resolution rather than number of channels when specifying correlation parameters. See the documentation at http://cira.ivec.org/dokuwiki/doku.php/difx/vex2difx. nChan still should work for simple cases.
• vex2difx: There is currently no clean way to specify different zoom bands for different observing modes in one .v2d files
• difx2fits: When running difx2fits on jobs that were restarted mid-way, the oldest data is used preferentially over newer data. It would make sense someday to change that.
• mpifxcorr: It has been observed in some cases (believed to be restricted to very high output data rates) that visibilities start getting dropped, likely as a result of the visibility buffer filling up. Increasing the size of that buffer and perhaps using single-scan jobs limits the problems.

• difx2fits: Fix model accountability bug when combining jobs
• mpifxcorr, vex2difx, difxio, difx2fits: Numerous fixes for zoom bands
• mpifxcorr: Native Mark5 has improved stability for cranky modules
• difx2fits (primarily) + mpifxcorr: Numerous fixes for DiFX-based phase cal extraction (mostly for multi-job)
• mpifxcorr: Fractional bit correction for lower sideband got broken in difx 2.0.0. Fixed.
• difxcalculator: Migrated to DiFX 2; was not complete for DiFX 2.0.0
• mpifxcorr: Clock offsets > 1/2 second were not handled properly for Mark5 module playback due to ignoring of intclockseconds
• vex2difx: no longer rounds down to zero the subintegration time when adjustment is needed for the highest spectral resolutions
• mpifxcorr: The amount of extra data sent to the cores to account for geometric delays was incorrectly scaled by number of channels. Fixed.

• new utility checkmpifxcorr (mpifxcorr/utils)
• mpifxcorr: Switched power detection
• mpifxcorr: Improvements to support infiniband speeds with native Mark5
• mpifxcorr: Early multi-thread VDIF format support
• multiple: RedHat RPM file generation for some packages (can extend to others on request)
• difx2fits: Improvements to method of selecting which pulse cal tones get propagated to FITS
• mpifxcorr: Initial complex sampling support
• mk5dir: now sorts by scan time (useful for modules assembled from e-transfered data)
• all: Support underscores in source names

• The sign of DiFX-extracted pcal phase has swapped since DiFX 2.0.0
• While supported at some level, support for decimation of oversampled data is not documented and is not known to work correctly; use at your own risk
• Default tones to put in the FITS file are the two tones in each sub-band that are nearest the edge, but if possible not those within the edge 1/8 of the band.
• vex2difx SETUP parameter nChan changed to refer to the number of output channels; nFFTChan replaces functionality of specAvg
• This release includes version 3.4 of the Haystack Observatory Postprocessing System (HOPS) for analyzing DiFX output that has been converted to the Mark4 correlator format with difx2mark. For more details, consult the documentation pages:
  • http://cira.ivec.org/dokuwiki/doku.php/difx/difx2mark4
  • http://cira.ivec.org/dokuwiki/doku.php/difx/hops

DiFX2.0 was the first version of a completely rewritten DiFX architecture. (Previous versions were DiFX-1.5.0 to 1.5.4, covering the period 2007-2010). The feature list below is not complete - someone would need to go through the individual change logs of all packages to gather all of the changes - there were very many. Below are listed the most important.

• mpifxcorr: Big speed improvements due to code improvements for cache management and some smarter algorithms: ~20% for “normal” observations, much more for high spectral resolution.
• mpifxcorr: Multiple phase centres in one pass
• mpifxcorr: “Zoom” bands (spectral selection)
• mpifxcorr: Production of spectral kurtosis estimates in additional to higher, configurable time resolution filterbank data
• mpifxcorr: LO offset correction
• mpifxcorr: More efficient writing of visibilities enabling higher sustained write speeds and smaller output files
• mpifxcorr/vex2difx: More finely grained control of the data to be correlated using “Rule” table in input file
• mpifxcorr: mostly complete Mark5B support
• mpifxcorr: Mark5B eVLBI support
• Allow for new VEX FILE line in calc file to be optional

• mpifxcorr: visibilities with zero weight are not written to disk
• mpifxcorr: polyco parser and checker is more forgiving of TEMPO2 generated polycos
• mpifxcorr: Catch rare datastream error where file ends exactly at end of databuffer for Mk5 files
• mpifxcorr: polycos with more than one entry now parsed correctly