User Tools

Site Tools


difx:changelog

DiFX 2.6.1 (Released Aug 28, 2019)

Sections below summarize the major changes to the DiFX code since the DiFX 2.5 series. Please look at individual module ChangeLog files for the nitty gritty.

Major new features

  • Improved VDIF support
    • Increased robustness in processing VDIF data with many gaps
    • Improvements in processing VDIF with frame sizes very different from 5000 bytes
    • New in-line reordering functionality via vdifreader…() functions; allows operation on more highly skewed VDIF files
  • mpifxcorr input, calc, threads, pulsar files are now only read by the head node, negating the need for these files to be visible to all nodes.
  • mpifxcorr can be provided a new stop time via a DifxParameter message; results in clean shutdown at that time.
  • mpifxcorr can extract pulse cals with tone spacing smaller than 1 MHz
  • Support for Intel Performance Primitives version > 9 (specfically IPP 2018 and 2019)
    • These newer IPP versions are more readily available than earlier versions
  • Improved support for Mark6 playback
    • Mark6 activity messages in difxmessage
    • Support in genmachines with updated mk5daemon
    • Support playback of Mark5B data on Mark6
    • New and improved mark6 utilities
  • difx2fits: populate antenna diameters and mount types for antennas known to the difxio antenna database
  • difx2fits: in verbose mode, explain why files are being split
  • difx2fits: new options for merging correlator jobs run with different clock models
  • vex2difx: new parameter “exhaustiveAutocorrs” can be used to generate cross-hand autocorrelations even when the two polarizations for an antenna come from different datastreams
  • difx2mark4: support multiple bandwidths in one pass
  • hops: to rev 3.19 (see notes on 2.5.3 below for details on several new and useful features)
  • polconvert: to rev 1.7.5 (see notes on 2.5.3 below for details)

Bug fixes

  • mpifxcorr: Retry on NFS open errors of kind: “EAGAIN Resource temporarily unavailable”
  • mpifxcorr: Fix weight issue when the parameter nBufferedFFTs > 1
  • startdifx/genmachines: Fixes for cases when multiple input files are provided
  • python 2 scripts now explicitly call python2
  • vex2difx: allow up to 32 IFs (was 4) and warn when this is exceeded
  • vex2difx: support units in the clock rate (e.g., usec/sec); in general support time in the numerators.
  • Sun RPC is on its way out; support for “tirpc” added to calcif2 and calcserver

Caveats

  • Moved “mark6gather” functions from vdifio to mark6sg; this changes the order of dependencies!
  • Various changes made between DiFX 2.5 and 2.6 are not API-compatible. Please don't mix packages from these two releases. If you have non-DiFX software that links against the DiFX libraries, be sure to recompile them. A small number of changes may result in need to restructure such code.
  • There is some suspicion that correlation of very narrow bandwidth VDIF modes on Mark6 media can result in premature termination of datastreams.
  • The .threads file must now exist; previously (before the change to only have manager read these files), a missing .threads file would cause each core process instance to have a single thread.
  • “difx_monitor” won't compile with IPP >= 9

DiFX 2.5.3 (released March 28, 2019)

This release is created to support EHTC and GMVA processing during the transition to 2.6.

updates
  • genmachines
    • r8264 genmachines and mark6 datastream updates
    • r8357 add mark6 activity message to mark6 datastream
    • r8409 allow multiple nodes to serve as datastream nodes for FILE based-data in the same location
  • hops 3.19 new features
    • increased the number of allowed frequency “notches” to ridiculous levels
    • an “ad hoc” data flagging capability to allow improved time / channel data selection for fringing
    • a capability to dump all the information on the fringe plots into ascii files for “roll your own” plotting
    • removed obsolete max_parity
    • introduced min_weight (to discard APs with very little correlated data in support)
    • vex2xml, a program that converts VEX (v1.5) into XML to allow easy parsing via standard XML parsers.
    • added type_222 to save control file contents, enabled by keyword gen_cf_record.
  • polconvert to v1.7.5 (mostly minor bug fixes and robustifications)

DiFX 2.5.2 (released Mar 6, 2018)

Major new features

  • Fixes for the HOPS 'rootid' rollover. The new rootcode is a conventional base-36 timestamp in seconds from the start of the new epoch (zzzzzz in the old epoch). This will last until 2087.
  • Major fixes/improvements to PolConvert for use with ALMA by the EHTC and GMVA

DiFX 2.5.1 (released Sep 25, 2017)

Major new features

  • Initial support for correlating Mark6. This is still much a work in progress.
  • Multiple datastreams per antenna supported via vex2difx
  • New delay model program: difxcalc11. No longer requires calcserver.
  • Support for more than 6 days of EOP values.
  • “Union mode” in difx2fits allows merging of correlation output that uses different setups. Some restrictions apply. Designed for GMVA and RadioAstron use.
  • Improved VDIF support: wider range of bits/threads, support for multi-channel, multi-thread VDIF, support for complex multi-thread VDIF
  • Support for new VDIF Extended Data Version 4 which is useful for multiplexed VDIF data. See: http://vlbi.org/vdif/docs/edv4description.pdf
  • Python bindings for vdifio and mark5access
  • mpifxcorr: per-thread weights implemented
  • Automatic selection of arraystride by mpifxcorr if set to zero; this is done per-datastream. Very useful for correlation of ALMA data or others with non-standard sample rates.
  • Automatic selection of xmacstride by mpifxcorr if set to zero
  • Automatic selection of guardns by mpifxcorr if set to zero
  • mpifxcorr can now operate with unicast messages instead of multicast. Useful in some situations where multicast is not supported.
  • New “dirlist” module/file directory listing format. * mk5cp append mode to resume interrupted copy
  • ALMA support in HOPS: non-power-of-two FFTs, up to 64 freq. channels, full linear/circular/mixed polarization support
  • HOPS improvemetns for VGOS through improved manual phase cal support
  • New package: polconvert. Used to post-correlation convert from linear to circular polariations
  • New package: autozoom. Helps a user develop .v2d file content when setting up complicated zoom band configurations.
  • New package: datasim: generate baseband data suitable for simulated correlation
  • Improved error reporting in many places

Bug fixes

  • fix for incorrect reporting of memory use by mpifxcorr (needed longer int sizes)
  • dataweights would sometimes be incorrect after abrupt ending of data from a datastream.
  • FITS-IDI files produced by difx2fits more standards compliant; fix problem that caused AIPS task VBGLU to fail.
  • Several segfaults across a number of programs/utils now are caught and provide useful feedback.

Caveats

  • Various changes made between DiFX 2.4 and 2.5 are not API-compatible. Please don't mix packages from these two releases. If you have non-DiFX software that links against the DiFX libraries, be sure to recompile them. A small number of changes may result in need to restructure such code.

DiFX 2.4 (released Mar 31, 2015)

Major new features and bug fixes in 2.4

  • A “jitter” of 0.5 ns when using 2Gbps Mark5B format was fixed. A fix was back-ported to DiFX 2.3.
    1. A similar jitter was corrected for high frame rate VDIF (problem identified by the VLITE project)
  • Fix case of intermittant fringes that was due to incorrect assumption about the sizeof(unsigned long):
    1. 32 bits on a 32-bit system vs. 64 bits on a 64-bit system
    2. Some other variable types were changed for long term type safety
  • Fix off-by-one in correlation using LSB and zoom bands together.

Other noteworthy changes in DiFX 2.4

In mpifxcorr
  • Support a FAKE correlation mode for multi-threaded VDIF.
  • The mpifxcorr produced PCAL files have had a format change that allows unambiguous interpretation across all use cases.
  • Add network support (TCP, UDP and Raw Ethernet) for multi-threaded VDIF.
    1. TCP and UDP variants not tested yet
    2. Raw Ethernet variant is used for the VLITE project
  • Support updated Mark5 module directories.
  • Better checking that Mark5 data being processed matches what is expected.
  • Improved Mark5B decoding:
    1. Mark5B data streams are now filtered for extra or missing data
    2. Packets with invalid bit (actually the TVG bit) set replace missing data
    3. This means any valid Mark5B data with the TVG bit set will not correlate
  • Information about each Mark5 unit used in “native mode” is emitted at start of jobs so it can be logged.
  • Ultra-low frame rate VDIF data was affected by allowing a long “sort window” in the VDIF multiplexer. This has been reduced to 32 frames and seems to work fine for all bandwidths now.
In difx2fits
  • Slightly improved compliance with the FITS-IDI convention:
    1. Invalid Tsys values become NaN, not 999.
    2. Populate DELTAT keyword in ModelComps table.
In difxio
  • Support for X/Y polarization correlation. Many fundamental issues with linear polarization remain though:
    1. This does not support in a meaningful way Linear*Circular correlations
    2. There is a terminology gap in many bits of software and file formats that confuses X/Y with H/V polarization bases
    3. The intent of this support is for short baselines (VLITE)
In mark5access
  • Support for “d2k” mode in Mark5B format (swapped sign and mag bits).
  • fixmark5b() function fixed for case that fill pattern is seen at the 1 second transition.
  • Make use of the TVG bit as an “invalid frame” indicator for Mark5B data.
  • m5bstate: support complex sampled data
In mk5daemon
  • New utility mk5putdir: reads a binary file and replaces a Mark5 directory with it.
  • mk5dir: when reading the directories, saves a copy of the binary representation in case it is needed later (perhaps via mk5putdir)
  • Reworked mark5 module directory support, including support for many new variants of the directory format.
  • mk5erase will save a “conditioning report” to $MARK5_CONDITION_PATH if that environment variable is set.
In vdifio
  • Fairly large change to the API. Please read the ChangeLog for details.
In vex2difx
  • Respect the record enable bit in the SCHED block. If that value is 0 no correlation will be attempted for that antenna.
  • Bug fixes preventing some LSB/zoom bands from being correlated.
  • Complex data type and # of bits are now read from vex file.

DiFX 2.3 (released Jan 18, 2014)

Bug fixes

  • mpifxcorr: Datasteam buffer send size now calculated correctly for complex sampled data
  • mpifxcorr: Avoid very rare bug where combination of geometric delay and data commencing mid-subint meant one invalid FFT might be computed
  • mpifxcorr: multicast weights are now computed correctly for mixed-sideband correlation
  • mpifxcorr: fixed bug where some autocorrelations were not saved in a mixed-sideband correlation
  • mpifxcorr: fixed bug where send size could be computed incorrectly by 1-2 bytes for Mark4/VLBA/Mark5B/VDIF formats, potentially resulting in very small amounts of data loss

New features

  • mpifxcorr: LO offsets are now corrected in the time domain when fringe rotation is also done in the time domain (the usual mode), allowing considerably larger LO offsets without decorrelation
  • mpifxcorr: Working polarization dependent delay and phase offsets
  • mpifxcorr: Experimental linear2circular conversion
  • mpifxcorr: Complex Double sideband (RDBE/Xcube) sampling support
  • mpifxcorr: new file/mk5 based vdif/mark5b datastream (faster and more robust)
  • mark5access: implement work-around for buggy kernel-driver combinations; Mark5 read sizes >20 MB now allowed
  • utilities: some new command line tools for mk5b and vdof files (vsum, mk5bsum, vmux, mk5bfix)
  • all packages: mk5 support for mark5a/b with sdk9
  • new options for passing calibration (Walter B: memo forthcoming)

Other news

  • Hops updated to version 3.9

DiFX 2.2 (released Jun 12, 2013)

Bug fixes

  • calcif2: fix azimuth polynomial generation in case of wrap
  • fix for FITS file generation for mixed sideband correlation
  • difx2fits now uses appropriate gain tables for S and X band in S/X experiments (Thanks to James Miller-Jones for reporting)
  • difx2fits: correct pcal, weather, tsys and flag data for observations crossing new year
  • vex2difx: check for numerical precision error in source RA/dec coordinates to avoid creating two copies of a single source
  • fixed scaling of autocorrelations for LBA format data
  • 0.5 ns wobble in delays for 2 Gbps Mark5B data fixed
  • Fix bug preventing subintegrations longer than 1 second. Now 2 seconds is allowed (this limit comes from signed integer number of nanoseconds).
  • Weights corrected in cases where two setups differening only by pcal setup were correlated against each other
  • Quashed data and weight echos that would occur for about 1 integration at the beginning of each scan for datstreams that ran out of data before end of job.
  • The multicast (diagnostic) weights were low or zero in case of frequency selection (zoom band or freqId selection). Fixed.
  • Mutex locking conflict for very short jobs fixed.
  • MODE errors no longer generated if a datastream runs out of data a long time before the job finishes.
  • For module-based playback of VDIF data, a rounding error could lead to wrong scan selection. Fixed.
  • difx2mark4 now correctly handles band-matching and zoom bands - several bugfixes were made.

New features

  • calcif2: ability to estimate delay polynomial interpolation errors
  • support for a “label” identifier for a local version of DiFX that will help discriminate exact version used.
  • faster Mark5 directory reading, for VDIF the threads are saved in the .dir file
  • Faster VDIF corner turning through customized bit shifting functions
  • mpifxcorr can now be built without Intel Integrated Performance Primitives, though resulting in a slower correlator.
  • vdifio: several new VDIF manipulation and processing utilities added: vmux, vsum, vdifd, vdifspec, vdiffold, vdifbstate. Also bug fixes in multi2singlethreadVDIF to cope with large amounts of packet loss.
  • difx2fits allows more phase cal tones
  • vex2difx allows different start and stop times per antenna
  • difxfilterbank now uses plplot instead of pgplot

Known issues

  • VDIF correlation fails when there is a gap in the baseband data (usually vex2difx's singleScan=true will overcome this)
  • VDIF corner turning is still a bit slow and will limit performance on Mark5 units

DiFX 2.1.1 (released Jun 07, 2012)

Bug fixes

  • mpifxcorr: fix a scaling issue with autocorrelations of LBA-format data in mpifxcorr. Does not affect Mark4, VLBA, Mark5B or any other format.

DiFX 2.1 (released May 25, 2012)

Version numbering change

With DiFX 2.1 comes a new version numbering scheme, designed to allow small updates “in-place”, reducing the lag between getting important bugfixes into production correlation. The collection of libraries, utilities and applications comprising the DiFX system will be given a 2-part version number with a major release. This version number (currently DiFX-2.1) refers to a particular set of versioned components. Incremental improvements to components can now be made along the way. If a minor and necessary change for a given component becomes available, this can be committed to the component within the DiFX-2.1 tag. When this occurs, both the component itself and the “whole DiFX” version acquire a third digit, indicating a revision number. This digit would then be incremented with each additional incremental revision. These minor releases should be restricted to bug fixes or new features that are both significant and deemed very unlikely to affect other portions of code. Importantly, binary compatibility of components should not change within a major (2-part version) DiFX release.

Bug Fixes

  • mpifxcorr: LBA format data now scaled roughly correctly (removing the need for large ACCOR corrections).
  • mpifxcorr: There was a bug when xmaclength was > nfftchan for pulsar processing. This has been corrected.
  • mpifxcorr: guardns was incorrectly (overzealously) calculated.
  • mpifxcorr: Mk5DataStream::calculateControlParams: bufferindex>=bufferbytes bug fixed.
  • mpifxcorr: Low weight reads could result in uninitialized memory; fixed.
  • mpifxcorr: Streamstor XLRRead() bug work-around installed several places (read at position 0 before reading at position > 0). This is thought not to be needed with Conduant SDK 9.2 but the work-around has no performance impact.
  • mpifxcorr: Fix to pulse calibration data ordering for LSB or reordered channels.
  • mpifxcorr: Pulse Cal amplitude now divided by pulse cal averaging time in seconds.
  • mpifxcorr: Pulse cal system would cause crash if no tones in narrow channel. Fixed.
  • vex2difx: Zoom band support across mixed bandwidths (see caveat below).
  • mpifxcorr: Fix for spurious weights at end of jobs (untested…)
  • mpifxcorr: Mixed 1 and 2 bit data are handled more cleanly
  • mpifxcorr: now terminates correctly for all short jobs. Previously it hung for jobs with a number of subints between nCores and 4 x nCores
  • mpifxcorr: Correctly scale cross-correlation amplitudes for pulsar binning when using Tsys>0 (accounts for varying # samples per bin c.f. nominal)
  • mpifxcorr: Lower side-band pulse cal tones had sign error. Fixed.

New Features

  • mpifxcorr: during Mark5-based correlation, emit multicast message containing drive statistics after each scan
  • mpifxcorr: Support for non power-of-2 FFT lengths
  • mpifxcorr: Multithread VDIF support enabled for the data sources FILE and MODULE, including stripping of non-VDIF packets
  • mpifxcorr/startdifx: Restart of correlation is now possible
  • mpifxcorr/vex2difx: new FAKE datastream type for performance testing
  • New utilities:
    • mk5map (limited functionality)
    • fileto5c
    • record5c
    • getsmart: during Mark5-based correlation, allows easy access to SMART data
  • mk5daemon: VSIS interface added
  • mk5daemon: Remote running of vex2difx enabled
  • New features added to existing utilities:
    • mk5cp: copy without reference to a module directory
    • mk5cp: ability to send data over ssh connection
    • vsn: get SMART data from disk drives
  • e-Control source code analysis (Martin Ettl, Wettzell)
  • difx2fits: -0 option to write minimal number of visibilities to FITS
  • difx2fits: write new RAOBS, DECOBS columns in source table
  • difx2fits: no longer copies autocorrelations to output FITS files except for phase centre 0 and pulsar bin 0 unless the -a option is used.
  • vex2difx: tweakIntTime has been re-enabled
  • diffDiFX.py can now cope with two files that don't have exactly the same visibilities (i.e., some visibilities are missing from one file)
  • plotDiFX.py and plotDynamicSpectrum.py now have better plotting and more options
  • Espresso, a lightweight system for managing disk-based correlation, has been added to the DiFX repository.
  • vex2difx: Option to correlate only one polarization has been added.
  • mk5dir can now produce .dir file information for VDIF formatted data.
  • Add NRAO's sniffer plotters to the repository.

Known issues

  • vex2difx: To support zoom bands across mixed bandwidths (and other unnusual modes), “number of channels” becomes a poorly defined quantity. There is a general change to use spectral resolution rather than number of channels when specifying correlation parameters. See the documentation at http://cira.ivec.org/dokuwiki/doku.php/difx/vex2difx. nChan still should work for simple cases.
  • vex2difx: There is currently no clean way to specify different zoom bands for different observing modes in one .v2d files
  • difx2fits: When running difx2fits on jobs that were restarted mid-way, the oldest data is used preferentially over newer data. It would make sense someday to change that.
  • mpifxcorr: It has been observed in some cases (believed to be restricted to very high output data rates) that visibilities start getting dropped, likely as a result of the visibility buffer filling up. Increasing the size of that buffer and perhaps using single-scan jobs limits the problems.

DiFX 2.0.1 (released Jun 24, 2011)

Bug Fixes

  • difx2fits: Fix model accountability bug when combining jobs
  • mpifxcorr, vex2difx, difxio, difx2fits: Numerous fixes for zoom bands
  • mpifxcorr: Native Mark5 has improved stability for cranky modules
  • difx2fits (primarily) + mpifxcorr: Numerous fixes for DiFX-based phase cal extraction (mostly for multi-job)
  • mpifxcorr: Fractional bit correction for lower sideband got broken in difx 2.0.0. Fixed.
  • difxcalculator: Migrated to DiFX 2; was not complete for DiFX 2.0.0
  • mpifxcorr: Clock offsets > 1/2 second were not handled properly for Mark5 module playback due to ignoring of intclockseconds
  • vex2difx: no longer rounds down to zero the subintegration time when adjustment is needed for the highest spectral resolutions
  • mpifxcorr: The amount of extra data sent to the cores to account for geometric delays was incorrectly scaled by number of channels. Fixed.

New Features

  • new utility checkmpifxcorr (mpifxcorr/utils)
  • mpifxcorr: Switched power detection
  • mpifxcorr: Improvements to support infiniband speeds with native Mark5
  • mpifxcorr: Early multi-thread VDIF format support
  • multiple: RedHat RPM file generation for some packages (can extend to others on request)
  • difx2fits: Improvements to method of selecting which pulse cal tones get propagated to FITS
  • mpifxcorr: Initial complex sampling support
  • mk5dir: now sorts by scan time (useful for modules assembled from e-transfered data)
  • all: Support underscores in source names

Caveats and other information

  • The sign of DiFX-extracted pcal phase has swapped since DiFX 2.0.0
  • While supported at some level, support for decimation of oversampled data is not documented and is not known to work correctly; use at your own risk
  • Default tones to put in the FITS file are the two tones in each sub-band that are nearest the edge, but if possible not those within the edge 1/8 of the band.
  • vex2difx SETUP parameter nChan changed to refer to the number of output channels; nFFTChan replaces functionality of specAvg
  • This release includes version 3.4 of the Haystack Observatory Postprocessing System (HOPS) for analyzing DiFX output that has been converted to the Mark4 correlator format with difx2mark. For more details, consult the documentation pages:

DiFX 2.0 (released Oct 13, 2010)

DiFX2.0 was the first version of a completely rewritten DiFX architecture. (Previous versions were DiFX-1.5.0 to 1.5.4, covering the period 2007-2010). The feature list below is not complete - someone would need to go through the individual change logs of all packages to gather all of the changes - there were very many. Below are listed the most important.

New features

  • mpifxcorr: Big speed improvements due to code improvements for cache management and some smarter algorithms: ~20% for “normal” observations, much more for high spectral resolution.
  • mpifxcorr: Multiple phase centres in one pass
  • mpifxcorr: “Zoom” bands (spectral selection)
  • mpifxcorr: Production of spectral kurtosis estimates in additional to higher, configurable time resolution filterbank data
  • mpifxcorr: LO offset correction
  • mpifxcorr: More efficient writing of visibilities enabling higher sustained write speeds and smaller output files
  • mpifxcorr/vex2difx: More finely grained control of the data to be correlated using “Rule” table in input file
  • mpifxcorr: mostly complete Mark5B support
  • mpifxcorr: Mark5B eVLBI support
  • Allow for new VEX FILE line in calc file to be optional

Bug fixes

  • mpifxcorr: visibilities with zero weight are not written to disk
  • mpifxcorr: polyco parser and checker is more forgiving of TEMPO2 generated polycos
  • mpifxcorr: Catch rare datastream error where file ends exactly at end of databuffer for Mk5 files
  • mpifxcorr: polycos with more than one entry now parsed correctly
difx/changelog.txt · Last modified: 2019/08/28 10:54 by adamdeller