User Tools

Site Tools


difx:vex2difx

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
difx:vex2difx [2017/04/30 04:46]
walterbrisken
difx:vex2difx [2021/09/16 22:36] (current)
janwagner [Table]
Line 28: Line 28:
  
 Note, only formal vex files are supported as input to vex2difx.  Similar looking ovex files used at some/all Mark4 correlators are not acceptable, however, with a small amount of work an ovex file can be hand converted to a valid vex file.  It would not be hard to write a conversion script to do this automatically. Note, only formal vex files are supported as input to vex2difx.  Similar looking ovex files used at some/all Mark4 correlators are not acceptable, however, with a small amount of work an ovex file can be hand converted to a valid vex file.  It would not be hard to write a conversion script to do this automatically.
 +
 +===== vex2 support =====
 +
 +On Aug 1, 2021, the vex2 committee completed the definition of vex2.  The final version of the format definition can be found [[https://safe.nrao.edu/wiki/bin/view/VLBA/Vex2doc|here]].
 +
 +difx2fits will be gaining vex2 capability in the [[https://svn.atnf.csiro.au/difx/applications/vex2difx/branches/vex2|vex2]] branch of vex2difx.  A log of supported vex2 features implemented in this branch can be found [[vex2support|here]].
  
 ===== The configuration file ===== ===== The configuration file =====
Line 47: Line 53:
 ==== Specifying data formats ==== ==== Specifying data formats ====
  
-The ''format'' parameter of an ANTENNA or DATASTREAM section in the ''.v2d'' file, or the ''track_frame_format'' within a vex TRACKS section gives ''vex2difx'' information needed to determine how the data is arranged on the media.  In the past (before DiFX 2.5the two sources of format information had different formatting options.  With DiFX 2.5 a new unified format decoding infrastructure has been added that give more flexibility.  With this formats, either in vex or ''.v2d'' files can be specified in one of several ways:+The ''format'' parameter of an ANTENNA or DATASTREAM section in the ''.v2d'' file, or the ''track_frame_format'' within a vex TRACKS section gives ''vex2difx'' information needed to determine how the data is arranged on the media.  Previous to DiFX 2.5 the two sources of format information had different formatting options.  With DiFX 2.5 a new unified format decoding infrastructure has been added that give more flexibility.  With this formats, either in vex or ''.v2d'' files can be specified in one of several ways:
  
   * <fmt>   * <fmt>
Line 76: Line 82:
   * LBA   * LBA
   * LBAVSOP   * LBAVSOP
 +
 +Note that some work-arounds are needed for complex data to handle limitations of the vex format - see [[complexsamplingvex|this page]] for details.
  
 Some general tips: Some general tips:
Line 129: Line 137:
 | maxReadSize    | int    | bytes | 25000000   | Max read size in bytes (larger values cause issues with Mk5 module playback) | | maxReadSize    | int    | bytes | 25000000   | Max read size in bytes (larger values cause issues with Mk5 module playback) |
 | minReadSize    | int    | bytes | 10000000   | Min read size in bytes (smaller values mean probable inefficiency) | | minReadSize    | int    | bytes | 10000000   | Min read size in bytes (smaller values mean probable inefficiency) |
-| delayModel     | string |       | calcif2    | The executable (must be in path) of the delay model program to run (DiFX 2.5 and later) |+| delayModel     | string |       | calcif2    | The executable (must be in path) of the delay model program to run 
 +| exhaustiveAutocorrs | bool |    | false      | When true, generate autocorrs as cross correlations; this is useful if two polarizations for one or more antenna are in different datastreams |
  
 Note that the baselines parameter supports the following syntaxes:  A1-A2   A1+A2+A3-A4+A5   A1-*  A1+A2-* and so on.  For each list member, all baselines consistant with an antenna match on both sides will be kept. Note that the baselines parameter supports the following syntaxes:  A1-A2   A1+A2+A3-A4+A5   A1-*  A1+A2-* and so on.  For each list member, all baselines consistant with an antenna match on both sides will be kept.
Line 146: Line 155:
   SOURCE 3c273 { source parameters go here }   SOURCE 3c273 { source parameters go here }
  
-^ Parameter name   ^ Type   ^ Units ^ Default ^ Comments ^ +^ Parameter name    ^ Type    ^ Units  ^ Default  ^ Comments                                                                                                                                                                                                  
-| ra                      | J2000 |         | right ascension, e.g., 12h34m12.6s or 12:34:12.6 | +| ra                        | J2000           | right ascension, e.g., 12h34m12.6s or 12:34:12.6                                                                                                                                                          
-| dec                     | J2000 |         | declination, e.g., 34d12'23.1" or 34:12:23.1 | +| dec                       | J2000           | declination, e.g., 34d12'23.1" or 34:12:23.1                                                                                                                                                              
-| name             | string |               | new name for source | +| name              | string                  | new name for source                                                                                                                                                                                       
-| calCode          | char         | ' '     | calibration code, typically A, B, C for calibrators, G for a gated pulsar, or blank for normal target |  +| calCode           | char           | ' '      | calibration code, typically A, B, C for calibrators, G for a gated pulsar, or blank for normal target                                                                                                     
-| naifFile         | string |               | Path to a leap second kernel file for SPICE.  Only used with near-field correlations | +| naifFile          | string                  | Path to a leap second kernel file for SPICE.  Only used with near-field correlations                                                                                                                      
-| ephemObject      | string |               | Name of the object from the ephemFile to be associated with this source. Only used for near-field correlations |  +| ephemObject       | string                  | Name of the object from the ephemFile to be associated with this source. Only used for near-field correlations                                                                                            
-| ephemFile        | string |               | Path to a planetary ephemeris file for SPICE.  Only used with near-field correlations.  bsp or tle files are allowed. | +| ephemFile         | string                  | Path to a planetary ephemeris file for SPICE.  Only used with near-field correlations.  bsp or tle files are allowed.                                                                                     
-| doPointingCentre | bool         | true    | Whether the pointing centre should be correlated (only ever turned off for multi-phase centre | +| doPointingCentre  | bool           | true     | Whether the pointing centre should be correlated (only ever turned off for multi-phase centre)                                                                                                            
-| addPhaseCentre   | string |               | contains info on a source to add, with ra, dec and optionally name/calcode with no spaces, "/" separation and "@" in place of "=" e.g., "addPhaseCentre = name@1010-1212/RA@10:10:21.1/Dec@-12:12:00.34" |+| addPhaseCentre    | string                  | contains info on a source to add, with ra, dec and optionally name/calcode with no spaces, "/" separation and "@" in place of "=" e.g., "addPhaseCentre = name@1010-1212/RA@10:10:21.1/Dec@-12:12:00.34"  |
  
 ==== ANTENNA sections ==== ==== ANTENNA sections ====
Line 163: Line 172:
 ^ Parameter name ^ Type    ^ Units ^ Default            ^ Comments ^ ^ Parameter name ^ Type    ^ Units ^ Default            ^ Comments ^
 | name           | string  |                          | New name to assign to this antenna | | name           | string  |                          | New name to assign to this antenna |
-| polSwap        | bool    |       | False              | Swap the polarizations (i.e. L ←→ R) for this antenna +| polSwap        | bool    |       | False              | Swap the polarizations (i.e. L <-> R) for this antenna 
-| clockOffset    | float   | us    | vex value          | Overrides the clock offset value from the vex file; used in conjunction with clockEpoch  |+| clockOffset    | float   | us    | vex value          | Overrides the clock offset value from the vex file  |
 | clockRate      | float   | us/s  | vex value          | Overrides the clock offset rate value from the vex file; used in conjunction with clockEpoch | | clockRate      | float   | us/s  | vex value          | Overrides the clock offset rate value from the vex file; used in conjunction with clockEpoch |
-| clockEpoch     | date    |       | vex value          | Overrides the epoch of the clock rate value; must be present if clockRate or clockOffset parameter is set |+| clockEpoch     | date    |       | vex value          | Overrides the epoch of the clock rate value; should be set if clockRate parameter is set (defaults to MJD 50000) |
 | deltaClock     | float   | us    | 0.0                | Adds to the clock offset (either the vex value or the clockOffset above) | | deltaClock     | float   | us    | 0.0                | Adds to the clock offset (either the vex value or the clockOffset above) |
 | deltaClockRate | float   | us/s  | 0.0                | Adds to the clock rate (either the vex value or the clockRate above) | | deltaClockRate | float   | us/s  | 0.0                | Adds to the clock rate (either the vex value or the clockRate above) |
Line 187: Line 196:
 | toneGuard      | float   | MHz   | 0.125 of bandwidth | When using toneSelection ''smart'' or ''most'' don't select tones within this range of band edge, if possible  | | toneGuard      | float   | MHz   | 0.125 of bandwidth | When using toneSelection ''smart'' or ''most'' don't select tones within this range of band edge, if possible  |
 | toneSelection  | string  |       | ''smart''          | Use an algorithm to choose tones for you.  Read the code to learn more. | | toneSelection  | string  |       | ''smart''          | Use an algorithm to choose tones for you.  Read the code to learn more. |
-| sampling       | string  |       | ''REAL''           | Set to ''COMPLEX'' for complex sampled data or ''COMPLEX_DSB'' for double sideband  |+| sampling       | string  |       | ''REAL''           | Set to ''COMPLEX'' for complex sampled data or ''COMPLEX_DSB'' for double sideband (see also [[complexsamplingvex|this page]]) |
 | fake           | bool    |       | False              | enable a fake data source | | fake           | bool    |       | False              | enable a fake data source |
 | mjdStart       | date    |       | obs. start         | discard any data from this antenna before this time | | mjdStart       | date    |       | obs. start         | discard any data from this antenna before this time |
 | mjdStop        | date    |       | obs. stop          | discard any data from this antenna after this time | | mjdStop        | date    |       | obs. stop          | discard any data from this antenna after this time |
-| machine        | string  |                          | //Coming in ver. 2.5// if writing a .machines file, link this machine to this ANTENNA's datastream process | +| machine        | string  |                          | if writing a .machines file, link this machine to this ANTENNA's datastream process | 
-| datastreams    | strings |       | (none)             //Coming in ver. 2.5// links to DATASTREAM sections; below for more info | +| datastreams    | strings |       | (none)             | links to DATASTREAM sections; below for more info | 
 +| polConvert     | bool    |       | false              | mark this antenna to have its polarization basis changed (e.g., from XY to RL) |
  
 The addZoomFreq parameter freq always specifies the **lower edge** of the frequency channel, regardless of whether or not the parent band is USB or LSB. The addZoomFreq parameter freq always specifies the **lower edge** of the frequency channel, regardless of whether or not the parent band is USB or LSB.
Line 216: Line 225:
 Please note that vex uses as a clock sign convention that is positive for a formatter with its clock running fast (i.e., the second tick happens too early).  The clockOffset and clockRate in this ANTENNA section, as well as FITS files, have the opposite sign convention.  Hint: If you use the delays returned by AIPS FRING program as clock modifiers without changing their sign, you should end up with output that has no residual delay. Please note that vex uses as a clock sign convention that is positive for a formatter with its clock running fast (i.e., the second tick happens too early).  The clockOffset and clockRate in this ANTENNA section, as well as FITS files, have the opposite sign convention.  Hint: If you use the delays returned by AIPS FRING program as clock modifiers without changing their sign, you should end up with output that has no residual delay.
  
-==== DATASTREAM sections (coming in DiFX 2.5) ====+==== DATASTREAM sections ====
  
-New in upcoming DiFX version 2.5 will be support for multiple datastreams per antenna.  Since vex1.5 does not have the concept of multiple datastreams per antenna the additional information must be provided explicitly in the .v2d file.  Within .v2d files datastreams are linked to antennas.  Logically speaking the datastreams are functions not only of antenna but also of setup; cases that have varying recording modes through an experiment invariably have changes to the datastreams, as used by DiFX, as well.  Thus the implementation described here does not provide a fully general solution.  In cases where this breaks down it is likely that multiple .v2d files, each acting on a subset of the setups used, will allow the needed flexibility.  Note that when vex2 is fully supported, the STREAMS block within vex will give users access to the full generality of DiFX on a setup-by-setup basis.+DiFX version 2.5 and beyond supports multiple datastreams per antenna.  Since vex1.5 does not have the concept of multiple datastreams per antenna the additional information must be provided explicitly in the .v2d file.  Within .v2d files datastreams are linked to antennas.  Logically speaking the datastreams are functions not only of antenna but also of setup; cases that have varying recording modes through an experiment invariably have changes to the datastreams, as used by DiFX, as well.  Thus the implementation described here does not provide a fully general solution.  In cases where this breaks down it is likely that multiple .v2d files, each acting on a subset of the setups used, will allow the needed flexibility.  Note that when vex2 is fully supported, the STREAMS block within vex will give users access to the full generality of DiFX on a setup-by-setup basis.
  
 To enable multiple datastreams for an antenna, simply define 2 or more DATASTREAM sections (described below) and link them with the appropriate antenna by using the datastreams parameter of ANTENNA sections.  By default if there are //N// DATASTREAMS defined for an antenna, each will get one //N//th of the channels with the order of the channels preserved, meaning that the order of the datastreams argument does matter.  This can be overridden with an nBand parameter. To enable multiple datastreams for an antenna, simply define 2 or more DATASTREAM sections (described below) and link them with the appropriate antenna by using the datastreams parameter of ANTENNA sections.  By default if there are //N// DATASTREAMS defined for an antenna, each will get one //N//th of the channels with the order of the channels preserved, meaning that the order of the datastreams argument does matter.  This can be overridden with an nBand parameter.
Line 241: Line 250:
 Setup sections are enclosed in braces after the word SETUP and a name given to this setup section.  The setup name is referenced by a RULE section (see below).  A setup with the special name ''default'' will be applied to any scans not otherwise assigned to setups by rule sections.  If no setup sections are defined, a setup called ''default'', with all default parameters, will be implicitly created and applied to all scans.  The order of setup sections is immaterial. Note: The use of nChan (plus optionally specAvg) to set final (and FFT) spectral resolution is discouraged.  It is maintained for backwards compatibility and convenience, but if you have different subband bandwidths, you **cannot** use nChan, and must instead use specRes (and FFTSpecRes, if you want to explicitly set the FFT spectral resolution, for example in multifield projects). Setup sections are enclosed in braces after the word SETUP and a name given to this setup section.  The setup name is referenced by a RULE section (see below).  A setup with the special name ''default'' will be applied to any scans not otherwise assigned to setups by rule sections.  If no setup sections are defined, a setup called ''default'', with all default parameters, will be implicitly created and applied to all scans.  The order of setup sections is immaterial. Note: The use of nChan (plus optionally specAvg) to set final (and FFT) spectral resolution is discouraged.  It is maintained for backwards compatibility and convenience, but if you have different subband bandwidths, you **cannot** use nChan, and must instead use specRes (and FFTSpecRes, if you want to explicitly set the FFT spectral resolution, for example in multifield projects).
  
-^ Parameter name        ^ Type      ^ Units  ^ Default     ^ Comments                                                                                                                                                                                                                                     +^ Parameter name        ^ Type      ^ Units  ^ Default     ^ Comments                                                                                                                                                                                                                                                                                                                              
-| tInt                  | float     | sec    | 2           | integration time                                                                                                                                                                                                                             +| tInt                  | float     | sec    | 2           | integration time                                                                                                                                                                                                                                                                                                                      
-| FFTSpecRes            | float     | MHz    | 0.125       | spectral resolution of first stage FFTs                                                                                                                                                                                                      +| FFTSpecRes            | float     | MHz    | 0.125       | spectral resolution of first stage FFTs                                                                                                                                                                                                                                                                                               
-| specRes               | float     | MHz    | 0.5         | spectral resolution of visibilities produced                                                                                                                                                                                                 +| specRes               | float     | MHz    | 0.5         | spectral resolution of visibilities produced                                                                                                                                                                                                                                                                                          
-| nChan                 | int              | 16          | number of post-averaged channels per spectral window; currently must be a power of 2.  Do not use in combination with specRes/FFTSpecRes; nChan is only for convenience in simple cases (all stations have the same bandwidth for all subbands)  +| nChan                 | int              | 16          | number of post-averaged channels per spectral window; currently must be a power of 2.  Do not use in combination with specRes/FFTSpecRes; nChan is only for convenience in simple cases (all stations have the same bandwidth for all subbands)                                                                                       
-| doPolar               | bool      |        | True        | correlate cross hands when possible?                                                                                                                                                                                                         +| doPolar               | bool      |        | True        | correlate cross hands when possible?                                                                                                                                                                                                                                                                                                  
-| subintNS              | int       | ns     | 160000000   | The mpifxcorr SUBINT NS; should eventually be set to a smarter default                                                                                                                                                                       +| subintNS              | int       | ns     | 160000000   | The mpifxcorr SUBINT NS; should eventually be set to a smarter default                                                                                                                                                                                                                                                                
-| guardNS               | int       | ns               | The mpifxcorr GUARD NS; 2000 is usually sufficient.  With DiFX 2.5, set to zero and mpifxcorr will calculate for you.  Override may be needed for non-sidereal sources     +| guardNS               | int       | ns               | The mpifxcorr GUARD NS; 2000 is usually sufficientset to zero and mpifxcorr will calculate for you.  Override may be needed for non-sidereal sources                                                                                                                                                                                
-| maxNSBetweenUVShifts  | int       | ns     | 2000000000  | Used for multiphase centre stuff. if better time resolution than 1 threads portion of a subint is required                                                                                                                                   +| maxNSBetweenUVShifts  | int       | ns     | 2000000000  | Used for multiphase centre stuff. if better time resolution than 1 threads portion of a subint is required                                                                                                                                                                                                                            
-| maxNSBetweenACAvg     | int       | ns     | 2000000000  | Used for STA dumping (transient searches) if better time resolution than 1 threads portion of a subint is required                                                                                                                           +| maxNSBetweenACAvg     | int       | ns     | 2000000000  | Used for STA dumping (transient searches) if better time resolution than 1 threads portion of a subint is required                                                                                                                                                                                                                    
-| specAvg               | int              | 8           | The spectral averaging to perform inside the correlator, at the end of a subint                                                                                                                                                              +| specAvg               | int              | 8           | The spectral averaging to perform inside the correlator, at the end of a subint                                                                                                                                                                                                                                                       
-| fringeRotOrder        | int              | 1           | The fringe rotation order - 0=post-F, 1=linear, 2=quadratic                                                                                                                                                                                  +| fringeRotOrder        | int              | 1           | The fringe rotation order - 0=post-F, 1=linear, 2=quadratic                                                                                                                                                                                                                                                                           
-| strideLength          | int              | 16          | The number of channels to "stride" for fringe rotation, fractional sample correction etc.  With DiFX 2.5, set to zero for automatic setting on a per-datastream basis (usually good).  This is almost mandatory for non-commensurate sample rates.   +| strideLength          | int              | 16          | The number of channels to "stride" for fringe rotation, fractional sample correction etc.  With DiFX 2.5, set to zero for automatic setting on a per-datastream basis (usually good).  This is almost mandatory for non-commensurate sample rates.                                                                                    
-| xmacLength            | int              | 128         | The number of channels to "stride" for cross-multiply accumulations.  With DiFX 2.5, set to zero for automatic setting (usually good). | +| xmacLength            | int              | 128         | The number of channels to "stride" for cross-multiply accumulations.  With DiFX 2.5, set to zero for automatic setting (usually good).                                                                                                                                                                                                
-| numBufferedFFTs       | int              | 1           | The number of FFTs to do in a row for each datastream, before XMAC'ing them all                                                                                                                                                              +| numBufferedFFTs       | int              | 1           | The number of FFTs to do in a row for each datastream, before XMAC'ing them all                                                                                                                                                                                                                                                       
-| postFFringe           | bool      |        | False       | do fringe rotation after FFT?                                                                                                                                                                                                                +| postFFringe           | bool      |        | False       | do fringe rotation after FFT?                                                                                                                                                                                                                                                                                                         
-| binConfig             | string    |        | none        | if specified, apply this pulsar bin configuration file to this setup                                                                                                                                                                         +| binConfig             | string    |        | none        | if specified, apply this pulsar bin configuration file to this setup                                                                                                                                                                                                                                                                  
-| freqId                | int list  |        | none        | a comma separated list of integers that are freq table indexes to select which bands to correlate; default is to correlate all. **Note:** this should be used to select parent bands for zoom frequencies if difx2fits is to be run.         +| freqId                | int list  |        | none        | a comma separated list of integers that are freq table indices to select which bands to correlate; default is to correlate all. **Note:** this should be used to select parent bands for zoom frequencies if difx2fits is to be run. **Note2:** freq table indices usable for freqId are found in the .input file FREQ TABLE section  
-| phasedArray           | string    |        |             | if specified, tells DiFX to produce a phased array output instead of cross correlations, using the setup specified in this phased array config file                                                                                          |+| phasedArray           | string    |        |             | if specified, tells DiFX to produce a phased array output instead of cross correlations, using the setup specified in this phased array config file                                                                                                                                                                                   | 
 +| outputBandwidth       | float     | MHz    | none        | if specified, enables DiFX 2.7 outputband mode and auto-determines bands that fit the VEX frequency setups                                                                                                                                                                                                                            | 
 +| addOutputBand         | string    |        | none        | if specified, enables DiFX 2.7 outputband mode and adds an explicitly specified outputband, syntax freq@<sky freq low edge MHz>/bw@<bandwidth MHz>                                                                                                                                                                                    |
  
  
Line 334: Line 345:
   }      }   
  
-==== COMMENT sections //coming in DiFX 2.5// ====+==== COMMENT sections ====
  
-Starting with version 2.5.0 one can include COMMENT blocks in the .v2d file.  These have no effect on the files written by vex2difx but allow comments (likely instructions to the person executing vex2difx) to be seen at the end of the output to the terminal.  Each COMMENT block will make a new comment, separated by one line of whitespace in the output.  A comment block starts with COMMENT { and ends with a }  For example:+One can include COMMENT blocks in the .v2d file.  These have no effect on the files written by vex2difx but allow comments (likely instructions to the person executing vex2difx) to be seen at the end of the output to the terminal.  Each COMMENT block will make a new comment, separated by one line of whitespace in the output.  A comment block starts with COMMENT { and ends with a }  For example:
  
   COMMENT   COMMENT
Line 458: Line 469:
   /data/mk/bx123.003.m5a  54322.766323 54322.812311    /data/mk/bx123.003.m5a  54322.766323 54322.812311 
  
-If times for a file are supplied, the file will be included in the .input file DATA TABLE only if the file time range overlaps with the .input file time range.  If not supplied, the file will be included regardless of the .input file time range, which could incur a large performance problem.+If times for a file are supplied, the file will be included in the .input file DATA TABLE only if the file time range overlaps with the .input file time range.  If not supplied, the file will be included regardless of the .input file time range, which could incur a large performance problem. Start and stop times for a Mark5 or VDIF file can be  
 +extracted using the ''m5bsum -s'' or ''vsum -s'' commands respectively.
  
 A few sample ANTENNA blocks are shown below: A few sample ANTENNA blocks are shown below:
difx/vex2difx.1493491616.txt.gz · Last modified: 2017/04/30 04:46 by walterbrisken