User Tools

Site Tools


difx:utils

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
difx:utils [2017/10/17 23:49]
helgerottmann
difx:utils [2019/09/12 11:52] (current)
chrisphillips
Line 12: Line 12:
   * ''difx2fits'' : Takes the output of ''mpifxcorr'' and generate ''.FITS'' files   * ''difx2fits'' : Takes the output of ''mpifxcorr'' and generate ''.FITS'' files
   * ''difx2mark4'' : Takes the output of ''mpifxcorr'' and generates Mark4 format files   * ''difx2mark4'' : Takes the output of ''mpifxcorr'' and generates Mark4 format files
-  * ''calcif2'' : Takes ''.calc'' file made by ''vex2difx'' and produces the delay model (''.im'') file +  * ''difx2profile'' : Takes the output of ''mpifxcorr'' and averages data of all antennas over the entire time, in preparation of ''profile2binconfig.py''. Related to pulsar processing. 
-  * ''CalcServer'' : A service that makes calls to the CALC program upon request by an RPC call +  * ''difxcalc'' : Takes ''.calc'' file made by ''vex2difx'' and calculates delay model (''.im'') file using built-in CALC 11 model 
-  * ''difxcalc'' : creates .im model (based on Calc11) from .calc files. Replaces the CalcServer.+  * ''calcif2'' : Wrapper for ''difxcalc'' or ''CalcServer'' Being phased out along with ''CalcServer'' in favor of ''difxcalc'' 
 +  * ''CalcServer'' : A service that makes calls to the CALC program (Calc9) upon request by an RPC call, easiest started with ''startCalcServer''.
  
 == DiFX Operations == == DiFX Operations ==
  
-  * ''difxcopy'' : Copies ''.input'' file, and other files referenced within, to new path, updating full paths internally+  * ''difxcopy'' : Copies DiFX output to another directory and updates internal path references (e.g., in ''.input'') accordingly. 
 +  * ''mvdifxhere'' : Moves DiFX fileset from a given path to the local directory, updating internal paths.
   * ''difxlog'' : Captures multicast status and messages from a running DiFX job and writes to a text file   * ''difxlog'' : Captures multicast status and messages from a running DiFX job and writes to a text file
   * ''difxsniff'' : Concattenates as appropriate output sniffer files from ''mpifxcorr'' and plots using ''plotwt'', ''plotapd'' and ''plotbp''   * ''difxsniff'' : Concattenates as appropriate output sniffer files from ''mpifxcorr'' and plots using ''plotwt'', ''plotapd'' and ''plotbp''
-  * ''genmachines''Determines location of Mark5 modules via request to ''mk5daemon'' processes and makes ''.threads'' and ''.machines'' files+  * ''filteredDifx2Fits''Prepare difx2fits jobs based on source, timerange and/or mode. 
 +  * ''genmachines'' : Makes ''.threads'' and ''.machines'' files depending on the data location and type (file, network, mark5, mark6) 
 +  * ''geteop.pl'' : Script to obtain EOP values from GSFC servers in VEX $EOP format.
   * ''makefits'' : Simplified wrapper interface for ''difx2fits'', tailored to NRAO's usage   * ''makefits'' : Simplified wrapper interface for ''difx2fits'', tailored to NRAO's usage
-  * ''filteredDifx2Fits'' : Prepare difx2fits jobs based on source, timerange and/or mode. 
   * ''oms2v2d'' : Makes skeleton ''.v2d'' file based on the ''sched'' ''.oms'' output file   * ''oms2v2d'' : Makes skeleton ''.v2d'' file based on the ''sched'' ''.oms'' output file
   * ''startdifx'' : Command line utility to start ''mpifxcorr''   * ''startdifx'' : Command line utility to start ''mpifxcorr''
   * ''stopdifx'' : Command line utility to stop a single running instance of ''mpifxcorr''   * ''stopdifx'' : Command line utility to stop a single running instance of ''mpifxcorr''
-  * ''vexpeek'' : Program that reads a ''.vex'' file and prints the project name and list of stations and the time period each observed; used by ''db2vex''+  * ''vexpeek'' : Program that reads a ''.vex'' file and prints the project name and list of stations and the time period each observed
   * ''vlog'' : Splits up a TSM style text log file into ''pcal'', ''weather'', ''tsys'' and ''flags'' files; mainly for VLBA operations   * ''vlog'' : Splits up a TSM style text log file into ''pcal'', ''weather'', ''tsys'' and ''flags'' files; mainly for VLBA operations
   * ''[[startcorr.pl]]'' : Simple DiFX launcher, with eVLBI support   * ''[[startcorr.pl]]'' : Simple DiFX launcher, with eVLBI support
   * ''difxwatch'' : Monitors progess of running mpifxcorr. Kills job if no progess for longer than 300s.   * ''difxwatch'' : Monitors progess of running mpifxcorr. Kills job if no progess for longer than 300s.
   * ''[[espresso]]'': a lightweight suite of programs designed for management of file-based correlation.    * ''[[espresso]]'': a lightweight suite of programs designed for management of file-based correlation. 
- 
- 
  
 == DiFX File Validation == == DiFX File Validation ==
Line 52: Line 53:
   * ''vsum'' : Prints summary information about a vdif file. Can produce filelist to be digested by vex2difx   * ''vsum'' : Prints summary information about a vdif file. Can produce filelist to be digested by vex2difx
   * ''vmux'' : A program to take a multi-thread VDIF file and multiplex into a multi-channel, single thread file   * ''vmux'' : A program to take a multi-thread VDIF file and multiplex into a multi-channel, single thread file
-  * ''printVDIFheader'' : A program to dump some basic info about VDIF packets to the screen+  * ''printVDIFheader'' : A program to dump some basic info about VDIF packets, including extended VDIF header information, to the screen
   * ''printVDIFgaps'' : A program to look for missing VDIF packets   * ''printVDIFgaps'' : A program to look for missing VDIF packets
   * ''printVDIF'' : A program to dump some basic info about VDIF packets to the screen   * ''printVDIF'' : A program to dump some basic info about VDIF packets to the screen
Line 58: Line 59:
   * ''vdif2to8'' : A program to take a VDIF file containing 2-bit samples and convert it to 8-bit samples   * ''vdif2to8'' : A program to take a VDIF file containing 2-bit samples and convert it to 8-bit samples
   * ''vdifChanSelect'' : A program select a subset of channels from a VDIF file   * ''vdifChanSelect'' : A program select a subset of channels from a VDIF file
-  * ''vdifcontinuitycheck.py'' : ? 
   * ''vdifd'' : A VDIF decoder for multi-thread, single channel VDIF data   * ''vdifd'' : A VDIF decoder for multi-thread, single channel VDIF data
   * ''vdiffold'': A VDIF decoder for multi-thread VDIF data   * ''vdiffold'': A VDIF decoder for multi-thread VDIF data
Line 64: Line 64:
   * ''vdifspec'' : A VDIF spectrometer for multi-thread VDIF data   * ''vdifspec'' : A VDIF spectrometer for multi-thread VDIF data
   * ''vdif_time'' : ?   * ''vdif_time'' : ?
-  * ''cleanVDIF'' : Walks packet-by-packet through a VDIF file looking for packets of invalid length (often ARP packets from a switch inadvertently captured by a Mark5C) or corrupted header info, stripping out bad packets and writing a 'clean' VDIF file.  FIXME: document in user guide +  * ''cleanVDIF'' : Walks packet-by-packet through a VDIF file looking for packets of invalid length (often ARP packets from a switch inadvertently captured by a Mark5C) or corrupted header info, stripping out bad packets and writing a 'clean' VDIF file.  FIXME: document in user guide 
-  *  ''multi2singlethreadVDIF'' : Takes a VDIF file with 2^N single-channel threads and multiplexes these channels into a single VDIF thread with multiple channels.  The resultant single thread output file can be processed by e.g. m5spec and other utilities, and by mpifxcorr. FIXME: document in user guide+  *  ''multi2singlethreadVDIF'' : Takes a VDIF file with 2^N single-channel threads and multiplexes these channels into a single VDIF thread with multiple channels.  The resultant single thread output file can be processed by e.g. m5spec and other utilities, and by mpifxcorr. FIXME: document in user guide
   * ''padVDIF'' : Walks packet-by-packet through a VDIF file looking for missing packets and inserting packets as necessary to make a continuous packet stream.  The inserted packets have the invalid bit set. FIXME: document in user guide   * ''padVDIF'' : Walks packet-by-packet through a VDIF file looking for missing packets and inserting packets as necessary to make a continuous packet stream.  The inserted packets have the invalid bit set. FIXME: document in user guide
   * ''printVDIF'' : Loops through a VDIF file inspecting each packet header and printing some basic summary info (time etc)   * ''printVDIF'' : Loops through a VDIF file inspecting each packet header and printing some basic summary info (time etc)
Line 72: Line 72:
 == Mark5 data tools == == Mark5 data tools ==
  
 +  * ''m5bstate'' : Mark5 state counter that produces sampler statistics.
   * ''m5d'' : Decodes the header and samples of baseband data from a mark5access supported baseband data file   * ''m5d'' : Decodes the header and samples of baseband data from a mark5access supported baseband data file
   * ''m5fold'' : Folds in time the voltage squared (akin to pulsar processing); can be used to identify switched power in a datastream   * ''m5fold'' : Folds in time the voltage squared (akin to pulsar processing); can be used to identify switched power in a datastream
Line 77: Line 78:
      
   * ''m5test'' : Reads through a baseband data file looking for missing or damaged frame headers   * ''m5test'' : Reads through a baseband data file looking for missing or damaged frame headers
-  * ''m5tsys'' : Extracts switched power from a baseband data file (rather, it extracts Pon and Poff for each sampled channel)+  * ''m5tsys'' : Extracts switched power from a baseband data file (it extracts Pon and Poff for each sampled channel).  The extracted values can be converted to system temperature (tsys) when combined with cal temperature data.
      
  
Line 83: Line 84:
   * ''zerocorr'' : A program that takes two baseband data signals and performs a zero-fringe-rate cross-correlation; can handle mismatched bands/sidebands and different data formats and sample rates    * ''zerocorr'' : A program that takes two baseband data signals and performs a zero-fringe-rate cross-correlation; can handle mismatched bands/sidebands and different data formats and sample rates 
   * ''m5spec'' : Forms spectra of each sampled channel in a baseband data file   * ''m5spec'' : Forms spectra of each sampled channel in a baseband data file
 +  * ''m5subband'' : Extracts via filtering an arbitrary narrow subband out a wideband recording.
  
 == Phasecal tools == == Phasecal tools ==
   * ''m5pcal'' : Extracts pulse cals from a baseband data file   * ''m5pcal'' : Extracts pulse cals from a baseband data file
-  * ''plotpcal''? +  * ''plotpcal''Versatile P-Cal tone plotting utility, with multiband delay estimation.  This is more or less unmaintained and is being superceded by ''plotpcal2''. 
-  * ''plotpcal2''? +  * ''plotpcal2''Faster, more capabile version of ''plotpcal''. 
-  * ''plotDiFXPCal.py''?+  * ''plotDiFXPCal.py''Fast P-Cal tone plotting utility, with multiband delay estimation within and accross bands, PDF export.
  
 == Pre correlation tools == == Pre correlation tools ==
 +  * ''geteop.pl'' : Obtain EOP parameters from usno_finals.erp
 +  * ''fslog2difx.pl'' : extract DiFX relevant information from the Field System station logs
  
 == Post correlation tools == == Post correlation tools ==
Line 97: Line 101:
   * ''plotDynamicSpectrum.py'' : Given a .difx file, makes a dynamic spectrum of the visibilities (either a single baseline or scalar averaged over all baselines). FIXME: document in user guide   * ''plotDynamicSpectrum.py'' : Given a .difx file, makes a dynamic spectrum of the visibilities (either a single baseline or scalar averaged over all baselines). FIXME: document in user guide
   * ''printDiFX'' : Prints summary visibility information of DiFX output files    * ''printDiFX'' : Prints summary visibility information of DiFX output files 
 +  * ''polswapDiFX.py'' : Swap polarization labels in DiFX .difx output file for specified station(s).
 +  * ''difx2difx.py'' : Take DiFX output and un-zoom selected zoombands into new contiguous bands
 +  * ''stripantennaDiFX.py'' : Make a copy of DiFX output, keeping or removing all visibilities on baselines to certain antennas. FIXME: document in user guide
 + 
 == DiFX Testing == == DiFX Testing ==
 +
   * ''diffDiFX.py'' : Generates a context-sensitive difference of two DiFX output files for detailed version testing.  Corresponding visibility records are differenced and statistics on the differences are accumulated and printed at the end of the processing.   * ''diffDiFX.py'' : Generates a context-sensitive difference of two DiFX output files for detailed version testing.  Corresponding visibility records are differenced and statistics on the differences are accumulated and printed at the end of the processing.
   * ''neuteredmpifxcorr'' : Runs mpifxcorr without actually writing out visibilities. FIXME: document in user guide   * ''neuteredmpifxcorr'' : Runs mpifxcorr without actually writing out visibilities. FIXME: document in user guide
Line 106: Line 115:
  
 == Mark6 Operations == == Mark6 Operations ==
-  * ''vdifuse'' : Fuse mount Mark6 scattered VDIF files+ 
 +  * ''fuseMk6'' : Fuse mount Mark6 modules with recordings in any format. 
 +  * ''vdifuse'' : Fuse mount Mark6 modules with recordings in VDIF format, with VDIF inspection
   * ''mk6mon'': Displays multicast information about Mark6 units   * ''mk6mon'': Displays multicast information about Mark6 units
   * ''mk5daemon'' : If invoked with -6 option will handle Mark6 operations e.g. module mounting on keyturn etc.   * ''mk5daemon'' : If invoked with -6 option will handle Mark6 operations e.g. module mounting on keyturn etc.
 +  * ''m6sg_mount'' : (Un)mount any inserted Mark6 modules 
 +  * ''scan_check'' : Examine a mark6 file (either sg fragments or fused-VDIF) and report issues 
 + 
 == Mark5 Operations == == Mark5 Operations ==
  
Line 125: Line 138:
  
 == HOPS related == == HOPS related ==
 +
   * ''difx2mark4'' : convert .difx correlator output to the mark4 output format digested by the HOPS programs   * ''difx2mark4'' : convert .difx correlator output to the mark4 output format digested by the HOPS programs
   * ''vex2ovex'' : convert a .vex file to .ovex which is needed by some utilities of the HOPS suite of programs.   * ''vex2ovex'' : convert a .vex file to .ovex which is needed by some utilities of the HOPS suite of programs.
- 
  
 == NRAO specific == == NRAO specific ==
Line 147: Line 160:
   * ''listcpus'' : Uses ssh to determine CPU type of all machines in a DiFX cluster   * ''listcpus'' : Uses ssh to determine CPU type of all machines in a DiFX cluster
   * ''makefits'' : Simplified wrapper interface for ''difx2fits'', tailored to NRAO's usage   * ''makefits'' : Simplified wrapper interface for ''difx2fits'', tailored to NRAO's usage
 +
 +== Libraries for developers ==
 +
 +  * Python module ''difxdb'' : ...
 +  * Python module ''difxfile'' :  ...
 +  * Python module ''difxutil'' :  ...
 +  * Python module ''mark5access'' :  ...
 +  * C library ''mark5access'' : decode VDIF and older raw data formats
 +
 == Other == == Other ==
  
   * ''dedisperse_difx'' : FIXME: document in user guide   * ''dedisperse_difx'' : FIXME: document in user guide
difx/utils.1508244552.txt.gz · Last modified: 2017/10/17 23:49 by helgerottmann