Data Verification
The programs vsib_checker and fauto can be used to check data during recording.
Vsib_checker does a number of simple checks on the data while fauto computes the bandpass and plots it. The number of channels and bandwidth are now written into the file headers. The common options needed to describe the data format include:
-b/bits/ <nbits> |
Number of bits of digitization. Defaults to 2 (and higher precision is not supported yet) |
-v/-vlba |
Bit encoding is in vlba (offset binary) rather than sign/magnitude. This is needed if the DAS "vsop" mode was used. |
vsib_checker
Vsib_checker has three main modes of operation. In PPS checking mode, it checks the 1 PPS marker is set every second or returns an error. This is a good way to check that recording is running smoothly. This is the default mode of operation.
> vsib_checker -fail vt01i1_131* # Check multiple files at once. Stop running if any problems are found
In sampler stats mode, the sampler statistics are calculated. By default the data for the whole file is used, which is slow. The -n <nbytes> option tells it to only check the first nbytes bytes. However this is rounds up to the internal buffer size (usually 2 MB). This means running with -n 10 will check 2 MB of data, give an accurate calculation and run fairly quickly. The sampler statistics should be 17.3 % very high and very low and 32.7% for the rest.
> vsib_checker -s -vlba test_131_013622.lba # Check sample stats on a single file, assuming vlba encoding
> vsib_checker -n 100 -s test_131_013622.lba # Check single file, only checking the first 2 MB of data
The third main mode for vsib_checker is to check data from the VSIC countdown mode (mode 6). This is extremely useful way of testing the VSIB and PC are working correctly (disks running fast enough etc). Vsib_checker will check that each subsequent sample is numerically 1 less than the preceding sample. It reports an error if not. Generally you will want to run with the -fail option as one error normally means many errors will be reported (but not always for intermittent errors). In countdown mode, vsib_checker uses a different definition of number of bits compared to the other tests. When recording 256 Mbps (vsib_record mode 3) the number of bits needs to be set to 8. For 512 Mbps rates (vsib_record mode 2), the number of bits must be set to 16.
> vsib_checker -count -b 8 test_131_013622.lba # The same test
> vsib_checker -x test_131_* # Check multiple files
> vsib_checker -x -b 16 test_131_* # Check data recorded in vsib_record mode 2 (16 bit/512 Mbps)
fauto
Fauto calculates the data bandpass and plots it using pgplot to either postscript or the screen. It simply converts the digitized data to a floating point number, FFTs and then averages the data stream. It can do a single 16 MHz channel in roughly realtime, but is computationally very intensive. Fauto should not be run constantly during data recording and probably not at all during 512 Mbps recording. The user can choose which channels are processed, by default all are processed. A zoom mode is available by setting the first and last channel to plot. A mode is also available to calculate a series of integrations times shorter than the entire length of the file, optionally pausing between plots. Currently the number of FFTs to average needs to be specified rather than the integration time in seconds (this will be changed when time permits). Example usage (obviously options would normally be combined):
> fauto -v test_131_13622.lba # Same but assume VLBA bit encoding
> fauto -n 1024 test_131_13622.lba # Use a 1024 point FFT (resulting in 512 spectral channels)
> fauto -C 1 test_131_13622.lba # Only process the first channel
> fauto -C 1 -C 3 test_131_13622.lba # Process the first and third channel
> fauto test_131_13622.lba -dev plot.ps/cps # Save output as a postscript file
> fauto test_131_13622.lba -dev plot.png/png # Save output as a .png file
> fauto test_131_13622.lba -n 1024 -sp1 100 -sp2 300 # Plot a zoomed channel range
Pre-experiment testing
The following is a set of tests that are recommended before starting a recording run (for example before the start of a VLBI session).
> vsib_checker test* # Check 1 PPS markers.
> vsib_checker -x test* # Check each sample.
> vsib_record -t 100s # Record data at 256 Mbps (16 bits per clock sample)
> vsib_checker test* # Check 1 PPS markers.
> vsib_checker -x test* # Check each sample. Need to say 8 bit data.
When directly attached to the DAS:
> vsib_record -t 100s # Record some data at 256 Mbps
> vsib_checker test* # Check the 1PPS markers are in the right place
> vsib_checker -s -n 100 test* # Check the sampler stats are 17.3 and 32.7%
> vsib_record -t 100s -c ooxx # Record some data at 128 Mbps. The compress mode depends on the current S2 mode
> vsib_checker test* # Check the 1PPS markers are in the right place
> vsib_checker -s -n 100 test* # Check the sampler stats are 17.3 and 32.7%
Bandpass: Use fauto to check the bandpass. It is also useful to turn on a weak test tone (much weaker than needed for coherence) a few MHz from the band edge, to check the band inversion and sky frequency is as expected. The same data as used for the data quality check can be used
> fauto -n 512 test_131_135631.lba # For direct recording modes
> fauto -n 512 test_131_135631.lba # For compressed recording via the S2
> fauto -n 512 -vlba test_131_135631.lba # If a DAS "vsop" mode was used
Testing during experiment
Data testing during an while recording is occurring is a good idea, but should be kept to a minimum to avoid overloading the system. 1 PPS checking should be fairly safe and checking sampler statistics on a small fraction of the data also. fauto is computationally intensive so generally should be avoided. vsib_record checks the 1PPS markers so further checking is at most needed only occasionally. If these checks are run, it is recommended to run the 1PPS checks on handful of recent files and check only a small section of one file for sampler statistics
> nice vsib_checker vt01i1_131_135* # Check 1PPS for 10 minutes of data
> nice vsib_checker -s -n 100 vt01i1_131_135620 # Check the sampler stats of a single file