LBA Disk Recording
Prev
Next

Data Verification

The programs vsib_checker and fauto can be used to check data during recording.
Vsib_checker does a number of simple checks on the data while fauto computes the bandpass and plots it. The number of channels and bandwidth are now written into the file headers. The common options needed to describe the data format include:


-b/bits/ <nbits>
 Number of bits of digitization. Defaults to 2 (and higher precision is not supported yet)
-v/-vlba
 Bit encoding is in vlba (offset binary) rather than sign/magnitude. This is needed if the DAS "vsop" mode was used.


vsib_checker

Vsib_checker has three main modes of operation. In PPS checking mode, it checks the 1 PPS marker is set every second or returns an error. This is a good way to check that recording is running smoothly. This is the default mode of operation.

> vsib_checker test_131_013622.lba      # Check a single file
> vsib_checker -fail vt01i1_131*        # Check multiple files at once. Stop running if any problems are found


In sampler stats mode, the sampler statistics are calculated. By default the data for the whole file is used, which is slow. The -n <nbytes> option tells it to only check the first nbytes bytes. However this is rounds up to the internal buffer size (usually 2 MB). This means running with -n 10 will check 2 MB of data, give an accurate calculation and run fairly quickly. The sampler statistics should be 17.3 % very high and very low and 32.7% for the rest.


> vsib_checker -s test_131_013622.lba         # Check sample stats on a single file
> vsib_checker -s -vlba test_131_013622.lba   # Check sample stats on a single file, assuming vlba encoding
> vsib_checker -n 100 -s test_131_013622.lba  # Check single file, only checking the first 2 MB of data

The third main mode for vsib_checker is to check data from the VSIC countdown mode (mode 6). This is extremely useful way of testing the VSIB and PC are working correctly (disks running fast enough etc). Vsib_checker will check that each subsequent sample is numerically 1 less than the preceding sample. It reports an error if not. Generally you will want to run with the -fail option as one error normally means many errors will be reported (but not always for intermittent errors). In countdown mode, vsib_checker uses a different definition of number of bits compared to the other tests. When recording 256 Mbps (vsib_record mode 3) the number of bits needs to be set to 8. For 512 Mbps rates (vsib_record mode 2), the number of bits must be set to 16.


> vsib_checker -x -b 8 test_131_013622.lba      # Test a single file
> vsib_checker -count -b 8 test_131_013622.lba  # The same test
> vsib_checker -x test_131_*                    # Check multiple files
> vsib_checker -x -b 16 test_131_*              # Check data recorded in vsib_record mode 2 (16 bit/512 Mbps)


fauto

Fauto calculates the data bandpass and plots it using pgplot to either postscript or the screen. It simply converts the digitized data to a floating point number, FFTs and then averages the data stream. It can do a single 16 MHz channel in roughly realtime, but is computationally very intensive. Fauto should not be run constantly during data recording and probably not at all during 512 Mbps recording. The user can choose which channels are processed, by default all are processed. A zoom mode is available by setting the first and last channel to plot. A mode is also available to calculate a series of integrations times shorter than the entire length of the file, optionally pausing between plots. Currently the number of FFTs to average needs to be specified rather than the integration time in seconds (this will be changed when time permits). Example usage (obviously options would normally be combined):


> fauto test_131_013622.lba                  # Calculate 4 bandpasses on a single file and plot to the screen
> fauto -v test_131_13622.lba                # Same but assume VLBA bit encoding
> fauto -n 1024 test_131_13622.lba           # Use a 1024 point FFT (resulting in 512 spectral channels)
> fauto -C 1 test_131_13622.lba              # Only process the first channel
> fauto -C 1 -C 3 test_131_13622.lba         # Process the first and third channel
> fauto test_131_13622.lba -dev plot.ps/cps  # Save output as a postscript file
> fauto test_131_13622.lba -dev plot.png/png # Save output as a .png file
> fauto test_131_13622.lba -n 1024 -sp1 100 -sp2 300 # Plot a zoomed channel range


Pre-experiment testing

The following is a set of tests that are recommended before starting a recording run (for example  before the start of a  VLBI session).


VSIC Countdown: Set the VSIC to mode 6 (0110). Run the recorder for a few minutes (if time, it can be worthwhile running these tests for many hours. However, you need to allow roughly the same amount of time to test the data as record it). Check data is recorded, the PPS markers are in place and run vsic_checker in countdown mode to check all the data was recorded correctly. Generally you would want to run this at 512 Mbps (to stress the system the most), but some hardware setups may not support 512 Mbps but are OK for 256 Mbps recording rates. To do a 512 Mbps test:


> vsib_record -m 2 -t 100s         # Record data at 512 Mbps (16 bits per clock sample)
> vsib_checker test*               # Check 1 PPS markers.
> vsib_checker -x test*      # Check each sample.

To do a 256 Mbps test:

> vsib_record -t 100s              # Record data at 256 Mbps (16 bits per clock sample)
> vsib_checker test*               # Check 1 PPS markers.
> vsib_checker -x test*       # Check each sample. Need to say 8 bit data.

Remember to set the VSIC back to mode E (1110) after these tests

Data quality: Set the VSIC back to mode E (1110), then record another few minutes of data from the DAS (either directly from the DAS or via the S2). Check the 1 PPS markers and sampler statistics are OK. It is probably worth running this test with the VSIC directly connected to the DAS and connected via the S2 and BG2 adapter.
When directly attached to the DAS:


> vsib_record -t 100s            # Record some data at 256 Mbps
> vsib_checker test*             # Check the 1PPS markers are in the right place
> vsib_checker -s -n 100 test*   # Check the sampler stats are 17.3 and 32.7%

When directly attached via the S2, enable data feed through in the S2 (uic feedthru on). 

> vsib_record -t 100s -c ooxx        # Record some data at 128 Mbps. The compress mode depends on the current S2 mode
> vsib_checker  test*                # Check the 1PPS markers are in the right place
> vsib_checker -s -n 100 test*       # Check the sampler stats are 17.3 and 32.7%

Bandpass: Use fauto to check the bandpass. It is also useful to turn on a weak test tone (much weaker than needed for coherence) a few MHz from the band edge, to check the band inversion and sky frequency is as expected. The same data as used for the data quality check can be used

> fauto -n 512 test_131_135631.lba        # For direct recording modes
> fauto -n 512 test_131_135631.lba        # For compressed recording via the S2
> fauto -n 512 -vlba test_131_135631.lba  # If a DAS "vsop" mode was used

Testing during experiment

Data testing during an while recording is occurring is a good idea, but should be kept to a minimum to avoid overloading the system. 1 PPS checking should be fairly safe and checking sampler statistics on a small fraction of the data also. fauto is computationally intensive so generally should be avoided. vsib_record checks the 1PPS markers so further checking is at most needed only occasionally. If these checks are run, it is recommended to run the 1PPS checks on handful of recent files and check only a small section of one file for sampler statistics


> nice vsib_checker vt01i1_131_135*             # Check 1PPS for 10 minutes of data
> nice vsib_checker -s -n 100 vt01i1_131_135620 # Check the sampler stats of a single file


ASKAP
Public