ATNF VLBI Wiki

ASKAP extensions useful to VLBI are:

From CSIRO phones replace prefix with just 99 and then the 4 digit extension.

In the beginning of VLBI operations at ASKAP 2010-2013, all observations were done on ANT29 (CETCI #1), from an installation in hut29 next to the antenna. Connections were made via direct cabling from the hut to the antenna as needed.

Since the start of 2014, all VLBI equipment has been moved to the shielded room in the main ASKAP control building. Communication with the antennas is via RFoF connections for the RF and 100 MHz reference (for the X-band Rx). Communication with teh antenna is via the standard C&M network (202.9.10.1xx, where xx is the CETCI antenna number).

From May 2014, VLBI operation has moved to antenna Ak32 (CETCI #26) for X-band, and Ak07 for L-band.

ASKAP antennas are accessed for VLBI via a VNC session on the Curtin Recorder (cira10) housed in the shielded equipment room at the ASKAP site at MRO. The vlbi recorder is on a 1 Gbps link to the routed CSIRO network. It can now be accessed directly as cira10.atnf.csiro.au, with both ssh or VNC from ATNF networks. If the VNC server is not running, ssh in and type “startvnc” to start it with the correct parameters.

To run the antenna for VLBI, the directory should be set to “askap/run” and commands can be run in separate xterms.

The driver task “askap_drv” needs to run first for communicating with the antenna, and then the other tasks are started. This can be done by:

• “askap #n”, where #n is the CETCI antenna number. 1 for Ak29 or 26 for Ak32 or 19 for Ak07. e.g.
  • “askap 26” to run with Ak32.

This starts askap_drv and the antenna gui in one step. It also displays the “askap_drv #n” session is a separate xterm to monitor messages.

This command also loads the correct pointing parameters from the master pointing file antenna_pointing.data in askap/run.

* N.B. If GUI crashes you MUST also kill the message window before restarting!! Otherwise the connection may not be properly established.

For vlbi observing via the schedules you need:

• “vlbi_drv_mk2” to run the vlbi schedule (a .psn file like Parkes).
  

(Older versions such as vlbi_drv and vlbi_drv_new are available but their use is not recommended).
The schedules are in sub-directory askap/sched. To run try:

• e.g. vlbi_drv_mk2 ../sched/vt10l-ak.psn

The main antenna control is run from the gui. Essential commands that can be typed in the command window are:

• “update” to synch with the ntp server. This is essential if “clk err” is large. (usually is below 0.1 second). Must NOT exceed 0.5 sec. Can only be updated when antenna is not tracking (counter not incrementing). Update may not work if antenna is not considred “active”.
  • IF the update does not work, it may imply ntp issues with cir10 (the vlbi observing computer.)
• “stow” to stow the antenna. During stowing the “high-stow” block will show “driving” and finally “in”. Once stowed many red blocks will be reported on the gui.
• “unstow” to get things moving. All red blocks should turn to yellow and the antenna would be ready to run.
• “drive_off” to manually turn drives off (especially if there is a red bar after the unstow)
• “drive_on” to manually turn drives on.

(Note: if the telescopes is having problem switching the drives, a sequence of drive_off and then drive_on often fixes things)

One can track a source from the gui e.g. for fringe checks.

• in the “select” line, select “J2000” and type in the RA and DEC required. Then make sure that “track” is selected in the “action” line and click on “start obs”. You can use the “stop” button to stop.

* During tracking make sure that the “stack” button is decrementing from about 38 to 1 in cycles. You can also monitor this in the askap_drv window.

For debugging problems other useful commands are:

• version - to give the firmware version
• k4stat - use after a problem to check for error messages. It clears when a drive command is given.

For pointing runs, a “spot” option is available instead of “track”. This does scans in a cross pattern, and tehse can be used to determine pointing offsets, especially for the X-band Rx.

The “scan” option allowas for controlled scans through a source.

The vlbi schedule for askap is a .psn file similar to Parkes. Normally this will have been already generated and copied to cira10. If you need to generate this from the vex file in the vlbi ftp area one needs to run “vex2psn.pl -askap <vexfile>” and scp to cira10 in the area askap/sched. e.g.

• vex2psn.pl -askap v252ab.vex > v252ab-ak.psn

in the v252ab area would create the .psn file, Note the _ak in the filename, so as to not confuse with the .psn for Parkes.

In another window in the askap/run area type “vlbi_drv_mk2”. It will ask for the schedule name. It should be provided by full path name e.g. “../sched/v272ab-ak.psn”. Or the schedule name can be provided in the command line (only for the _mk2 version of the driver).
It should then run for the duration of the schedule and the telescope should slew and track at the appropriate times. If a source has set or has not risen the schedule will report “sleep” and will wait till the next scan.

• N.B.: Since August 2013, please run “vlbi_drv_new” instead.
• N.B.: Since December 2013, please run “vlbi_drv_mk2”. This can take the schedule name as a parameter on the same line. Then ^ works to get back to the previously run command, including the schedule name.

N.B. 1: If you stop vlbi_drv with ^C it can hang the gui and/or crash askap_drv. If you need to stop vlbi_drv best use the “q” command in that window. But q does not always work so ^C can be used if needed.

N.B. 2: If schedule is started early and sources are below horizon, vlbi_drv behavior may be uncertain. Watch closely and restart schedule if necessary.

N.B. 3: Even when vlbi_drv is running, the telescope can be stopped from the gui. If you do that, it is not clear if the schedule picks up ok again. vlbi_drv may need to be restarted.

Note that the system use “automount”. Currently it is configured with three data disks /mnt/raid_0, /mnt/raid_1 and /mnt/raid_2. You may need to “cd” into the directories before they seem visible.

Before recording data the directory should be created in /mnt/raid_0 e.g “mkdir v272ab” and then cd to that area before starting the recording program px14_record (similar to vsib_record). To run a quick check one can use e.g.

• px14_record -v 12 -t 1m -f 1s which will record 1 minute of data in 1 second files.
  • if -t and -f are not specified the system records 60s of data in 1 file.

• Note 1: Check the output lines in px14_record window. The “Stddev=x, y” line should report numbers around 1000-2000 and it is controlled by the “-v” flag.
  

N.B. For the X-band Rx on Ak32 (since May 2014) you may have to use “-v 18” to get the rms values between 1000 and 2000 counts.

• Note 1a: One can inspect the recorded band with fauto e.g.
  • fauto -n 512 <filename.lba> – gives the averaged band in the file
  • fauto -n 512 -nint 32000 <filename.lba> – gives spectrum every 1s.
  • fauto -n 512 -tint 1 <filename.lba> – gives spectrum every 1s.

• Note 2: The last number in the output record is the time difference in the sampler. It should be small (order 1). If it is very large (many 10s) then the system may NOT working, especially if this number is changing rapidly.(If stable then the system may be ok). Most likely culprit is ntp server on cira10.
• Note 3: The ntp server on cira10 should be checked and/or restarted. Root access is necessary for restarts.
  • ntptime - gives the ntp status in the machine. The error should be microsecs. If it is hundreds of microsecs or millisecs then the ntp demon is not working properly.
  • to restart the ntp demon try “sudo /etc/init.d/ntp restart”.
  • the command “ntpdate -q 202.9.10.254” checks system. “ntpdate 202.9.10.254” resyncs.
  • IF all else fails cira10 may need to be rebooted. Then the full VNC will need setting up again.
• Note 4: To check correct timing of the sampler over longer periods, the .log file in the recording subdirectory can be plotted and inspected i.e.
  • quickplot.pl -time -x 2 -c 9 *.log (or give the .log name you want). Maybe add “-line” option to connect the points in the graph. (Ignore PGPLOT::HANDLE error message).
  • to just get the .log data for monitoring time one can run the recorder with -nodata
    • e.g. px14_record -n 12 -nodata -rms -t 12h (to get 12h of timing and rms data)
  • If the time difference is high but steady, it may indicate an H-maser to GPS time difference (provided the ntp is working ok). A re-sync of the H-maser should fix this.

Correct operation can be checked by running fauto with say “-n 512 -tint 1” on one of the recorded files.

Real-time fringe-checking is done for all LBA setups and will be the primary means of verifying correct operation. It is run centrally for all telescopes.

For production recording longer files should be run and an output name should be given e.g.

• px14_record -o v272ab -v 12 -t 12h

will create the default 1 min length files for 12 hours.

• For delayed start time add “-start dateTtime” e.g. “-start 2012-03-10T08:02:00”
• N.B.: the “-invert” option should be used for inverted bands. Currently for the 1.6 GHz band.

NOTE 1: To stop px14_record it is recommended to use ESC rather than ^C.

NOTE 2: When restarting make sure you use all the correct parameters e.g. the “-invert” option. Best to use up-arrow and edit the previous line appropriately.

NOTE 3: The PX14400 kernel driver sometimes gives errors like “failed to allocate DMA buffer”, and needs to be restarted. This can only be done as root. The command then is “/etc/init.d/px14400 restart” (the available commands are: stop, start, restart).

The H-maser controls the timing for vlbi in the askap shielded room. It is possible to monitor and control the H-maser via a VNC (mro-maser1.atnf.csiro.au:0). Check with the vlbi group or Peter Mirtschin before attempting any adjustments on the H-maser.

The 1pps from the H-maser has glitched occasionally, most likely due to power cycling of the site, even though the H-maser is on 2 levels of UPS. If fringes are not found, the H-maser can be re-synched remotely.

The difference of the 1pps from the H-maser and the 1pps from the GPS (the ntp server in the main comms rack) can be monitored with a local timer-counter, but this can only be done manually.

The ATDC computer provides a station clock. It is synched to the GPS and the 5 MHz reference from the H-maser. It counts its own 1pps and this can be used to synch the samplers. This clock is displayed on the station and the tick-phase is monitored and logged.
BUT it has been playing up recently (March 2012) and it should be checked out before used.

• N.B. No ATDC is installed in the shielded room (May 2014)

The sampler in the VLBI computer expects a 64 MHz IF at 288 MHz (256-320 MHz. It uses a 128 MHz sampling clock provided by a SML 01 synthesiser (output level at 5 dBm), locked to a 10 MHz external reference from the H-maser. Make sure that “ext ref” is showing.

The LO for the L-band is provided by an Agilent at the antenna pedestal. This is locked to a 10 MHz external reference from the H-maser. Output RF level needed is 11 dBm. Again “ext ref” MUST be showing.

• For 21 cm we use a 1240-1500 MHz filter (1370MHz, 260 MHZ BW).
  • e.g. for 1400, we set the LO to 1400-288 = 1112 MHz. “Low LO == non-inverted band”

• For 18cm we use a wider filter 1.13 -1.83 MHz (1480/T700)
  • e.g. for 1665 MHz, we use 1665+288=1953 MHz. “High LO == inverted band”
  • e.g. for 1660 MHz, we use 1660+288=1948 MHz. “High LO == inverted band” (e.g. RadioAstron)

These filters are inside the conversion box and cannot be easily swapped. If both 18cm and 21cm are used in one session, the wide filter MUST be used.

The X-band Rx has a fixed 1st LO at 7.1 GHz, locked to a 100 MHz reference signal for the H-maser. This is housed in the Rx box with the feed.

• N.B. Make sure that the 100 MHz reference signal is connected after receiver changes.

The 2nd LO is provided by the L-band system conversion equipment, as detailed above.

• e.g for 8425 MHz, we get 8425-7100=1325 MHz IF, and need 1325-288=1037 MHz L-band LO. “All LOs are low == non-inverted band.”
• For 8441 MHz, we have: 8441-7100=1341 MHz IF, need 1341-288=1053 MHZ L-band LO.

The Agilent providing the L-band LO is setup to remember its last state and recover to that on power-up.

The Agilent synthesiser can be controlled via Ethernet. It required the installation of a local ethernet switch at the antenna pedestal and the use of an ethernet-to-GPIB dongle.

• N.B. Remote control not currently available (May 2014)

The software to control the synthesiser is called locontrol.pl and takes options on the command line i.e.

• locontrol.pl -status – Report the status of the Agilent
• locontrol.pl -lo=f – Setup the LO to f MHZ directly (see earlier discussion)
• locontrol.pl -lfreq=f – Setup LO to give sky frequency of f MHz (at L-band)
• locontrol.pl -xfreq=f – Setup LO to give sky frequency of f MHz (at X-band)
• locontrol.pl -power=level – Setup LO power to level e.g +11 dBm for the LO.
• locontrol.pl -rfon – Turn RF ON
• locontrol.pl -rfoff – Turn RF OFF
• locontrol.pl -help – get a list of commands and syntax

The -lfreq and -xfreq commands report the LO that has been setup and also if the band is inverted or not. The only inverted band at present is when lfreq is around 1.6 GHz. If the band is inverted do not forget to set the -invert option in px14_record.

It is prudent to sent a “status” command every time setup changes are made.

Other options are:

• locontrol.pl -reset – Reset Agilent if in pathological case.

These should be used with CAUTION.

The antenna status can be monitored via Monica, running at the ATCA.

• “askapmon.pl” starts the generation of local monitoring from an xterm. The monitoring strings are displayed on this xterm.

To get to the Monica host, an ssh tunnel is needed, from within a screen session, logged on as vlbi. The command for the tunnel (e.g from Cetus) is:

ssh -R 8051:monhost-nar:8051 vlbi@cira10

To setup (or reconnect) to the screen session run the following (but NOT from cira10)

ssh vlbi@cetus
screen -ls

If no sockets are found then run

screen
ssh -R 8051:monhost-nar:8051 vlbi@cira10
^a d  (Control-a d) 

this leaves the screen session detached so you can log off and leave it.

If the output of screen -ls lists one (or more) sessions then you can reattach using

screen -r

if, for example, the ssh tunnel has closed you need to restart. Detach with “^a d” when done

Monica also enables monitoring via the LBA web monitoring page at
https://www.narrabri.atnf.csiro.au/vlbi/lba_monitor.html

The perl script px14mon.pl emulates the LBADR recorder_server and allow remote monitoring of the recorder, particularly for the “recmon” webpage. It can be run in any xterm (usually within the cira10:1 vnc) as simply px14mon.pl. It can be left to run “permanently”.