User Tools

Site Tools


lbaops:atnfmonitoring

Monitoring ATNF VLBI experiments

The following notes assume all 3 ATNF telescopes (Parkes, ATCA, Mopra) are being monitored. Ignore the parts of the notes which refer to telescopes not being used for the current experiment. These notes also do not explain how to setup an experiment - refer to the telescope specific notes elsewhere in this wiki.

Remote observing qualification

You must be appropriately qualified for remote observing with Parkes and ATCA, even if observing from the SOC. Refer to the specific observing documentation beforehand, or discuss with CASS staff.

VLBI observing Quiz

Please ensure you know the answer to ALL of the questions on this quiz page. If you cannot answer them, then please re-watch the videocasts and re-read this documentation.

If you feel the documentation is not clear about any of this, then please send feedback to Chris.

Observing Quiz

Setup

Antenna Monitoring

Run the following standard telescope monitoring pages:

Make sure you register as as OIC at Parkes and ATCA and you “book” the observing beforehand

VLBI Monitoring

Run the following VLBI specific monitors

https://www.narrabri.atnf.csiro.au/vlbi/lba_monitor.html
http://www.evlbi.atnf.csiro.au/recmon/index.php

recmon must be run on a machine with working sound turned on. Test this “Alarm Test” button.

Make sure all appropriate telescopes/recorders are enabled (or disabled).

UTAS telescopes use a grafana page (login required). UTAS perform their own monitoring, so ATNF observers typically don't need this except for curiosity:

https://space.phys.utas.edu.au/monitoring/

The following telescope specific webpages are superseded by the LBA monitor above, but may be useful:

Troubleshooting

VNC

Connect your local VNC client to the following VNC sessions. Refer to these notes for notes on setting up ssh tunneling if observing remotely. If you do not know the current VNC password, refer to these notes

Parkes joffrey:1 All control
Parkes joffrey:2 Not used
ATCA xbones:1 ATCA antenna control
ATCA xbones:2 VLBI control
ATCA caccc1:1 CABB control and monitoring
Mopra bigrock:1 Antenna and VLBI control

Antenna notes

Troubleshooting

Some general notes on trouble shooting issues. There are ATNF specific, but not antenna specific (see notes above for antenna specific issues).

Troubleshooting

Video Screencasts

Video tutorials for observing

Monitoring Summary

Regular Monitoring

Check the following every ~10min on lba_monitor and recmon web monitors.

  • Pages are updating (check UTC time)
  • No telescopes idle (see below)
  • All telescopes on the same source (see below)
  • Wind speed
  • Recorder is running
  • Data sampler statistics are OK
  • PPS errors are low

Note that when ATCA or Mopra are hardware wind stowed, they may report to be “idle” or “drive error”. Look at the wind monitor to decide if that is what has happened.

Because of the elevation limit, Parkes often misses out some sources. It will then just go to the next source it can reach. If the ATCA/Mopra elevation is < 30deg, this is most likely what has happened. You can verify this if you want by looking at the experiment “.sum” file in the ftp area. ASKAP has a similar issues because sources rise at ASKAP much later than on the East coast.

Full check (every ~30min)

Check the following in the specific telescope VNC windows

ATCA, Mopra, Parkes

  • DAS levels OK
  • Recmon running

ATCA

  • caobs running OK
  • phases look OK on strong compact sources (see below)

The PI should have scheduled automatic phasecal commands within the schedule. If not you should run “corr pcal” in caobs if the phases drift too far apart, or consider dropping antenna from the tie array output if the phase stability is too poor on the longer baselines.

Parkes

  • atsnap running

Mopra

  • vlobs running
  • Tsys values look reasonable (not ~0)
  • GTP/SDO values OK

ASKAP

  • px14_record running
  • Stack counting down while tracking
  • Antenna tracking
  • No drive failures (red light on ASKAP GUI)
  • ASKAP GUI not frozen
  • VLBI drive not frozen

Windstow

Each telescope has different windstow/storm stow behaviour

Mopra

The system will reliably detect high wind and lightening activity and stow the antenna. It will automatically recover one the wind/lightning has reduced to an acceptable level.

  • The stow is at a hardware level, so atdrivemon will show Drive Error
  • The protection summary MoniCA page will show Lightning Stow or Hardware Stow as true
  • At the end of a stow event, atdrivemon may show “idle” briefly and the above monitor point indicate no stow (showing “false”). This usually lasts 1-2 minute
  • The telescope will only resume at the start of the next scan (possibly explaining the above behaviour).

ATCA

ATCA has two levels of windstow - software and hardware (PMON), For a software windstow, the observer will have to manually restart the observation in caobs (start). Wait 10-15 minutes after the wind has died to a reasonable level. For a PMON windstow, the system will be locked out for 15 minutes. You will also need to manually restart in caobs once the telescope is released from windstow. During a PMON wind stow, trying to drive the antennas will result in a “DRIVE ERROR” in atdrivemon. If all antennas report drive error at the same time, it is more than likely a PMON wind stow.

Storm stows are done in software, and while the lightning indicator is “STORM STOW” caobs will not allow you to start observing, and will send the antennas to their stow position. There will be no indication in caobs when a storm stow has been released, so you will need to look in MoniCA to determine when you can begin observing again.

Parkes

Parkes weather stow is controlled by TPS. If high wind or lightning activity is detected, TPS will stow the antenna and not release for 15 min. Often observing will resume automatically. In gusty high wind it is generally best to stop atsnap (^C; atsnap stow) and wait till the wind has calmed down. Resume with atsnap unstow, waitm then atsnap <exper>.psn (or up arrow twice). In some circumstance TPS will stow the antenna. In which case it will need to be unstowed manually before resuming.

lbaops/atnfmonitoring.txt · Last modified: 2023/02/10 17:41 by cormac