User Tools

Site Tools


lbaops:webmonitoring

This is an old revision of the document!


The web page that we now commonly use for monitoring the entire LBA is at:

http://www.narrabri.atnf.csiro.au/vlbi/lba_monitor.html

The monitoring the page uses requires access to a number of MoniCA servers, and some pre-computed JSON files generated by scripts. This wiki describes how to start all the connections and scripts required to make the LBA monitor page work.

MoniCA connections

Positional and weather information comes from MoniCA for ATCA, Parkes, Mopra, Hobart, Ceduna, ASKAP and Warkworth. To have this work requires that the webserver (namoi) have access to the MoniCA ASCII port 8051 on each of the MoniCA machines. For ATCA, Parkes, Mopra, ASKAP and Warkworth, namoi can access the MoniCA servers directly.

For the other observatories (Hobart and Ceduna) we need to set up some SSH tunnelling to allow namoi to get access. For this we make the ATNF machine mentok link to the Hobart MoniCA machine newsmerd, and the ATNF machine kaputar link to the Ceduna MoniCA machines ops1cd.

Setting up/troubleshooting Hobart

To setup the Hobart connection, login to mentok as vlbi and give the command:

 screen -S hobart ssh -L mentok:8051:newsmerd:8051 -L mentok:8052:newsmerd:8052 \
     observer@newsmerd.phys.utas.edu.au 

You will need to know the observer password for newsmerd for this to work. If this command is successful you will end with an observer prompt on newsmerd. To exit screen without affecting the tunnel, press Ctrl-a d. You will get dropped back to a mentok prompt, but the tunnel will continue working in the background.

Of course, since we are running with screen, this SSH tunnel will survive even if VLBI logs out of mentok (as long as you haven't quit the screen session). So before you try starting a new screen session as above, try connecting to an existing one with:

 screen -x hobart 

If no screen is available called hobart, you will get a message saying as much. In that case, restart it with the “screen -S” command.

If the screen is present, but the monitoring is clearly not working on the web page, first check that the connection to newsmerd has not been dropped. If after connecting to the hobart screen you don't get a prompt on newsmerd then you will need to exit the screen, and restart it. In theory, the way we started the screen means this should never happen: if the connection to newsmerd dies, the screen session should disappear.

If the newsmerd connection is there, but the monitoring is not working, you will probably need to restart the MoniCA process on newsmerd. To do this:

  • Login to newsmerd as observer in another terminal (do not use the connection in the screen session for this).
  • Look for the processes 'socket' and 'monica_monitor' with the command:
     ps -u observer | grep 'socket\|monica_monitor' 
  • Take a note of the two PIDs that come back from that command, or if they are there at all (sometimes these processes fail).
  • Stop the MoniCA server with:
     ~/usr/bin/open-monica-server.sh stop 
  • Kill the 'socket' and 'monica_monitor' processes, and make sure they are dead by checking with 'ps'.
  • Restart the MoniCA server with:
     ~/usr/bin/open-monica-server.sh start 

At this point the monitoring on the web page should return. If it doesn't, please reload the web page. If this still doesn't work, call Jamie Stevens.

Setting up/troubleshooting Ceduna

To setup the Ceduna connection, login to kaputar as vlbi and give the command:

 screen -S ceduna ssh -L kaputar:8051:ops1cd:8051 -L kaputar:8052:ops1cd:8052 \
     observer@newsmerd.phys.utas.edu.au 

You will need to know the observer password for newsmerd for this to work. If this command is successful you will end with an observer prompt on newsmerd. To exit screen without affecting the tunnel, press Ctrl-a d. You will get dropped back to a kaputar prompt, but the tunnel will continue working in the background.

Of course, since we are running with screen, this SSH tunnel will survive even if VLBI logs out of kaputar (as long as you haven't quit the screen session). So before you try starting a new screen session as above, try connecting to an existing one with:

 screen -x ceduna 

If no screen is available called ceduna, you will get a message saying as much. In that case, restart it with the “screen -S” command.

If the screen is present, but the monitoring is clearly not working on the web page, first check that the connection to newsmerd has not been dropped. If after connecting to the ceduna screen you don't get a prompt on newsmerd then you will need to exit the screen, and restart it. In theory, the way we started the screen means this should never happen: if the connection to newsmerd dies, the screen session should disappear.

If the newsmerd connection is there, but the monitoring is not working, you will probably need to restart the MoniCA process on newsmerd. To do this:

  • Login to ops1cd as observer in another terminal (do not use the connection in the screen session for this). If you're within the ATNF firewall, you'll need to do this via newsmerd first.
  • Look for the processes 'socket' and 'monica_monitor' with the command:
     ps -u observer | grep 'socket\|monica_monitor' 
  • Take a note of the two PIDs that come back from that command, or if they are there at all (sometimes these processes fail).
  • Stop the MoniCA server with:
     ~/usr/bin/open-monica-server.sh stop 
  • Kill the 'socket' and 'monica_monitor' processes, and make sure they are dead by checking with 'ps'.
  • Restart the MoniCA server with:
     ~/usr/bin/open-monica-server.sh start 

At this point the monitoring on the web page should return. If it doesn't, please reload the web page. If this still doesn't work, call Jamie Stevens.

Monitoring other observatories

Two other observatories are monitored without MoniCA: Tid and Hart.

The monitoring from Tid is pushed by the Tid computers to the location /u/vlbi/tid/cdscc_monitor, and is in JSON format. The monitoring web page knows how to get this information. If the Tid monitoring isn't working, contact Tid staff, as there is nothing we can do to fix it.

For Hart we need to use a script to grab the monitoring information they make available from http://spyda.hartrao.ac.za/cgi-bin/hartrao_26_lba.txt and turn it into a JSON file so it can be manipulated properly by the monitoring page. The script that does this is located at /n/vlbi/monitoring/hart2json.pl. We run this in a screen session as well and it will fetch new information every second and turn it into the file /n/vlbi/monitoring/hart.json, which the web page knows how to get.

To do this, login to mentok as the vlbi user, and give the command:

 screen -S hart -d -m /n/vlbi/monitoring/hart2json.pl 

There should be no feedback from this command except the monitoring working on the web page.

Monitoring the recorders

The VSIB recorders are monitored using a script that queries the recorder's server process and turns the result into JSON output. The script that does this is /n/vlbi/monitoring/recorder2json_tunneled.pl.

One instance of this script has to be run for each observatory, and for all observatories except ATCA an SSH tunnel must be setup to get access to the recorder_server query port 50080.

Setting up ATCA recorder monitoring

For the ATCA, all you need to do is use the script. Start the script on mentok with the command:

 screen -S cavsijson -d -m /n/vlbi/monitoring/recorder2json_tunneled.pl \
   --prefix ca --host cavsi1:cavsi1:50080 --host cavsi2:cavsi2:50080 

This will create a file /n/vlbi/monitoring/carec.json that will refresh every 20 seconds or so.

Setting up Parkes recorder monitoring

First, you will need to start an SSH tunnel on mentok with the command:

 screen -S pkvsitunnel ssh -L 50082:pkvsi1:50080 -L 50083:pkvsi2:50080 vlbi@pkvsi2-ext 

Once this is done (you may be prompted for the vlbi password), exit the screen session (Ctrl-a d) and then run the command:

 screen -S pkvsijson -d -m /n/vlbi/monitoring/recorder2json_tunneled.pl \
   --prefix pk --host pkvsi1:localhost:50082 --host pkvsi2:localhost:50083 

This will create a file /n/vlbi/monitoring/pkrec.json that will refresh every 20 seconds or so.

Setting up Mopra recorder monitoring

First, you will need to start an SSH tunnel on mentok with the command:

 screen -S mpvsitunnel ssh -L 50080:mpvsi1:50080 -L 50081:mpvsi2:50080 vlbi@mpvsi1-ext 

Once this is done (you may be prompted for the vlbi password), exit the screen session (Ctrl-a d) and then run the command:

 screen -S mpvsijson -d -m /n/vlbi/monitoring/recorder2json_tunneled.pl \
   --prefix mp --host mpvsi1:localhost:50080 --host mpvsi2:localhost:50081 

This will create a file /n/vlbi/monitoring/mprec.json that will refresh every 20 seconds or so.

Setting up Hobart and Ceduna recorder monitoring

First, you will need to start an SSH tunnel on mentok with the command:

 screen -S utastunnel ssh -L 50084:hovsi:50080 -L 50085:cdvsi:50080 vlbi@hovsi.phys.utas.edu.au 

This sets up the SSH tunnels for both the Hobart and Ceduna recorders. Once this is done (you may be prompted for the vlbi password), exit the screen session (Ctrl-a d) and then run the commands:

 screen -S hovsijson -d -m /n/vlbi/monitoring/recorder2json_tunneled.pl \
   --prefix ho --host hovsi:localhost:50084 
 screen -S cdvsijson -d -m /n/vlbi/monitoring/recorder2json_tunneled.pl \
   --prefix cd --host cdvsi:localhost:50085 

This will create the files /n/vlbi/monitoring/horec.json and /n/vlbi/monitoring/cdrec.json that will refresh every 20 seconds or so.

lbaops/webmonitoring.1401413470.txt.gz · Last modified: 2015/12/18 16:39 (external edit)