PROCESSING AT EPPING:           pmsurv/docs/epping_processing.txt
				Froney Crawford - July 23, 1998 
				FC/RNM          - July 31, 1998
				FC              - Aug 7,   1998
				RNM             - Sept 21, 1998
				RNM             - Jan 11, 1999
							                    
0) Login as pulsar in all places.
   Set up windows: 1 for tape reading on attila (tape window)
                   1 for the master on attila   (master window) 
                   1 for pmproc on attila       (pmproc window) 

1) in tape window (attila): 
   cd to bdhunt directory.
   eliminate old stuff if it is there: 
	     mkdir PM00YY (if bdhunt is filled with junk) 
	     mv PM00YY_* PM00YY/.
	     mv digitizer.offsets PM00YY/.
	     mv PM00YY.pmbs PM00YY/.
	     (keep everything else)
   mt rewi (make sure TAPE env var is set to /dev/rmt/0n on attila)
   get digitizer.offsets: 
       sc_td -D -c 3.5 -n 1 -A PM00XX 
       when done, copy relevant line to "digitizer.offsets" 
   mt rewi 

2) In pmproc window (attila): 
      type "pmproc" pmproc> n PM00XX (this prepares for new tape PM00XX) 
   In tape window (attila): 
      pmtape b
      (make sure it starts reading data) 
   In master window (attila):
      type ps gux and make sure no other pmchief_hunter process is
      going -- if it is, kill it.  
      type pmchief_hunter (should start printing lines every
      few secs) 
   In pmproc window (attila):  
      b SLAVENAME /full_path/name/SLAVENAME_slave/
   (repeat for other slaves -- recommend 4 birdslaves
   to keep up with the tape read)
   Currently available directories are: 
      /psr5/pm/attila0/
      /psr5/pm/attila1/
      /psr5/pm/attila2/
      /psr5/pm/attila3/
      /psr9/pulsar/pm/raptor0/
      /psr9/pulsar/pm/raptor1/ 
      /psr9/pulsar/pm/raptor2/ 
      /psr9/pulsar/pm/raptor3/
      /psr9/pulsar/pm/raptor4/ 
      /DATA/RAPTOR_3/pulsar/pm/raptor5/
      /DATA/RAPTOR_3/pulsar/pm/raptor6/
      /DATA/RAPTOR_6/pulsar/pm/raptor7/
      /DATA/RAPTOR_11/pulsar/pm/raptor8/
      /DATA/RAPTOR_11/pulsar/pm/raptor9/
      /DATA/PUPPIS_2/pulsar/pm/puppis0/
      /psr6/pm/tycho0/
      
   Use pmproc to monitor processes, takes about 1 min/beam to process.
   
3) When birdie hunting is finished:
   In pmproc window: 
      pmproc> t  (stops tape script)
   In tape window: 
      mt rewi (after tape script has stopped)
   In pmchief_hunter window: 
      should exit automatically after birdie summary is done
      If birdie_summary doesn't run do: %pmbird PM00XX 1 n
   pmbirdslaves:
      you can keep them going, if you need to stop them,
      type s attila or s raptor in the pmproc window.

4) In pmproc window:
      pmproc> n PM00XX (sets up new tape again for processing)
   In tape window:
      copy digitizer.offsets from bdhunt dir to data_pool dir.
      pmtape d (make sure script starts reading data)
   In master window:
      make sure no other pmmaster is going (ps -fA | grep pulsar) 
      If it is, kill it. 
      pmmaster (starts printing a line every few secs)
   In pmproc window:
      l SLAVENAME /full_path_name/SLAVENAME_slave/
      see birdslave examples for format...
      each slave takes 2 GB. 
   Monitor processes with pmproc. 
   (Each beam takes about 5 hr to process.)

5) All files pertaining to a particular pointing should be eliminated
   from the data_pool automatically when the pointing is done being
   processed. If there is a problem with one of the beams, it will not
   automatically eliminate it. It will keep the pointing around for
   manual perusal. If you want it gone, edit "complete_pts.txt" in the
   conductor dir and add a line with the name of the pointing you want
   eliminated (e.g., PM0039_010). This should do the trick.

[NOTE: If the number of mis-processed beams becomes large enough that
this is a nuisance, perhaps the script should be changed to erase the
data beam by beam -- i.e., keep only the beams around which are bad,
not the whole pointing.]

To recover if things get in a mess:

At the pmproc prompt, exit pmtape and pmmaster. 

cd to conductor dir
  rm * in conductor
  touch bigsister.data
  touch processing_status.log
  chmod 664 processing_status.log

cd to data_pool 
  rm PM* 
  rm ex*  (leave digitiser.offsets)

Normally you will want to rewind the input tape 
  mt rewind
and (later) edit valid_pointings.txt to restart at the correct place (see below).

cd to $PM_BOOK on each processor being used and kill slave processes:
  kill_slaves
On attila also do
  clean_slaves
to clear all scratch directories.

Check that no pm jobs still running:
  ps -fA | grep pulsar

After 5 min, check that the bigsister cron jobs are running on each processor to
be used. In the conductor directory, look for files bs_out_proc.log
  ll bs*

If these don't appear, on the processor concerned
  crontab -r to remove
  crontab /psr/bin/sun4sol/pmcron (or crontab /psr/bin/sgi/pmcron)
Then after 5 min, check that the bs files appear.

If birdie hunting needs to be repeated:
cd to bdhunt
  rm PM0039_* in bdhunt 
  rm birdie_files, exit_status_sc_td, candidates, checkbmtmp, checktmp
  in bdhunt
cd bdhunt/PM0039
  cp * ../. 

cd to book dir
  either rm or mv to some other place the pointing dirs you want to repeat 

Start pmproc and initialise tape
  >n PM0039

Start the tape reading
  pmtape d (or b)
If you want to begin part way down the tape, you have one minute to edit
  $PM_HOME/conductor/valid_pointings.txt to remove the files you don't want 
to repeat before sc_td starts.

Start pmmaster

Launch slaves in pmproc. Check that the slaves go into WAIT state and then
TREE state when data are available.