This loads a font easier to read for people with dyslexia.
This renders the document in high contrast mode.
This renders the document as white on black
This can help those with trouble processing rapid screen movements.

CASDA Users Guide

Version 1.3 December 2016

Welcome to the CSIRO ASKAP Science Data Archive (CASDA) Users Guide. This Guide is intended to help astronomers get started with finding and making use of data products from the Australian Square Kilometre Array Pathfinder (ASKAP).

The first ASKAP data products have been produced from science commissioning observations taken with the Boolardy Engineering Test Array (BETA). These show the potential and unique wide-field fast survey capabilities of ASKAP and provide some demonstration data sets to the astronomy community.

For information on the initial data release see the BETA Data Release Notes.

For a reference document that preceded the construction of CASDA see the pdf document:
CSIRO ASKAP Science Data Archive: Overview, Requirements and Use Cases (Chapman et al. 2014)

Contents

1 CASDA Overview

2 Using CASDA with the CSIRO Data Access Portal

3 Using CASDA with Virtual Observatory Services for Catalogues

4 Using CASDA Virtual Observatory Services for Images and Image Cubes

5 Restricted CASDA Services

6 Publications and Acknowledgements

7 Links to external documentation

8 Document versions


1 CASDA Overview

1.1 About CASDA

The CSIRO ASKAP Science Data Archive (CASDA) provides the long term storage for ASKAP data products and the hardware and software facilities that enable astronomers to make use of these. Data products are stored at the Pawsey Supercomputing Centre in Perth, Western Australia.

ASKAP is a data driven facility where the data rates are extremely high. In full operations, the ASKAP data rates arriving at the Pawsey Centre will reach around 75 Petabytes (PB) per year. This is beyond the current ability to archive data and so raw data are not archived. Such high data rates require instead that ASKAP data processing is carried out using automated pipelines to produce 'data products' and associated metadata. These are stored and made available through the science archive. The archive can be thought of as the end stage of the full system.

ASKAP Early Science observations are due to begin later in 2016. So that users can gain experience and provide feedback to the CASDA team, CASDA now provides access to demonstration data products obtained during science commissioning and pilot observing programs that have been carried out with a small number of ASKAP antennas from the BETA array.

1.2 CASDA data products

CASDA provides three types of data products:

  1. Calibrated visibility files

    Calibrated visibility files are stored for total intensity and polarisation continuum observations. During Early Science, calibrated visibility data files for spectral line observations will be archived on a best efforts basis. Visibilities are stored in CASA measurement set format.

  2. Images and image cubes

    Radio continuum and spectral line images and image cubes are stored in FITS format.

  3. Catalogues

    As part of the data processing pipelines, detection algorithms are used to search images and image cubes for source detections. Detection parameters such as positions and flux densities are captured in catalogues. The CASDA catalogue registry includes source detection catalogues with parameters determined from continuum, polarisation and spectral line images or image cubes.

1.3 CASDA services on the CSIRO Data Access Portal

CASDA provides data services in two ways: using search interfaces and tools that are accessed through the CSIRO Data Access Portal (DAP), and using Virtual Observatory (VO) Services.

The CSIRO DAP provides access to many archival data products managed by CSIRO across the organisation. These include a wide range of science areas in addition to astronomy. Many customised tools have been added to the DAP to support CASDA. See section 2 for instructions on how to use the CASDA DAP services.

1.4 CASDA Virtual Observatory services

The VO uses standard protocols to enable catalogue or image data obtained from one facility to be easily compared with similar data from other facilities. For example a user might wish to compare source detections from a radio survey with detections from an optical or infrared surveys, or he/she might wish to plot a set of radio source positions on top of an optical image.

For catalogues, we strongly recommend using TOPCAT (Tool for OPerations on Catalogues And Tables). TOPCAT is a freely available program that provides an interface to Virtual Observatory compatible data products. TOPCAT is like a 'browser' that provides access to VO supported data products together with many tools for working with these.

TOPCAT is a richly tooled program with many features that support working with catalogues. In this guide we give some start-up instructions for using CASDA with TOPCAT. For full tutorial and reference documentation please see the TOPCAT website.

As TOPCAT does not currently support images, CASDA provides access to images and image cubes through web-based interfaces. In addition, users can programatically access images using python or other scripts. Image cut-outs, where sub-sections of images or cubes are extracted, can be generated and downloaded.

1.5 User Authentication and OPAL registration

CASDA allows users to search for data and to see what is available in the archive, without any authentication. However, user authentication is required to download data files including calibrated visibilities, images and image cubes.

For science users, CASDA supports two types of user authentication as follows:

  1. OPAL For general use, we recommend using OPAL authentication. To register with OPAL, go to the OPAL Home Page and click on the link to 'Register'. Enter your email address, name, affiliation and a password. The OPAL application will register you straight away.

    OPAL user accounts are self-managed. Please keep your account details up to date. To change user-registration details, or to request a new OPAL password, use the link to 'Log in or reset password'.

  2. CSIRO Nexus authentication is available for individuals who have CSIRO NEXUS accounts.

1.6 Restricted CASDA tasks and project roles

In addition to general user authentication, a small number of CASDA tasks are restricted to archive administrators and/or members of the Survey Science teams. These include setting data validation flags, and/or accessing project data prior to public release. Tasks with restricted access are described in section 5.

CASDA recognises special roles for Administrators and Survey Science team members as follows:

  • Individuals with project administration rights can add and remove individuals from the project team. They assign roles to team members, can enter data validation information and access unreleased data products for their projects. At present the project-administration tasks are restricted to CASDA staff administrators.

  • 'Validators' enter data validation information and access unreleased data products for their projects.

  • Other team members are given access to unreleased data products for their projects but do not have permission to enter data validation information.

1.7 Pawsey Supercomputing Centre user accounts

For users who have user accounts at the Pawsey Supercomputing Centre, CASDA provides fast downloads to the Pawsey Galaxy and/or Magnus supercomputers.

Pawsey accounts are restricted to individuals who are part of science teams that have been granted access to Pawsey facilities through competitive merit allocation processes. For further information please refer to the Pawsey website or contact the CASDA helpdesk.

1.8 Getting Help

CASDA provides user support and documentation in several ways:

  • This User Guide is intended to provide an overview of the CASDA data services and to help new users get started. To get going we recommend trying some of the items described in sections 2 and 3.

  • The CASDA application pages on the CSIRO Data Access Portal provide some 'tooltips'. These are shown as yellow question marks. Click on a question mark to read the tip.

  • Online documentation is available with the DAP . Click on the 'help' link at the top of the page.

  • For enquiries and staff support relating to all CSIRO radio astronomy data archives, including CASDA, please send an email to the helpdesk: atnf-datasup@csiro.au. You will receive an automated email from our helpdesk to acknowledge that your request has been logged. A CASS staff member will reply soon afterwards. We aim to send an initial reply to user queries within four business hours.

    Suggestions for improvements to CASDA tools and to this Guide are always welcome. Please send all comments to the helpdesk.



2 Using CASDA with the CSIRO Data Access Portal

The information in this and the following section is presented as a set of 'how to instructions' using specific data sets as examples.

2.1 Login to the CSIRO Data Access Portal (DAP) using an OPAL or NEXUS account
  1. Check that your browser options are set up to allow pop up windows.
    For example, in Firefox - to allow pop up windows use the browser Options. Then select Content -> and check that the 'block pop up windows' tick box is unselected.
  2. Go to the DAP Home Page
  3. Select the registration scheme in the box headed 'Registered Users'
  4. Enter your account details using either an OPAL or CSIRO Nexus account.
Notes

2.2 Find information about ASKAP Data Collections
  1. Go to the DAP Home Page
  2. Enter a keyword in the green search box. For BETA data products, enter the project code 'AS031' then click on 'Search'. The search will return links to two data 'collections'.
  3. Click on the link to '...catalogues' and have a look at the information provided about the collection.
  4. Note that at the top and bottom of the page there is a link to 'show all'. Click on this to reveal additional information about the collection including the list of individuals associated with the OPAL proposal for this project.
  5. There are two tabs labelled 'Description' and 'Files'. Click on the 'Files' tab. This will bring up a page with a table showing two catalogues associated with AS031.
  6. For more information about a column in the table use the mouse to hover on the column heading.
  7. Go back to the Description tab. On the right of the main screen is a link to 'related' collections. This link is useful for switching back and forth between related collections.
  8. Click on the link to the related collection for images and visibilities and then select the 'Files' tab. You will see now see tabs with tables for visibility and image files that correspond to project AS031. The total number of files is given just above the table.
  9. To see a version of a table that should be easier to read, click on the 'expand' link. The columns can be resized as needed. To return to the original view click on 'close'.
  10. Try switching between related collections and looking at the different tabs to get a feeling for how the data products are organised.
Notes
In general terms, a data collection is a group of similar data files. For CASDA, each ASKAP project has two collections. One collection holds the data catalogues and the other the data product files for images, image cubes and visibilities.
The CSIRO DAP holds data for thousands of collections. You might like to explore the DAP using different keyword searches. Some example keywords are 'pulsars', 'mosquitos' and 'climate change' or try guessing.
AS031 is the (only) project code used for ASKAP BETA science commissioning observations.
On the DAP Home Page, lists of collections can be sorted using the sort options that are shown under the large blue search button.

2.3 Find a persistent link and a Digital Object Identifier for a data collection
  1. Bring up the collections information for AS031 (section 2.2).
  2. See the link near the top of the DAP page to 'copy this persistent link...'. This link provides an indefinite link for the collection.
  3. Cut and paste this link into your browser. It will take you back to the same page. This link can be retained for later use.
  4. See the link to the digital object identifier (DOI) given on the same collections information page. A DOI provides a 'data citation' that allows a user to obtain a set of data corresponding to the date and time the DOI was issued (or 'minted').
Notes For CASDA the DOIs will be updated at intervals of around six months from March 2016 onwards. DOIs are part of the collections information and can be included with journal publications.

2.4 Search for data products using the DAP search form
  1. Go to the DAP Home Page
  2. Click on the large blue button that says 'Domain Search' and then select 'CASDA Observations Search' - this brings up the DAP search form.
  3. Enter some search criteria. For example you could enter 'AS031' as the project code or enter 'BETA' as the project title.
  4. At the bottom of the page check that the option for 'released' data is selected. Then click on 'search'.
  5. You will see a page returned that has up to three tabs for 'Catalogues', 'Image cubes' and 'Visibilities'.
  6. Have a look at each of these tabs to see what they contain.
  7. Note the 'Refine Search' button that takes you back to the search page with your previous search criteria.
  8. On the search form, notice the little question mark next to the frequency range option. This is a 'tooltip'. Click on this to see extra information.
Notes
The search form will return information for all collections and data products that match the criteria. Only tabs that have associated data files are shown.
Selecting a data tab and then clicking on the project number will open the collections information that is relevant to that tab.
'Unreleased' data are only available to team members. Data are released following a data validation process.

2.5 Carry out a cone search using the CASDA Observation Search form

This example finds data products associated with a single cone search in the Tucana region.

  1. Go to the CASDA search form (section 2.4)
  2. In the cone search box enter
    • right ascension = 22:58:30
    • declination = -55:56:00
    • search radius = 1 arcmin
  3. This will return a page with a list of 21 images for the Tucana region that have central positions within this range. Only one tab is seen (for images) as CASDA has no corresponding visibility files or source catalogues for this region.

CASDA can resolve some objects by name, based on Simbad and NED catalogues. Just type in the object name and click on the 'Resolve' button.

Cone searches can also be run for multiple sky positions. Click on the 'multiple positions' radio button. An ascii (text) file can now be uploaded, with one position per line. The search returns the combined results from running multiple cone searches.

Each row of the source files should contain: right ascension (J2000), declination (J2000), radius in arcmin (optional) with spaces between entries. Position separators can be either spaces or colons. Blank lines are permitted. If no radius is included on one or more lines, then a default radius must be entered as the search radius.

Examples of accepted formats are:

  • 22:58:30 -55:56:00 1.0
  • 22 52 26.419 -60 17 57.93
  • 278.555 27.829 1.2
Notes
The online help tip (small question mark) also explains the format for source files.
Source lists are currently limited to 50 rows for Internet Explorer browsers and 100 rows for other browsers.

2.6 Download a single file to an external location
In this example an image covering 150 square degrees of the Tucana region, created by Ian Heywood from data combined over four sub-bands and three epochs is downloaded to your local computer. This approach can be used to download any available data products including images, visibilities and catalogues.

  1. Navigate to the CASDA Observation Search form as described in section 2.2
  2. Select Scheduling Block ID = 1206 and then click on search.
  3. You will see a single tab for image cubes - with a list of associated files.
  4. To select the file corresponding to the demonstration image for this region, select the tick box next to the
    filename = 'SB1206-1207-1229-1231_fullstack_reframed.fits'. (This is near the end of the list)
  5. In the box below the list select the option for 'web' and click on the link to 'retrieve selected data'.
  6. You will now see a new browser page or new tab on your browser with a link to the retrieved data and status information. The page also shows you how long you have left to retrieve the file(s)
  7. The information on this page is refreshed automatically every thirty seconds. When the status changes to green, click on the link to the data file and save it to a local location.
Notes
For this example, the image file size is 113 MB. This may take some time to download across the networks.
For each data file, an additional file is provided with checksum information. This can be used to check the download has fully worked.
The Schedule Block ID (1206) corresponds to the first of four schedule blocks used for this data product. For BETA observations, where more than one observation block is used to create a data product CASDA associates the data file with the first scheduling block (only).
For ASKAP Early Science data products (not yet available), a search for a scheduling block will return all data products associated with the scheduling block.

2.7 Download multiple files to an external location
If you wish to download a number of files, a better approach is to save the links as a text file to your local computer by clicking on the link to 'Save links as text file'. This can then be used together with a script to access the files.

There are many options for downloading files from a list of URLs. Here we describe two methods. These use the unix commands 'xargs', 'wget' and 'parallel'.

  • To download multiple files one at a time: xargs -i wget -c '{}' < links.txt
  • where links.txt is the text file with the list of links.

  • To download several files in parallel: cat links.txt | parallel -j5 --gnu "wget -c {}"
  • where j5 indicates that five parallel streams are used.
Notes
If the Unix commands are not already available then you may need to ask a system adminstrator to install them for you.
For downloading multiple files an alternative approach is to use a download manager. Many options are freely available on the internet. One example, Download Fast is an open source, free multiplatform download manager that supports multithreaded (i.e. parallel) downloading of large files.

2.8 Generate and download a single cutout of an image or cube
In this example a cutout is generated from an image. To begin with we select the 150 square degree image of the Tucana region from Section 2.6.

  1. Navigate to the CASDA Observation Search form as described in section 2.2
  2. Select Scheduling Block ID = 1206 and then click on search.
  3. You will see a single tab for image cubes - with a list of associated files.
  4. To select the file corresponding to the demonstration image for this region, select the tick box next to the
    filename = 'SB1206-1207-1229-1231_fullstack_reframed.fits'. (This is near the end of the list)
  5. Click on the 'create cutout' button.
  6. You will now see a CASDA Cutout Service page. The top portion of the page shows the image / cube information. You can make cutouts using two methods. The first is by providing RA, Dec and frequency values into the form. The second is by using the Aladin-lite GUI to select the region.
  7. Try the Aladin-lite GUI by clicking in the image under 'Enter Cutout Parameters' and selecting a rectangular region. The coordinates of the selected region are automatically filled in the cutout form. We leave Frequency blank in this example as it is a continuum image. Click on Submit.
  8. You will now see a new browser page or new tab on your browser with a link to the generated cutout and status information.
  9. It may take some time to generate the cutout. The information on this page is refreshed automatically every thirty seconds. When the status changes to green, click on the link to the data file and save it to a local location. The page also shows you how long you have left to retrieve the cutout.
Notes
The background image in the Aladin-lite GUI defaults to SUMSS, to show bright radio sources in the area of interest. You can click on the icon in the top left corner of the GUI to select different images (e.g. DSS or WISE).
A checksum file is provided so that the user can double check that the cutout file download has worked.


3 Using CASDA with Virtual Observatory Services for Catalogues

The notes given in this section provide an introduction to finding and working with CASDA VO tables, mostly together with TOPCAT. These are intended to be sufficient to get started but do not cover many of the available tools and features.

The notes given here correspond to TOPCAT version 4.3.

3.1 Install TOPCAT
  1. Go to the TOPCAT downloads webpage and download the full jar file.
  2. You may also need to install a Java Runtime Environment (JRE) if this is not already installed on your computer. If so see the advice and links on the TOPCAT page.
Notes
TOPCAT and Java are freely available. To see which version of TOPCAT you are using open TOPCAT then click on Help > About TOPCAT.

3.2 Find and download CSIRO astronomy catalogues using the VO Table Access Protocol (TAP)
In this example, you will find and download a list of VO catalogues supported by CSIRO and then select a catalogue, casda.continuum_island. This contains parameters for radio astronomy 'islands' detected using source find algorithms for an image of the Tucana region.

  1. Open TOPCAT. You will see a window open up. In this guide this will be called the 'main window'. This window is like a 'command centre' so try not to close it!
  2. Click on the link in the top menu bar to 'VO'.
  3. Click on the drop down option for 'Table Access Protocol (TAP) Query'.
  4. In the Keywords box enter 'CASDA' and then click on 'Find Services'.
  5. Select the service 'CSIRO ASKAP TAP' and then click on 'Use Service'.
  6. You will now see a list of catalogues. Click on the catalogue called 'casda.continuum_island'.
  7. Now enter a simple query to select the all rows and columns from the table 'casda.continuum_island'.
    In the text box near the bottom of the page type:
    • select *
    • from casda.continuum_island
  8. A local version of the table will now appear in the table list seen in the Main Window.
  9. To save a table to your laptop click select a table from the Table List. Then use: file > save Table(s)/Session and follow the prompts to choose an output table format and location etc.
Notes
TOPCAT write options include a range of table formats including VO-table, html, FITS, ascii and csv.
To write out a VO table use the format option = 'votable-tabledata'.
To navigate to a selected location on your computer use the 'Filestore Browser'.
To find all CSIRO TAP services enter 'CSIRO' as a keyword.

3.3 Run simple queries in Astronomical Data Query Language (ADQL)
Here are three simple examples using ADQL to query a catalogue.
  1. Follow the steps above (section 3.2) and use TOPCAT to find the table casda.continuum_island
  2. In the 'ADQL Text' box try out the simple queries shown below.
  3. Click 'Run Query' after each one is entered)
  4. Now click on the 'Examples' button and choose one or more of the basic options. This will auto-enter a simple example query.

    • Example 1: Select the top 1000 rows
      • select TOP 1000 *
      • from casda.continuum_island

    • Example 2: Select first 400 rows and four columns
      • select top 400
      • id, catalogue_id, first_sbid, n_components
      • from casda.continuum_island

    • Example 3: Select all rows with schedule block id = 609
      • select *
        from casda.continuum_island
        where (first_sbid = 609)
Notes
The asterisk in examples 1 and 3 indicates that all columns are selected.
Line returns are optional in ADQL queries.
ADQL queries are generally straightforward to construct. Here is a link to the full ADQL documentation .
The ADQL examples provided in TOPCAT provide an easy way of seeing some of the syntax.

3.4 Try out some TOPCAT Table Icons
  1. Follow the instructions in section 3.2 above and use TOPCAT to add one or more tables to your Table List.
  2. Select a table from the Table List in the Main Window.
  3. Click on each of the three TOPCAT table icons that look like this:

    • - The first of these opens a window with the table data.
    • - The second opens a window with metadata that describe the table.
    • - The third opens a window with metadata that describe the table columns. These include units and descriptions.

  4. Have a look at each of these to get a feel for the data in the table and how this is described.
  5. Now try out the statistics icon . This is an easy way of seeing a quick summary of catalogue statistics.
Notes
Using the mouse to hover over an icon will bring up a description of it.
Note that the icons are arranged in groups. This group of five icons provides diffferent displays of the table data and metadata.

3.5 Create sky plots showing positions of radio detections
This example shows how to do a sky plot for a set of source positions.

  1. Follow the instructions in section 3.2 above and use TOPCAT to download the catalogue CASDA.continuum_island.
  2. In the TOPCAT Main Window: the middle group of icons has five icons for different types of plots:


    Click on the the middle one in this group (sky plotting window). This will plot the positions for 1,578 locations identified as radio continuum 'islands'.

To get started with plots, here are a few suggestions to try out:

- To change the plot centre - use the mouse with a left-click.
- To zoom in and out - use the mouse with a right-click.
- To write a plot to a file - select 'Export' (at top of window) and then select a file format.

Notes
TOPCAT provides powerful tools for plotting table data and allows a considerable degree of customization. In this guide we provide a few examples. For full tutorial and reference documentation, see the TOPCAT user manual .

3.6 Create an x-y scatter plot
This example generates an x-y plot that shows the major axis determined for radio continuum components, plotted against the ratio of the integrated to peak flux densities. Point sizes are scaled using the ratio of the major to minor axes.

  1. Open TOPCAT and run a query using the CASDA TAP service to create a local version of the table casda.continuum_component. (section 3.2).
  2. In the TOPCAT Main Window select the catalogue and click on the plane plot icon .
  3. Select the ‘Position’ tab and enter the expression 'flux_int/flux_peak' for the x-axis and 'maj_axis' for the y-axis.
  4. Set a point type by clicking on the tab for ‘Form’. From the options shown, select the 'Size form'
  5. In the box for ‘Size’ select the field to use as the point type. To illustrate this, set the size to 'maj_axis/min_axis'.
  6. In the same window, use the scale slider to adjust the global scaling of point sizes.
  7. Below the slider see the option for changing colour. You will need to deselect the tick box next to 'By Subset'.
  8. To add a linear fit through the data select the linear fit icon .

    Here are a few further plot options:

  9. To change the plot to a log-log plot select the ‘Axes’ button (on left) and tick x-log and y-log.
  10. To change the labelling for the x- and y-axes, select 'Axes' and then the tab for 'labels'. Deselect the ‘auto’ options and enter your labels.
  11. To add a plot title select ‘Frame’ (on left) then the tab for ‘Title’ and enter text.
  12. To export the plot to a file: Select ‘Export’ in the top bar. The select the output file format and save the file as needed.
Notes
Data subsets can also be plotted and could be used, for example, to show data subsets using different colours.
Note that selected plot options can be turned on and off using tick boxes located to the left. In effect, plots are constructed using a set of layers that can be included or excluded as needed.

3.7 Create a histogram plot
This example shows how to create a plot with two overlaid histograms that show the peak and integrated flux densities for radio continuum components.

  1. Open TOPCAT and run a query using the CASDA TAP service to create a local version of the table casda.continuum_component. (section 3.2)
  2. In the TOPCAT Main Window select the catalogue and click on the histogram plot icon .
  3. Select the 'Position tab' and then select 'flux_peak' from the drop-down list of values for 'X:'.
  4. Select the 'Axes' button and tick 'X Log'.
  5. Select the 'Bins' button and use the slider to adjust the bin width.
  6. To add a second histogram on the same diagram select the histogram button (located below the plot). Select a table and the parameter to plot. For this example the table casda.continuum_components is again used, with X: = 'flux_int'.
  7. You will now see both histograms on the same plot. Labels can be added as described in section 3.6.
Notes

3.8 Create a new column and add to existing table
In this example an extra column is added to a table, corresponding to the logarithm of the peak flux density.

  1. Open TOPCAT and download the catalogue CASDA.continuum_island (section 3.2).
  2. In the TOPCAT Main Window, select the Menu option for 'Views' then select 'column information'
  3. Click on the add icon .
  4. Enter details for a new column. For example:
    • Name: flux_peak_log
    • Expression: log10(flux_peak)
    • Units: mJy/beam
    • Description: flux_peak_log
    • UCD: For now, leave this blank
    • Index:19 (adds as 19th column).
  5. Click OK.
The new column is now added to the same table. To see this look again at the column information or open up the full table.
Notes
If no value is given the column is added as the last column in the table.
Note that column headers and table contents can be edited.
Universal Content Descriptors (UCDs) are part of a formal, controlled vocabulary for astronomical data, provided by the IVOA. The use of UCDs facilitates sharing information. For further information, see UCD1+ controlled vocabulary.

3.9 Carry out cone searches on a catalogue using the VO Cone Search Protocol
This example searches the CASDA table casda.continuum_island for sources within a given radius of a position and writes the results into another VO table.

  1. Open TOPCAT
  2. In the Main Window select 'VO' then 'Cone Seach'
  3. In the Keywords box enter 'CASDA' and then click on 'Find Services'.
  4. Select the 'CSIRO ASKAP Cone' Service
  5. You now need to select the catalogue for the Cone Search (Make sure you select the correct catalogue from the list). In the second box select the row with the description 'Continuum island data'.
  6. Enter some cone search criteria. For the Tucana region try something like this:
    • RA = 340.5 degrees
    • dec = -60.5 degrees
    • radius = 4.0 degrees
  7. Click on OK to run the query
  8. In the main window, you will see a table added to the Table List. This table includes the subset of rows that correspond to positions within the selected search region.
Notes
CASDA currently restricts cone searches to a maximum radius of five degrees If this is a problem for you please let us know.

3.10 Search source detection catalogues programmatically using user scripts
The CASDA TAP service can be used together with user scripts. This can be helpful, for example, to facilitate access to large numbers of files.

In this example we provide a sample python script, tapquery_vo.py that could be easily modified as needed for your specific purposes. This connects to the CSIRO TAP Service and finds a catalogue, casda.continuum_component. A simple ADQL query is used to search the catalogue for radio components with peak flux densities below 600 mJy. The output is written in xml/VO-format to a file data.xml in the local folder.

To run the script:

  1. Download the python script tapquery_vo.py (text file) to your computer
  2. If needed, install python, together with the python modules urllib2 and astropy.
  3. For unix/linux, run the script using 'python tapquery_vo.py.'
  4. To see this table in TOPCAT use open TOPCAT. Use FILE > LOAD and navigate to the file on your local folder.
Notes
You may need to ask computing staff for assistance with installing python modules such as astropy.
For users with Pawsey accounts; astropy is installed on the Galaxy supercomputer in the Pawsey Supercomputing Centre.
For this script the astropy module is used to write the output to a VO-format table. Here is a similar script that can be run without astropy tapquery_csv.py. This version writes the output to a csv file. This can also be read by TOPCAT.

The sample script can be modified to connect to different catalogues and to carry out different ADQL commands. For additional documentation see the internal comments in the script.

3.11 Plot positions of radio detections on an optical image covering the sky region using TOPCAT with Aladin (version 9)
  1. If you don't already have this, download the Aladin desktop application. To do this:
    Go to Aladin desktop and follow the instructions to download the Aladin.exe file
  2. Open Aladin, enter a location (for this catalogue try 22:45:00 -60:00:00) and click on the link to the Digital Sky Survey (DSS).
  3. If you are not familiar with using Aladin, take a moment to try out some of the Aladin options that allow you to navigate around the DSS and use zoom controls etc. For example, you can use the mouse wheel to zoom in and out. Hold the mouse down to pan around the sky.
  4. Open TOPCAT
  5. As described in section 3.2, find the CASDA table casda.continuum-island and add this to your local table list.
  6. In the TOPCAT Main Window, select the table.
  7. Select the Broadcast Table icon (3rd from left in Main Window). This will transmit the CASDA table data to Aladin.
  8. Now switch to Aladin to see the radio catalogue positions plotted on top of the DSS images.
  9. Clicking on a catalogue position on the Aladin plot will list the catalogue parameters below the plot.
  10. To see parameters for a region of sky hold the cursor down to select a region.
Notes
Aladin is an interactive sky atlas from the Centre de Donneées Astronomique de Strasbourg (CDS).
Note the other surveys provided by Aladin Desktop. The same approach can be used with any of these.

3.12 Cross-match information from a CASDA catalogue with a catalogue obtained VizieR and generate a merged catalogue
  1. Open TOPCAT
  2. Select VO - TAP Query and find the table casda.continuum_island (see section 3.2).
  3. In the query box enter the ADQL query:
    SELECT TOP 1000 * FROM casda.continuum_island and run the query.
  4. In the TOPCAT main window: Select VO > VizieR Catalogue Service
  5. Enter the following criteria:
    • Go to the 'Row Selection' part of the screen.
    • In the 'Object Name' enter 'Tucana' then click on 'Resolve'
    • This will add in the RA and Dec for the Tucana region.
    • Set a radius = 10 degrees
    • Use the default Maximum Row Count: 50,000
    • Now go to the 'Catalogue Selection' and select the 'Surveys' tab
    • Select PGC – HYPERLEDA Catalog of galaxies and click OK. This will add a second tale to the Main Window.
    • In the TOPCAT Main Window: Select Joins > Pair Match
    • Enter criteria and set up the match. For example:
      • Max Error: 1 arcmin
      • Table 1: TAP_1_casda_continuum_island
      • RA Column: ra_deg_cont
      • Dec Column: dec_deg_cont
      • Table 2: VII_237_pgc
      • RA column: _RAJ2000
      • Dec Column: _DEJ2000
    • Click Go. This will create a third table using the matched results.
    • Select 'Plot Results' to see a sky plot with the two surveys and the matched results.
    • A matched table is also produced with the combined results. Use TOPCAT to save this if desired.
Notes
TOPCAT provides several options for joining tales and matching positions. For full details please see TOPCAT documentation.


4 Using CASDA Virtual Observatory Services for Images and Image Cubes

For images and image cubes, the Virtual Observatory provides two protocols. The SIAP (Simple Image Access Protocol) is used to 'discover' relevant files, whilst the SODA (Server-side Operations for Data Access) protocol provides the data access.

These services are primarily intended for use with scripts written by science teams using python or other scripting languages. This approach allows a high degree of customisation together with the ability to handle many files at a time.

These protocols may be somewhat difficult to use whilst some knowledge of python is needed to develop scripts. We are happy to help users get started and would welcome any feedback to assist us improve the features described in this section. Please send any comments to ATNF Data Support.

4.1 Find images and image cubes and see access services.
This item describes how to navigate to a list of images and image cubes services. When one image or cube is selected, several data access services are shown. Sections 4.2 and 4.3 provide examples of accessing files and generating cut-outs using python scripts.

  1. Open this CASDA SIAP link in your web browser. You will see a table with a list of all available CASDA images and image cubes.
  2. The following examples include some search filters in the URL:
  3. Click on a link to an image or image cube (given here as the access_url)
  4. A new page is shown with two links. Select the second link and login using your OPAL account.

    This will return a web page with a table showing five rows. These correspond to different actions that can be taken as follows:

    Row What this does Notes
    1 Direct file download Click on the link to download the file directly. This is a quick-access option that is available for filesizes below 50 GB and where the files are available in cache.
    2 Scripted file access This allows highly customisable access full sized images or image cubes using user scripts. Files of any size can be downloaded (from either disk or tape). See section 4.2 for an example.
    3 Scripted cut-out service This allows highly customisable access to 'cut-out' sections from images or image cubes using user scripts. Cut-outs can be obtained from 2-d images or 3-d image cubes. In both cases the first two axes must correspond to sky positions. See section 4.3 for an example.
    4 Simple cut-out service Click on the link to bring up a simple form to use with a single cut-out.
    Cut out images or cubes have a square cross-section. For this form the term 'radius' corresponds to the distance from the image centre to the image edge along the x- or y- axis.
    5 Access through the Data Access Portal This option will be discontinued in the next CASDA release.
Notes
The query string 'BAND = 0.25 0.30' refers to a wavelength range, with units given in metres.

4.2 Use the VO Simple Image Access Protocol for programmatic data access
In this example we provide a sample python script, siap.py, that can be used to download images and image cubes overlapping a particular sky region. Supported formats for right ascension are 22:58:04.88 22h58m4.88s and 344.52039 , and for declination -36:25:39.4 -36d25m39.4s and -36.42758

The script will query the CASDA SIAP service for images that overlap a 0.1 degree radius circle centred on the coordinates provided. All images and image cubes will then be downloaded in full.

To run this script:

  1. Download the python script siap.py(text file) to your computer
  2. Install python and astropy (if not already available)
  3. Run the script using > python siap.py OPAL_username ra dec Destination_Directory

As an example, the following command will download HI images of the galaxy group IC1459. The output files are written into a folder 'output'. For a 0.1 radius, three images are produced.

>> python siap.py OPAL_username 344.52039 -36.42758 output

Notes
This script provides an example that can be customised for your own purposes. Extensive internal documentation is included.

To provide more detail, the script:

  1. Creates the destination directory if needed.
  2. Runs a query to retrieve a list of images and/or image cubes overlapping the sky location.
  3. For each image/image cube it then:
    • Retrieves a datalink.
    • Retrieves the authorised id of the file.
  4. Creates an asynchronous task to access the images and/or image cubes.
  5. Adds the authorised id for each image and image cube.
  6. Starts the task to access the images/image cubes.
  7. Waits until the job has completed.
  8. Downloads all images/image cubes.

To access files, the script uses OPAL authentication and this must be provided for all downloads. You will only be able to retrieve images which have been openly released or that you have access to as a member of a science team.

Please note that the OPAL_Password should be surrounded by single quotes on a mac or linux command line if there are any non-alphanumeric characters in the password.

4.3 Generate cut outs from images and image cubes using a script
In this example we provide a sample python script, cutouts.py, that can be used to generate a set of cut outs obtained from the images and image cubes generated from observations associated with a scheduling block.

The script reads a VO-format catalogue, casda_continuum_component, that includes positions for radio continuum components. For each component with a peak flux density above 150 mJy it generates a small cut-out image from the associated image files where the detected position falls within the full-size image or cube.

The cut out images have sizes of 0.2 x 0.2 degrees in right ascension and declination and contain the same number of planes as is the full sized image or cubes. Thus a cut-out from a full spectral line image cube may have up to 16,200 channels along the third axis. A cut-out from a single plane image will also be a single plane image.

To run this script:

  1. Download the python script cutouts.py (text file) to your computer
  2. Install python and astropy (if not already available)
  3. Run the script using > python cutouts.py  OPAL_username scheduling_block_id Destination_Directory

As an example, the following command will generate cut outs from a radio continuum image of the Tucana region. The output files are written into a folder 'output'. For a flux-cut-off of 150 mJy, 89 cut-out images are produced.

>> python cutouts.py OPAL_username 609 output

Notes
This script provides an example that can be customised for your own purposes. Extensive internal documentation is included. Note that the script includes a switch to retrieve entire image cubes instead of cutouts.

To provide more detail, the script:

  1. Creates the destination directory if needed.
  2. Runs a TAP query to retrieve a list of images and/or image cubes associated with the scheduling block number.
  3. For each image/image cube it then:
    • Retrieves a datalink.
    • Obtains a cut out URL and authenticated id for the image cube.
  4. Create an asynchronous task to access the images and/or image cubes in the scheduling block.
  5. Runs a second TAP query to retrieve the continuum catalogue components of interest.
  6. Adds position criteria for each catalogue entry of interest.
  7. Starts the task to generate a cut-out.
  8. Waits until the job has completed.
  9. Downloads all cutout images/image cubes.

To access files, the script uses OPAL authentication and this must be provided for all downloads. You will only be able to retrieve images which have been openly released or that you have access to as a member of a science team.

Please note that the OPAL_Password should be surrounded by single quotes on a mac or linux command line if there are any non-alphanumeric characters in the password.



5 Restricted CASDA Tools

5.1 Download data to a location in the Pawsey Supercomputing Centre (Pawsey user account required)

In this example CASDA is used to generate a set of data links that are then used to transfer the files to a location in the Pawsey Supercomputing Centre (usually on Galaxy or Magnus). The files can only be accessed from a Pawsey user account.

The following steps are carried out from a Nexus or OPAL account:

  1. Go to the DAP Home Page and login with your OPAL or Nexus account.
  2. Click on the Domain Search and select ‘CASDA Observation Search’ to bring up the search form.
  3. Enter your selection criteria and click on the search button to bring up a list of results.
  4. Select the required products for downloading using the checkboxes on the left of the table.
  5. At the bottom of the page, select ‘Access data with Pawsey HPC account’.
  6. Read the text for 'this statement' and (if appropriate) click the check box ‘To access data at Pawsey, I accept this statement.’
  7. Click the ‘Retrieve selected data’.
  8. You will now be presented with a ‘Retrieve Files’ page showing links to the selected files and corresponding checksums.
    The links shown are only directly accessible from Pawsey accounts.
  9. Wait until the status indicator goes green. You will now also see a link to 'save links as Text file.'
  10. Save this .txt file to your local computer.
    The following steps use a Pawsey account:

  11. Login to Pawsey using your Pawsey account details (for example with 'ssh username@galaxy.pawsey.org.au').
  12. Navigate to the desired path. For the Galaxy supercomputer the path will be similar to: > 'cd /scratch2/askap/username/foldername'.
  13. Transfer the text file you have saved to Pawsey (for example using ftp or WinSCP for Windows)
  14. See section 2.7 for advice on how to access the data files using a text file with links and a script.
Notes
See section 1.6 for information on Pawsey accounts
If you are waiting some time for files to download, bookmark or otherwise save the URL for the webpage with links.
As a note for Windows users: To cut and paste text between files in different locations or between Windows and Unix, i) highlight the text to copy, ii) cntrl-C to copy, iii) right-click to insert.

5.2 Set data validation flags and information (Survey Science Team members with validator permissions )
Data validation is carried out using the CSIRO DAP or Survey Science Projects by members of the Survey Science teams. This task can only be carried out by individuals with validation-level access to the data products.

The validation process involves setting a flag in the data base and adding additional information. Validation is applied at the level of individual data products. So for a set of images created for a project - a validation flag is set for each image. Following validation data products are 'released'. The steps below describe the process:

  1. Go to the DAP Home Page and login with either an OPAL (recommended) or CSIRO NEXUS account
  2. Click on Domain Search and select 'CASDA Observation Search' to bring up the search form.
  3. Enter any selection criteria to restrict the search. For example Project code = 'AS031' will select data products for BETA science observations.
  4. At the bottom of the page select "unreleased data" and then click on Search.
  5. This will now bring up a list of any unreleased data products.
  6. Click on the 'My Admin' tab that you will see near the top-right corner of the page.
  7. Click on the title for the scheduling block of interest.
  8. On the form select any of the indicators that you think are relevant.
  9. Now click on the tab for 'Notes' and add any additional comments (if you wish).
  10. Return to the validation tab, check the details, then click 'Submit'. This will change the validation status to allow the data product to be released.
Notes

For help with the validation process please contact the helpdesk: atnf-datasup@csiro.au.

5.3 See the list of individuals assigned to a science team (CASDA Administrator Task)

  1. Go to DAP Home Page and login with NEXUS account.
  2. Click on the tab for 'My Admin' and then select the last tab on right for 'Data Admin' .
  3. Click on the link to 'Manage CASDA Team Members'.
  4. In the section under 'Manage existing team members' select a project from the drop down list. This will bring up a list of project team members and their current project role.
Notes
In a later CASDA release, this task will later be made available to project-level adminstrators.

5.4 Manage roles for team members (CASDA Administrator Task)
  1. Follow the steps in section 5.3 to bring up the DAP page for managing team members.
  2. In the section for managing team members, select the project.
  3. This will bring up a list of individuals that are recognised as team members in CASDA.
  4. Update this list as needed to update project roles.
  5. To remove an individual from the list select the 'delete?' option.
  6. Click on the 'Update team members' to complete.
Notes
In a later CASDA release, this task will be made available to project-level administrators.

5.5 Add an individual to a science team (CASDA Administrator Task)
  1. Follow the steps in section 5.4 to bring up the DAP page for managing team members.
  2. In the section for adding new members, start typing a surname for the person of interest. A list of matching names will appear. Note that some individuals may appear under two accounts (OPAL and NEXUS). Generally, we recommend using the OPAL accounts.
  3. Click on a name to select the required individual.
  4. Select the appropriate project.
  5. Select the required role.
  6. Click 'Add Team Member' to complete.
Notes
In a later CASDA release, this task will later be made available to project-level administrators.
As a starting point, CASDA generates a list of team members from the PI and co-authors on the associated OPAL project proposals. This list is then managed through CASDA.

5.6 Publish a Science Team catalogue: General advice (for Survey Science Teams)
For some ASKAP surveys, science teams will generate catalogues with final science results. These are referred to here as 'Science Team catalogues'. As an example, a catalogue might contain a set of polarisation properties for a list of detected source. Another catalogue could contain properties derived from HI spectra for nearby galaxies, together with optical identifications. Compiling such catalogues is the responsibility of Science Teams.

CASDA provides a service so that Science Team catalogues can be published as part of the ASKAP science archive. If your team is interested in publishing level 7 catalogues please note the following advice.

  • Catalogues must be provided in a VO-compliant .xml table format.
  • We strongly recommend that before submitting a catalogue for publication you consult with CASS staff for advice on generating VO-compliant tables. CASS staff can help you ensure your tables are VO compliant and can check whether your table is formatted correctly.
  • By publishing a catalogue a Science Team gives approval for the Catalogue to be made openly available through CASDA.
  • Once a catalogue is published it cannot be removed from the archive, or corrected for minor errors. However later versions can be provided, for example as data release 1, data release 2 etc.
Notes
CASDA catalogues are published together with digital object identifiers (DOIs). These can be cited in journal publications.

5.7 Submit a catalogue for CASDA publication (Nexus account required)
  1. Go to the DAP Home Page and login using a Nexus account.
  2. Click on the 'Deposit Data' button at the top of the page.
  3. Read the deposit checklist and then click on the tick box next to this.
  4. Under 'Select Deposit Method' select the option for 'ASKAP Science Team Catalogue Deposit'.
  5. Click 'Start'.
  6. Enter the ATNF Project Code and select 'Create Draft Collection'.
  7. You will now see four different tabs for 'Description', 'Citation', 'Files' and 'Restrictions'.
  8. Look carefully at the information on each tab and edit or add to these as needed. In particular:
    For the description tab - you will need to add at least one 'Collaborating Organisation' and a 'Data Owner'.
    Use the citations tab to modify the list of contributors.
    Use the data tab to upload your VO catalogue.
  9. At any time you can save your entries as a draft. When all entries are complete select 'Publish'
  10. You will now be asked to select from a list of 'approvers'. Choose one of the options and then click on 'Submit'.
  11. Your catalogue will now be parsed through some software to check for VO compliance and the approver will make sure it is an appropriate catalogue for CASDA.
  12. You will be advised if there are any problems with the catalogue. Otherwise this will proceed to publication.
  13. To see the status of collections that you have submitted, go to the DAP Home Page and select the 'Manage My Data' button at the top of the page. This will show your collections together with their current status.
  14. Once published, your catalogue will be available using the CASDA web and VO search tools.
Notes
You may need some assistance for catalogue deposits. Please see the help tips (yellow question marks) and/or contact ATNF data support.

5.8 Approve a Science Team catalogue (Approvers only)
  1. Login to the DAP using an appropriate account
  2. Select 'My Admin' and then 'My Tasks'
  3. For items with a task type = 'Collection Approval', click on the title to open and view the collection
  4. Use the decision tab to enter your decision together with any comments.
  5. Submit your decision
Notes


6 Publications and Acknowledgements

See CASS publications and acknowledgments

7 Links to external documentation

See Data Glossary and Links.

8 Document versions

Author User Guide
Version
Date Latest CASDA version
(release date)
TOPCAT
Version
Notes
M Huynh 1.3 07 Dec 2016 1.4 (Oct 2016) 4.3 Small updates and new sub-section on DAP cutout service.
J Chapman 1.2 01 Jun 2016 1.2 (Apr 2016) 4.3 Many small updates and new sub-sections on generating plots.
J Chapman 1.1 24 Mar 2016 1.1 (Feb 2016) 4.3 Updates and new content in sections 1 to 5.
J Chapman 1.0 05 Nov 2015 1.0 (5 Nov 2015) 4.2 Initial release