User Tools

Site Tools


Distribution of output data to PIs

Note: Data are now distributed via the Pawsey Data portal. See archiving notes. Below is only for reference in case an alternative is required.

The ftp server is no longer in use. Below are instructions for transfer of data to AARNet's CloudStor+ for distribution.
(Note there are various ways to do this. The instructions below use davfs2 to access your CloudStor+ area via the command line on cuppa. )

Initial set up

If you haven't already done so, you need to set a password for webdav access to CloudStor+ (this is different from your institutional login). To do this, log in to CloudStor+ on your web browser, click on your email address in the top right-hand corner of the page and go to your "Personal" page.

On cuppa21 (logged in to your own account):

You can make your own directory for distributing data to PIs (or geodetic analysts) on cuppa21 under: /data/corr/corrdat/DistributeData/

e.g. /data/corr/corrdat/DistributeData/hayley/ownCloud

In your home area, create a directory called .davfs2 and within that, a file secrets with the line (inserting your credentials): <username> <password>

Make sure this file is accessible only to you, with chmod 600 ~/.davfs2/secrets

Contact someone with superuser access to add you to the davfs2 group:

sudo usermod -aG davfs2 <user>

and to add the following line to /etc/fstab (to mount in your area on cuppa21): /data/corr/corrdat/DistributeData/hayley/ownCloud davfs user,rw,noauto 0 0

Don't let davfs2 use home area for caching:

davfs2 caches files for transfer and by default will use ~/.davfs2/cache for this. This is a potentially fatal problem when trying to transfer large files as the /home partition on cuppa has limited space. The file being copied will be truncated when the /home partition is full (not to mention other problems that causes). To avoid this, create a directory in the data area on cuppa21 (outside of your ownCloud area), e.g.:

mkdir /data/corr/corrdat/DistributeData/hayley/davfs2-cache

Remove any existing ~/.davfs2/cache, and create a soft link to the new area:

ln -s /data/corr/corrdat/DistributeData/hayley/davfs2-cache ~/.davfs2/cache

Mounting via the command line

You should be able to mount your CloudStor+ area with (e.g.) mount /data/corr/corrdat/DistributeData/hayley/ownCloud

You can add the following to your .bashrc (or .mybashrc) if you want to mount automatically on login to cuppa21 (after a reboot etc.)

if [ -d /data/corr/corrdat/DistributeData/hayley/ownCloud ]; then
    mount /data/corr/corrdat/DistributeData/hayley/ownCloud

Distributing data

On cuppa (logged in to your own account on the correlator output data node):

To copy FITS files etc. from the correlator output area to a folder on CloudStor+ (can also just use scp or rsync if preferred): -m VX021A.FITS /data/corr/corrdat/vx021a hbignall@cuppa21 /data/corr/corrdat/DistributeData/hayley/ownCloud/VLBI/vx021a

To copy the pipeline outputs: /data/corr/pipe/vx021a/out hbignall@cuppa21 /data/corr/corrdat/DistributeData/hayley/ownCloud/VLBI/vx021a/pipeline-output

On your local web browser:
Go to your files on CloudStor+. Hover the mouse over the experiment folder to see the option “Share” - select “Share with link” and send the link to the PI in the data release email. Clicking on “Download” for a folder will result in a zip file for download.

As an alternative to the web interface, the entire experiment folder may be retrieved with: wget -O {name}.zip “{shared link}&download”. However note: there may be a (2GB?) file size limit for zip file creation, so this may not be viable for most experiments (suspected due to getting “HTTP request sent, awaiting response… 504 Gateway Time-out” when trying to get a folder of ~2GB, whereas the wget option worked for a smaller folder.)

If your CloudStor+ area is getting close to full (100GB free storage limit), you can move older directories out to .. on cuppa21 and permanently remove from your “Deleted files” on CloudStor+. (Keeping them on cuppa21, provided there is space, means you can readily restore if the PI asks for access again.)

correlator/distribution.txt · Last modified: 2016/04/13 18:29 by cormac