From: <stingay_at_email.protected>
Date: Sat, 4 Feb 2006 10:01:26 +1100 (EST)

Dear All,

I'm forwarding this email from Walter Brisken, who visited Swinburne in
January. NRAO seems quite impressed by our new developments and
capabilities, and there seems to be a good chance here for a close
collaboration. I think we should take great encouragement from this and
focus our effort to get our systems fully operational as soon as possible,
to take advantage of our good position.

We should also be holding this exposure up to the "high-ups" in our
respective institutions, in order to get our VLBI/eVLBI projects properly
funded in the long term.

Cheers, Steven

##=====================================================##
Dr Steven Tingay stingay_at_astro.<!--nospam-->swin.edu.au
Swinburne SKA Project Leader
Associate Professor

Centre for Astrophysics and Supercomputing
Swinburne University of Technology
Mail No H39
P.O. Box 218, Hawthorn, Vic. 3122, Australia

ph: +61 3 9214 8758
fax: +61 3 9214 8797

http://astronomy.swin.edu.au/ska
##=====================================================##

---------- Forwarded message ----------
Date: Fri, 3 Feb 2006 13:38:41 -0700 (MST)
From: Walter Brisken <wbrisken_at_aoc.<!--nospam-->nrao.edu>
To: Jon Romney <jromney_at_aoc.<!--nospam-->nrao.edu>
Cc: Claire Chandler <cchandle_at_aoc.<!--nospam-->nrao.edu>,
    Mark Claussen <mclausse_at_aoc.<!--nospam-->nrao.edu>, Vivek Dhawan <vdhawan_at_aoc.<!--nospam-->nrao.edu>,
    Doug Gerrard <dgerrard_at_aoc.<!--nospam-->nrao.edu>, Mark McKinnon <mmckinno_at_aoc.<!--nospam-->nrao.edu>,
    Amy Mioduszewski <amiodusz_at_aoc.<!--nospam-->nrao.edu>,
    Peggy Perley <pperley_at_aoc.<!--nospam-->nrao.edu>, Michael Rupen <mrupen_at_aoc.<!--nospam-->nrao.edu>,
    Lorant Sjouwerman <lsjouwer_at_aoc.<!--nospam-->nrao.edu>,
    Jim Ulvestad <julvesta_at_aoc.<!--nospam-->nrao.edu>, Craig Walker <cwalker_at_aoc.<!--nospam-->nrao.edu>,
    Joan Wrobel <jwrobel_at_aoc.<!--nospam-->nrao.edu>, Barry Clark <bclark_at_aoc.<!--nospam-->nrao.edu>,
    Adam Deller <adeller_at_astro.<!--nospam-->swin.edu.au>,
    Craig West <cwest_at_astro.<!--nospam-->swin.edu.au>,
    Steven Tingay <stingay_at_astro.<!--nospam-->swin.edu.au>
Subject: Minutes of the Australian VLBI discussion

Minutes of the Australian VLBI discussion, 2006 Feb 1

Attended: W Brisken, V Dhawan, D Gerrard, A Mioduszewski, P Perley, J Romney, L
Sjouwerman, J Ulvestad, C Walker

Topic: The state of Australian VLBI, software correlators, and how
this may fit into the VLBA's future. After the 4:00 PM nominal ending
of the meeting three of us (WB, JR, CW) brainstormed until 5:15. The notes
below are not necessarily in time order and cover both parts of the meeting.

1. Software correlators

Swinburne University of Technology in Melbourne, Australia, integrates
closely their astronomy and supercomputer departments making an ideal
setting for software VLBI. They operate several Beowulf-style clusters (meaning
inexpensive desktop PCs running Linux connected via a fast network).
Each cluster consists of 30 to 70 identical computers, most with either two
CPUs, dual-core CPUs, Hyperthreaded CPUs, or some combination there of. They are
nearly finished writing their second software correlator. The first
is an XF style correlator which was the topic of Craig West's master's thesis.
The second is an FX correlator being written by Adam Deller as part of his PhD
thesis. The later is more appropriate for any significant number of antennas.
My visit to Swinburne was mainly for use of the FX
correlator for a pulsar VLBI project that required capabilities no hardware
correlator could offer. The flexibility of the software correlator was
quite impressive to me. It is likely that within this calendar year all LBA
(see sec 3 below) observations will be correlated in software.

Correlation of VLBI data is among the class of "embarassingly parallelizable"
problems. That is intra-node communication is not required so the process
scales very linearly with number of CPUs. They use a logical scheme of load
distribution : All correlation products for a given 10 second interval are
performed on a single node while other 10 second intervals are
being processed on the other nodes. FX correlators have two distinct
phases which scale differently with number of antennas. The 'F' is linear
in number of antennas. For the Swinburne correlator, this antenna based portion
of the processing dominates the processing cost for arrays up to between 20 and
30 antennas, depending on the number of output spectral channels. The 'X' stage
is linear in number of baselines and will inevitably dominate the cost for large
enough arrays. For VLBI, typical array sizes are small enough to be dominated
by 'F', the bulk of which is in the fast Fourier transform.

I did a quick calculation suggesting that about 70 processor nodes could
correlate all polarization products of a 10 station array recorded at 256 Mbit/s
with 1024 channels/IF output and 1 second dump times. The processing cost
changes very slowly with output data rates until they start
getting quite high. Such a cluster would cost today about $80,000.

It was also noted that a software correlator might be ideal for low duty
cycle, session-based VLBI networks such as EVN/Global observations. Such
as system would eliminate need for hardware compatibility among VLBI recorders.

2. NRAO - Swinburne collaboration

There are several VLBA projects, mainly of the low record data rate, high
spectral resolution variety and/or very high dump rates, that could benefit from
the flexibility of a software correlator. Steven Tingay, the head of
Swinburne's correlator group, is quite excited about the possibility of a formal
collaboration. The exact mechanics of such a cooperation would need to be
worked out, but probably we could copy the Mark5 recordings to an
external Firewire or USB disk (which are as large as 2-TB these days)
instead of correlation on the VLBA correlator. Perhaps allowing a few projects
with total recorded data not exceeding the size of an external disk
would be a good start.

3. Australian VLBI hardware

Currently the Australian Long Baseline Array (LBA) consists of
ATCA, Mopra, Hobart, Ceduna, Parkes, and Tidbinbilla. In the near
future a 6m antenna in New Zealand may join the array and the Australian
New Technologies Demonstrator (NTD), an array of about 12 15m antennas
being planned as an SKA demonstrator, would likely participate in L-band
observations.

The Australian recorder unit is a mostly VSI compliant device consisting
of two PC units (containing the Metsahovi VSIB card that receives VSI input
and puts it on the PCI bus of the PC. Two such PCs then feed a single Apple
Xraid device containing two banks of 7 discs. 1 Gbit/sec is the maximum
rate sustainable with this configuration. A redesign of the VSIB card for the
more modern PCI-X bus would likely allow 2Gb/sec recording using a single PC.
In contrast to Mark5 (A or B) this system, like PC-EVN, records data on a
Linux-friendly filesystem in header-less files each 10 seconds long, each
starting on a 10-sec tick. This is quite natural for their software correlator.
The picetag for this system is ~$18,000 US -- comparable with a
Mark5 (A or B) system. The Xraid system dominates the price; cheaper hardware
could likely replace this functionality.

4. How should NRAO proceed?

WB, JR and CW discussed/brainstormed about the VLBA and if any of the above
influences the direction we should take. The VLBA is in a complicated
time period right now -- desire for higher bit rates is strong, much work
would be needed to adapt the VLBA to record and correlate VSI data, and
thus reach 1 Gbps, the Widar correlator is not likely to be VLBI ready for 4+
years, and ability to correlate in software is becoming more practical every
year. Also our "natural" upgrade path to Mark5B is likely to be no more natural
than moving to other data recording systems because of the compatibility between
A and B that has been dropped.

This summer Adam Deller (Swinburne's software FX correlator author) will likely
come to Socorro for the Summer School. We are hoping to have Adam come early
and perhaps do an eVLBI demo using the VLBA antennas. We think this should be
simple for small data volumes. If such a test could be automated it could be
useful for diagnostics.
Received on 2006-02-04 10:01:46