ATNF Scientific Computing Group Review
3 May 2005, ATNF Headquarters, Marsfield
Background
The ATNF Director requested a review of the activities of the ATNF Scientific Computing Group (SCG). The Director raised the prospect of ATUC participation at the ATUC meeting of 9/10 December 2004. ATUC agreed to participate and assist with the organization of the review. Two ATUC panel members and three ATNF panel members were agreed to and subsequently selected. The timing of the review was set to occur soon after the arrival of one of the panel members in
The full terms of reference for the panel are contained in the Appendix A. The program of presentations is given in Appendix B, and the summary of the SCG’s activities are listed in Appendix C.
Review panel
Lister Staveley-Smith (ATNF; Chair), Tim Cornwell (ATNF), Erwin de Blok (RSAA, ATUC), Naomi McClure-Griffiths (ATNF), Steven Tingay (Swinburne, ATUC).
Observers/presenters
Tara Murphy, Dave McConnell, Chris Phillips, Mark McAuley, Vince McIntyre, Malte Marquarding, Mark Calabretta, Neil Killeen (telecon), Dick Manchester.
Summary of procedure
The panel met for approximately 6 hours on May 3 (agenda attached in Appendix B), to address the Terms of Reference for this review (Appendix A). The review panel considered a document prepared by Neil Killeen outlining the activities of the SCG (Appendix C) and heard a number of presentations from group members on these activities during the morning of the review. In the afternoon, the panel formulated the basis for its recommendations. In the days following May 3, the panel put together its detailed recommendations and comments via email.
Executive summary
The panel recommends:
- That the SCG is a valuable part of the ATNF and should be maintained with critical mass.
- The core business of the SCG should be:
- Maintaining and developing existing data reduction packages.
- Acting as a coordinating body for ATNF software standards and data archiving.
- Strategic algorithmic (for xNTD, SKA, etc) and standards (e.g. WCS) development.
- That the development activities of the SCG should be better defined and, where possible, better aligned with the core activities of the ATNF, via the now established ATNF project management structures.
- That the level of strategic development of software for large projects such as the xNTD be increased significantly over the next 12 months.
Detailed recommendations and comments from the review
Before addressing the specific questions posed as part of the Terms of Reference, the panel have a number of broader comments which set the context for our respones to the specific questions:
- From the presentations made during the review, it was apparent to the panel that the work of the members of the SCG is of a very high standard, is well regarded within the ATNF user community, and is acknowledged internationally. The group members are to be commended for their application to their assigned tasks.
- The panel concluded that the SCG is a valuable entity within the ATNF and that this group should be maintained with critical mass, composed of people that have skills and experience in both the astronomy and computing domains. This will be particularly true over the next 5 years as projects with large software components such as the CABB and xNTD are undertaken. The advantage of the SCG is that it forms a competent technical and practical resource that can be drawn upon by ATNF projects, as well as a self-contained strategic resource for the ATNF. How this group should sit within the overall ATNF organisational structure is best addressed by ATNF management.
- The panel was concerned that, while the output of the group was very good, the scope of the activities undertaken was in some ways poorly defined and could be better aligned with the focus of other ATNF activities. The panel discussed the purpose of the SCG within the ATNF, in particular looking forward to future large software projects. The panel concluded that the purpose of the SCG was to focus on activities that contain a significant research and development component and that support core ATNF objectives. The panel suggest that the core activities of the SCG should include:
- Maintaining and developing existing data reduction packages, in particular extending algorithms for existing and future ATNF observing facilities.
- Playing a coordinating role in implementing software standards and policy across the ATNF, for example ensuring that ATNF data archiving policies are implemented uniformly across all data types (ATCA, single-dish, continuum, spectral line, pulsar etc).
- Strategic algorithm development aimed at near term (e.g. xNTD) and far term (e.g. SKA) instruments.
- Strategic standards development (e.g. WCS).
Of the activities listed in the summary document, the panel felt that some activities did not fit the core purpose of the SCG and should be transferred to ATNF Operations. These activities are: telescope control and monitor (ATOMS, TCS); compute facility.
In addition to the core activities that the SCG should be involved in, the panel discussed the interaction between the SCG and the ATNF project management process. The panel felt that, in addition to the core activities of the SCG, that SCG members should be available to participate in project-based activities, as required. The panel felt that a clear
demarcation between SCG core activities and ATNF project activities would make resource management more transparent. In addition, since the aim of the SCG is research and development orientated, the SCG members, where appropriate, should be given a fraction of their time for self-directed research. This should be accounted for in any resource management model applied to the SCG.
We now list our responses to the specific questions posed by the Terms of Reference:
Operations
1. Is the ATNF producing telescope interfaces and observation support software to a high-enough standard?
In short, yes.
The panel note that with regard to telescope interfaces, virtually all of the development and maintenance comes from Narrabri staff (and Mike Kesteven) in the case of the ATCA system and from Parkes staff and Mike Kesteven (not an SCG member) for the 64m. The only Marsfield-based support of telescope interfaces from SCG staff is the 0.05 FTE contribution to TCS from Mark Calabretta. The panel therefore recommend that software effort supporting telescope interfaces cease to be considered an SCG activity and responsibility for telescope interfaces be counted under Observatory operations.
2. Is the current (non)-uniformity of telescope interfaces between different ATNF telescopes a significant User issue?
No, this issue has never been raised by Users as a significant impediment to their scientific productivity.
3. Is the ATNF providing well-maintained software of a sufficient standard to facilitate rapid off-line reduction of data? Are there significant problem areas?
Apart from minor maintenance issues that Users raise from time to time, the only significant recent problem in this area was with single dish spectral line reduction software. Users raised a number of difficulties with SPC and have called for a replacement package over the last few years. Development of the new ASAP package was undertaken to address this User request and the software is now almost ready for general release. The uptake of ASAP over the next 12 months should be monitored to determine if the objectives of the SCG and the ASAP project have been met.
The other major data reduction packages are very mature, reflecting the mature nature of the instruments they support, for example MIRIAD and the ATCA. MIRIAD is a good example of a package that has worked very well over a long period of time, absorbing changes to the instrumentation such as the recent millimetre upgrade of the ATCA. The panel felt that maintenance and development of off-line data reduction software was something that the SCG did very well, in general. To ensure that this high level support of existing facilities continues, the panel felt that development of off-line data reduction software should be a core activity of the SCG.
A significant issue going forward in terms of off-line software is the distinct lack of concrete thinking about the evolution of this software for future facilities. There is a danger associated with the fact that MIRIAD and other packages are mature and require a relatively small amount of support. The danger is that the evolution (or replacement) of these packages for the support of instruments like the xNTD are not receiving the attention that they warrant. The panel therefore felt that the strategic development of data reduction algorithms should be a core activity of the SCG. This development may need to be directed from within the SCG since existing project plans may not be able to fully identify the software requirements for these projects.
A further issue going forward is the fact that MIRIAD does not support 64-bit operating systems and significant development work may need to take place if MIRIAD is to be used on new systems.
4. Are there important WWW issues which are of major concern to Users?
Apart from a relatively small area of development in web services, the major effort of the SCG on the WWW front has gone into the production of data archives, the ATCA imaging pipeline, and some visualisation software such as RVS. This effort can be considered as focussed on the provision of an ATCA archive and associated interfaces. The panel note that the ATCA archive has been warmly received by Users and will be of great benefit to the ATNF in years to come. Those involved should be congratulated.
In as far as the archive can be considered a WWW application, the panel expressed concerns with some aspects of the development of this project. First, the development of RVS, while it is an interesting technical undertaking, is not obviously a development that is going to have a benefit to the ATNF and its Users beyond incorporation in the image and rpfits archive applications. This is especially true since the overarching justification of the RVS project, the Virtual Observatory, has a very uncertain future in
The panel recommends that RVS be in effect frozen in a state that will allow it to be used as part of the ATCA online archive/pipeline system. Further, the panel recommends that software developments for data visualisation be highly integrated into projects that support the core activities of the ATNF.
The panel also noted another issue of relevance to data archives and the WWW, that the ATCA archive project, while having delivered a great product, was perhaps too narrowly focussed. This is illustrated by the fact that in parallel and completely independently to the ATCA archive project, the development of Parkes (pulsar and spectral line) data archives is taking place in other ATNF groups. The panel felt that in the interests of uniformity and efficiency (and with the benefit of hindsight) that a better outcome could have been achieved if the high level policy on ATNF data archiving (all data types) had been passed to the SCG, with the responsibility of developing and enforcing a uniform method for archiving and archive interfaces. While data archives should be made available for all ATNF data types, and uniform interfaces are an issue for usability, the deeper issue here is the procedure for the implementation and delivery of products relevant to ATNF policy (that fall under the purpose of the SCG).
The panel thought it reasonable that this type of coordination and responsibility for implementation of high level policy (where appropriate in the software arena) be considered a core activity of the SCG.
5. Is there an adequate and timely response to software problems affecting the acquisition and reduction of data?
In general terms, any problem (hardware or software) that immediately affects the acquisition of data with ATNF facilities is addressed very promptly, either by members of the SCG or operational groups. Users greatly appreciate this and it is a strength of the ATNF, through their large number of dedicated individuals. These problems can be raised through a number of channels, most often through fault reports at the telescope.
As mentioned above, in general, problems with off-line software are well addressed, when they do occur. The timescale for resolution of the problem varies with the problem and since reduction of data is not as time-critical as data acquistion, timescales may be significantly longer. Users are not aware of major problems in this area. Again, the panel felt that this was a good reason for including maintanence and development of off-line software as a core activity for the SCG.
Strategic developments
1. Does the ATNF invest too much/little in algorithmic development and/or blue-sky software development?
At this point in time the panel felt that, as a proportion of resources, the investment in blue-sky software could be described as appropriate. The panel expressed some concern that the blue-sky software development that has been taking place over the last two or three years could have been more closely directed towards outcomes. While that statement could be taken as contradictory (blue-sky research should not be directed to any particular outcome), the panel viewed this question from the context of major projects that are on the ATNF horizon: eVLBI, CABB, xNTD. It was clear to the panel that the research and development effort within the SCG will need to ramp up considerably to address the challenges of these new projects. In these cases, these large projects have reasonably clear design and science goals which will require a large supporting software effort. At this point the software effort required is not well understood. What is certain however, is that new algorithms will need to be developed to deal with the high data rates, bandwidths, and fields of view that these instruments will provide. Blue-sky software research within the SCG should be directed into those areas that are likely to benefit these projects.
The current level of strategic or blue-sky software research will therefore likely be unacceptable 12 months from now, far more will be required. As stated above, the panel felt that this would be a natural core activity for the SCG.
2. In the past, the ATNF has developed highly-successful image display and analysis tools (karma, viewer, moment). Should the ATNF continue developments in this or similar niche areas?
It is certainly true that, in the past, the ATNF has produced a number of visualisation tools that are highly valued by users. Visualisation is an historical strength of the SCG. However, as mentioned above, the panel felt that further development of visualisation software must be tied to specific eventual goals that have a tangible benefit to ATNF facilities and Users. This can be addressed via specific project plans or via strategic software development. However, as with any major software efforts, the development of visualisation software must be somehow reviewed in the context of the core goals of the ATNF.
3. There will be a strong pressure for the ATNF to devote a large and sustained software effort to make the xNTD a success. Given the limited resources, does the panel have a recommendation on the relative resources to allocate to xNTD versus normal operation of existing AT facilities?
The xNTD is the biggest and most challenging new initiative for the ATNF and it is important that it is a success, not only as a User facility in the near term, but as a demonstration of technology supporting
The panel list support of software for existing facilities as a core activity of the SCG. The panel recommend that this support should not be compromised by future developments, including the xNTD. The panel recommend a rearrangement of some existing SCG activities (e.g. moving telescope interfaces and compute facility to operations) and effective termination of other activities (e.g. freezing RVS at its current state) that may free up some resources. These resources should probably be moved to xNTD-related strategic research.
It was apparent to the panel that the size of the SCG would probably need to increase to support xNTD developments (in addition to any resource claims from other large projects). The panel therefore recommend that the relatively small amount of time currently spent by the SCG on successful support of software for existing facilities be retained. New resources available in the SCG (via rearrangement or new hires) should be primarily directed to the xNTD.
Management
NB. The panel was happy to stick with dispensing high-level advice and therefore only lightly touched on management issues which are responsibility of the ATNF. There was one important exception to this.
1. In its role as a National Facility, how careful should the ATNF be in separating operational and strategic software development?
Probably more careful than it has been in the past. The current model has worked well for a long period of time, mainly because the supported systems have been relatively stable. Looking to the future, management of strategic developments will be more important and will need to be addressed more closely. The panel noted from the summary document that it was far from clear what the breakdown of effort was over the various SCG activities, making it somewhat difficult to review the level of resources that have been expended on various projects, in turn making it difficult to objectively determine if the projects have been successful.
The panel have made some recommendations for removing certain activities from the SCG and placing them in operational categories. Clearly delineating operational and strategic software should make the allocation and use of resources more transparent and trackable.
The panel noted that the first example of a pure software project initiated and brought to completion under the new project management process, the ASAP project, is probably the best understood of the SCG activities summarised. In this case there will be a good chance that the goals of the project, compared to the effort expended on it, can be
assessed.
The panel felt that as more software projects progressed through the project management process, this would become more common.
2. Managing a multi-tasking, multi-site group is challenging. Does the Review Panel have any specific recommendations regarding human resource management?
Broadly the panel's recommendations have tended to move the SCG toward strategic research more closely aligned with overall ATNF goals, along with other core activities which play a central role in the development of existing facilities. As such the panel feel that SCG group members should be able to feel clear about the activities that they are working on, core SCG activities, work as part of other ATNF projects, and any self-directed research. As a highly skilled group, this would include having the opportunity to contribute to the direction of software projects, particularly considering the long term nature of some software. The panel felt that, where appropriate (with astronomy PhD graduates for example), a fraction of an FTE would be available for self-directed research.
3. Considering the size of the ATNF, does the panel notice any significant over or under-resourcing of the areas under review?
As mentioned above the current model has worked more or less adequately for a long time at the current resource level. As also noted above, in the near future a much larger level of resources is likely to be required in the area of scientific computing, to address several large and challenging projects over the next few years.
The panel felt therefore felt that the area of strategic software development directed toward these large projects is likely to be significantly under-resourced in the near future.
APPENDIX A
TERMS OF REFERENCE
A review has been requested by the ATNF Director and Projects Review Board to advise on the current and future priorities for the Scientific Computing Group and its management.
The format of the review follows.
- A review panel will comprise ATNF staff and representation appointed by ATUC, with the ATNF staff in the majority. The membership is:
- Lister Staveley-Smith (ATNF, chair)
- Steven Tingay (Swinburne, ATUC)
- Erwin De Blok (RSAA, ATUC)
- Tim Cornwell (ATNF)
- Naomi McClure-Griffiths (ATNF)
OPERATIONS
- Is the ATNF producing telescope interfaces and observation support software to a high-enough standard?
- Is the current (non-)uniformity of telescope interfaces between different ATNF telescopes a significant User issue?
- Is the ATNF providing well-maintained software of a sufficient standard to facilitate rapid off-line reduction of data? Are there significant problem areas?
- Are there important WWW issues that are of major concern to users?
- Is there an adequate and timely response to software problems affecting the acquisition and reduction of data?
STRATEGIC DEVELOPMENTS
- Does the ATNF invest too much/little in algorithmic development and/or blue-sky software development?
- In the past, the ATNF has developed highly successful image display and analysis tools (karma, viewer, moment). Should the ATNF continue developments in this or similar niche areas?
- There will be strong pressures for the ATNF to devote a large and sustained software effort to make the xNTD a success. Given limited resources, does the panel have a recommendation on the relative resources to allocate to xNTD versus normal operation of existing AT facilities?
MANAGEMENT
- In its role as a National Facility, how careful should the ATNF be in separating operational and strategic software development?
- Managing a multi-tasking, multi-site group is challenging. Does the Review Panel have any specific recommendations regarding human resource management?
- Considering the size of the ATNF, does the panel notice any significant over or under-resourcing of the areas under review?
APPENDIX B
AGENDA
09.30 Welcome and review of Terms of Ref (LSS)
Overview (VJM)
- Software at the ATNF
- The Scientific Computing Group
- Visitor services
- Software support
Coffee
10.45 Virtual Observatory (TM,MM)
- Overview
- Pipeline demo
- RVS demo
- Web services
11.30 Pulsar software (RNM)
- overview
- Archive interface
12.00 TCS/Multibeam (MRC)
12.15 Discussion
12.30 Lunch
ASAP (MM)
14.30 Review Panel Discussion
closing discussion with DMcC and VJM
LSS - Lister Staveley-Smith
MM - Malte Marquarding
MRC - Mark Calabretta
RNM - Dick Manchester
TM - Tara Murphy
VJM - Vince McIntyre
DMcC – Dave McConnell
APPENDIX C
ATNF Scientific Computing Review
Neil Killeen
February 2005
Introduction
ATNF computing and software activities are undertaken by a wide range of people. There is no ‘ATNF Computing Group’. Instead, each site is quasi-autonomous, with coordination from the Observatories Computing Committee and other communication channels.
At Marsfield, most of the National Facility software effort comes from the Engineering Group and the Scientific Computing Group. The former builds correlators and real-time systems. The latter builds post-correlations data processing and support software. However, there is plenty of other software activity at Marsfield, most notably the Pulsar science group.
We have divided software and computing activities into ‘Essential’ and ‘Non-Essential’. The former are things that are required for astronomers to do the basic science from ATNF telescopes. The latter are things that add significant value, but are not crucial to do the basic science goal.
It is a fair general statement to say that activities on Essential items are largely undertaken by permanent staff, whereas activities in the ‘non-essential’ category are largely undertaken by post-docs or positions funded through the ARC. Funding used for the ‘non-essential’ category is generally only available for that category. There are of course overheads and resources needed from permanent staff to support these latter activities.
This document largely excludes activities from the ATNF Engineering Group.
Essential Components
Telescope Control and Monitor
ATOMS - Software to control ATCA. Maintained and developed at Narrabri with ** FTE
TCS - Software to control Parkes (and Mopra). Developed and maintained largely at Marsfield (Kesteven and Calabretta), but also with effort from Parkes. ** FTE (Kesteven), 0.05 FTE (Calabretta), ** FTE (Parkes)
Data Acquisition
VLBI
Data Processing
Miriad - Main package used to process ATCA data. Maintained at Marsfield and Narrabri. Recent developments have largely been around changes for mm calibration and large file support (provided by Narrabri). FTE 0.05 (Marsfield), Narrabri (0.05 FTE).
AIPS - Package used by some to process ATCA data. Provided by NRAO
AIPS++ - The AIPS++ international consortium was disbanded, and has been replaced by an MOU between interested parties (NRAO, ATNF, ASTRON) which supplies coordination through regular meetings and email contact (essential). ATNF does little direct AIPS++ development, but uses AIPS++ heavily in its other software projects. 0.1 FTE (
Multibeam - This software is used to process Parkes (and now
SPC - This software is used to process single-beam single-dish data. Maintained at Marsfield. 0.025 (Calabretta)
ASAP - This software is being developed to replace SPC for single-beam single-dish analysis. It is being built with the AIPS++ C++ toolkit bound to Python. Developed at Marsfield. 0.8 FTE (Marquarding), 0.1 FTE (Phillips), 0.2 FTE (
Pulsar - Pulsar data analysis software has always been provided very successfully by the pulsar community itself. ** FTE (
Data Visualization
ATNF has a long history in astronomical data visualization (e.g. Karma software, supported by Richard Gooch, used all around the world). At present, our only activity in direct data visualization is in the ‘Non-Essential’ category.
Algorithm Research
During the early years of development of the ATCA, ATNF (largely through the efforts of Bob Sault) was very active in synthesis processing algorithm development. As the instrument and techniques have matured, our effort here has diminished.
Wideband imaging - Since Bob Sault’s research in the 90s, we have done nothing. With the upcoming new broad-band receivers for Narrabri, new effort is required here as the algorithms don’t exist (provided there is a Scientific case for them).
Widefield imaging - The ATCA has been well served by Bob Sault’s mosaicing algorithms in Miriad. With NTD and xNTD taking us into a new space with multi-beam focal-plane arrays, activity needs to begin again here.
Multibeam analysis - We do some ongoing algorithmic development for multi-beam processing. This largely revolves around specialized robust band-pass algorithms in the face of RFI or too much signal (galactic plane). This is really included in Calabretta’s work in multibeam processing.
RFI mitigation - Some substantial effort has gone on here over the years in the research space for RFI mitigation. ** FTE (Mitchell), ** FTE (Kesteven), ** FTE (Sault), ** FTE (Ekers), ** FTE (Upall/Manchester)
I am not sure what fraction of this research has been captured in production software (perhaps the pulsar people are the only ones to do so).
Compute Facility
Each observatory provides a compute facility and IT resources with the Marsfield facility holding the most resources. These are an essential part of our service ATNF and visiting astronomers who process their data here. We also provide an interface to CSIRO IT for our staff and visitors. 0.8 FTE (McIntyre), 0.2 FTE (
Non-essential Components
Standards
Over the last decade, ATNF has contributed to international standards work through Mark Calabretta’s (and international collaborators) work on World Coordinate Systems. New internationally ratified WCS-FITS standards, so far two published papers, and reference software is the output of this. ** 0.3 FTE (Calabretta)
Virtual Observatory
ATNF is a founder member of Aus-VO (formed 2003), funded through ARC/LIEF (with matching contributions from institutes). Aus-VO failed to gain funding for 2005. Its future is uncertain.
There is a huge amount of effort going into VO infrastructure around the world. Aus-VO, and ATNF in particular, have focussed more on the application space, as it is sparsely sampled at the moment.
Remote Visualization System
This project has run for 2 years, finishing May 2005. Initially it explored a generic application design for VO and then demonstrated that with RVS. RVS is designed to deliver imaging capability of data from archives without the user having to download the images (which can be very large in radio astronomy). RVS has been deployed in extant ATNF archives and is being made available to other archive providers. RVS is built with the AIPS++ display library, Java, SOAP, and Corba. 2 FTE years (Chandra), 1 FTE year (Tokachichu), 0.1 FTE (Marquarding)
ATCA Pipeline
This project (initially in collaboration with the CSIRO ICT Centre) is developing a pipeline to process ATCA data. The pipeline (built with the AIPS++ synthesis modules) has an automatic mode which is attached to the ATCA on-line data archive allowing users to make a quick inspection of processed potential data and display it with RVS. Current activity is to provide it with a GUI to allow uses to use it to process their own data with more control over the parameters. The pipeline will also be deployed at Narrabri to services on-line imaging during observations.
2 FTE years (Murphy), 0.1 FTE (
Web Services
Another contribution we are making is to provide some basic web-services; these can be called from an HTML page we have provided, but also by other software processes. The services we have provided offer things like coordinate conversions, velocity and frequency inter-conversions. The functionality is provided by the AIPS++ toolkit and serviced via GSOAP. 0.3 FTE years (Davidson [Summer Student] and Murphy), 0.05 FTE (
Data Archives
Provision of data archives can be thought of as a VO activity, but also stands on its own.
ATCA Online Archive
The ATCA archive is now available online. The initial stages of this project were a collaboration with the ICT Centre) was to bring the ATCA archive online. This is now operational. The archive is integrated with the automatic ATCA pipeline and RVS. 0.3 FTE year (Owen, McIntyre)
Image Archive Infrastructure
We are developing some infrastructure so that it is easy to deploy new image archives. Currently, PIs of projects have put their own together. This infrastructure will provide a common HTML front-end (tunable via style sheets), and a common back-end (provide through AIPS++) which fetches images from FITS archives. At the same time, the back-end will also be able to provide the data for IVOA data-access protocols (Simple Image Access and Simple Spectral Access). 0.1 FTE years (Tokachichu)
Pulsar Archives
Something from RNM and JER
Algorithm Research
Automatic Source Finding
A long-standing and difficult area is that of automatic source finding and characterization. We are just beginning a project to work in this field (1-, 2- and 3-dimensions) to establish robust and efficient algorithms to service the growing number (and size of) of image archives being generated. Ultimately these algorithms will also get integration with the VO space. 3 FTE years (Whiting)
Marsfield Scientific Computing Group
McIntyre (Acting Group leader; permanent)
Amy (permanent; presently on secondment from CSIRO IT)
Calabretta (permanent)
Marquarding (permanent)
Cornwell (permanent; arr April 2005)
Murphy (CSIRO Emerging Science postdoc to Nov 2006)
Whiting (CSIRO Emerging Science postdoc starting May 2005)
Chandra (ARC/LIEF, 2 years to May 2005)
Tokachichu (ARC/LIEF, 1 year to May 2005)