Assessment techniques in an online astronomy course

Margaret Mazzolini, PASA, 17 (2), 141

Assessment Techniques in an Online Astronomy Course

Margaret Mazzolini(1)

(1) Centre for Astrophysics & Supercomputing,
BSEE, Mail Box 31, Swinburne University of Technology,
PO Box 214, Hawthorn VIC 3122 Australia
(E-mail: mmazzolini@swin.edu.au)

.Keywords: astronomy education

 

Abstract

Teaching an astronomy course online involves many challenges. One major challenge is the design of appropriate, practical and challenging forms of assessment which focus both on the astronomy content of the course and on the particular astronomy interests of the enrollees.

This paper will discuss the assessment approaches we have chosen for our introductory-level subjects, the outcomes obtained and the reactions of the course participants. In particular, some of the wide variety of project work undertaken by the participants will be described.

1. Overview

Swinburne Astronomy Online, an online Graduate Certificate in Applied Science (Astronomy), commenced in Semester 1, 1999 with 2 introductory-level subjects (Exploring the Solar System, Exploring Stars and the Milky Way) which assume no previous knowledge of astronomy or physics.

In Semester 2 1999 and Semester 1 2000 we added a three more subjects: another introductory-level subject (History of Astronomy) and two more advanced subjects (Exploring Galaxies and the Cosmos, Theories of Space and Time). Further advanced subjects are being developed for delivery in the second half of 2000 and beyond. The more advanced subjects assume familiarity with introductory, tertiary-level mathematics, whereas the general-interest subjects are open to all.

We offer open entry for single-subject enrolment in the general-interest subjects, but graduates wishing to obtain the Graduate Certificate must successfully complete 4 subjects. From Semester 1, 2000, we are also offering Master’s (12 subjects) and Graduate Diploma (8 subjects) degrees.

Our participants are drawn from anywhere in the world (23 countries in Semester 1 2000): the only requirements are that they have regular access to a personal computer with a CD ROM player and an Internet connection, and have sufficient English-language skills to cope with a course conducted in that language. They are "mature-age" (average age in mid-40s); highly motivated individuals who want to learn about professional astronomy rather than become professional astronomers. They may include amateur astronomers (who at present represent the majority of our participants), secondary teachers, members of the general public with an interest in astronomy, students at other institutions without astronomy courses, and individuals working in astronomy-related fields.

We use a hybrid CD ROM/Internet delivery. Participants are provided with custom-made multimedia material on CD-ROM, in the form of "Activities" intended to provide overviews of the course content, motivate learning, and provide simulations of astrophysical processes. A prescribed textbook complements the multimedia course content on the CD ROM. The Internet is used for asynchronous newsgroup discussions that form a central part of the course, and also to access astronomy sites.

2. The Assessment Mix

Before assessment tools were chosen, it was important to decide what sorts of learning outcomes were intended in this course. Unlike conventional graduate astronomy courses aimed at teaching the basics of professional astronomy, this course is designed to teach participants about astronomy, rather than to prepare them to be professional astronomers. The course also differs from conventional conceptual astronomy courses offered to non-astronomy majors at undergraduate level, because our participants are typically mature-age students and most already have tertiary qualifications. Typically many of our participants are capable of researching and discussing topics in astronomy at a similar depth, but not at a similar mathematical level, to what one would expect to find in a mainstream graduate astronomy course.

Given the interests and backgrounds of the participants (largely amateur astronomers plus educators), it was decided to use a range of appropriate assessment tools intended to encourage active learning in order to assess

  • overall conceptual understanding,
  • communication skills about astronomy, especially to non-specialist audiences,
  • thorough learning across the full range of the course content,
  • ability to research a topic in depth, and
  • performance on a major project chosen to complement the interests of participants.

For the two introductory-level subjects which were offered in Semester 1, 1999 for the first time, the assessment mix was made up of assessable newsgroup contributions, a ‘misconceptions assignment’, computer-managed testing, an essay and a project. The testing mix does not include traditional examinations. A similar assessment approach will be used as much as possible for the other units, but modified where necessary to suit the particular outcomes sought from some of the advanced units.

2.1 Assessable newsgroup contributions

Participants are required to make at least two postings to the subject newsgroups in every two-week period. One posting should be a question or comment about the course content for that two-week period, and at least one other posting should be an answer to another participant’s query. In this way, participants answer each other’s queries after researching the answers in the course material, textbook and from the Internet. Instructors intervene where necessary, or where they can add interesting insights, but the instructor’s role is as a ‘guide on the side’ rather than as a ‘sage on the stage’. At the end of each semester the participants themselves nominate which of their newsgroup contributions should be assessed. So far the depth of the discussions on the newsgroups has been on the whole extremely impressive. Both as an assessment tool and as an active learning technique, the newsgroup contributions have been a great success and received very positive feedback from both participants and instructors, though a small minority of participants found the perceived level of expertise of others in the course somewhat daunting.

2.2 Misconceptions assignment

At the start of the Semester the students were given a set of statements about fundamental concepts in astronomy, and asked to indicate which statements, in their opinion, were true and which were false. These included statements like "There is no gravity in space." The marks for this pre-test did not contribute to the final assessment, but the results of the test were very useful in informing the instructors on the average state of initial knowledge of the student group. In contrast to undergraduate student groups (including both science and non-science majors) who took a longer version of the test at La Trobe University in 1997 and 1998, the Swinburne online participants in 1999 scored very highly on this pre-test, which probably reflected the fact that most of them were amateur astronomers.

As a follow-up, at the end of the semester the participants were given some of the statements again, in a "Misconceptions Assignment", again asked to state which were correct, but this time asked to provide corrected versions of incorrect statements.

This is a modified version of a technique that has been used with success at La Trobe University to encourage students to correct initial misconceptions over the period of instruction by offering them the ‘carrot’ of being able to score marks for giving corrected versions at the end of semester. In this case, however, it backfired: most of the participants in the online course knew the correct answers in the first place, so using the Misconceptions Assignment as the follow-up turned out to be largely a waste of time! We will continue to pre-test our participants, because the information it gives us on the background knowledge of each new group of students is valuable for our teaching, but we won’t follow it up with the Misconceptions Assignment in future.

Better pre-tests than our home-grown one are available (Sadler 1992, Hufnagel, Slater, Deming and Adams 1999) and we intend to trial the use of the Astronomy Diagnostic Test (Hufnagel, Slater, Deming and Adams 1999 for our introductory subjects.

2.3 Computer-managed testing

As well as encouraging participants to study particular topics in depth (e.g. for the essay and project), the assessment mix should reward participants who work through all the course material conscientiously. We require participants to answer two or three computer-managed tests during the Semester. These are multiple-choice tests designed to cover the range of topics in the course material but also to emphasize conceptual understanding over rote learning. Of necessity, these Internet-delivered tests are ‘open-book’, and so it is particularly important to choose questions which emphasized conceptual understanding, because it means that they could not be answered by simply flicking through the index of the text without any prior understanding of the material. Typically, the participants again have done very well in these tests, as might be expected, so these tests were not very discriminating. However they did serve the useful function of encouraging thorough learning across the full range of the course content.

2.4 Essay

Given the backgrounds and interests of most of the participants, we have felt that a major aim of the course should be to develop their skills and confidence in communicating astronomy concepts to non-specialist audiences (e.g. fellow amateur astronomers, secondary school students, newspaper & magazine subscribers). We also wish to encourage synthesis skills: in particular, the ability to draw facts and concepts together from different parts of the course to form a coherent explanation of a general astronomy topic. In each Subject the participants are given a choice of several essay topics, each of which have been chosen to draw together concepts from different parts of the course material.

The essays are submitted and marked in mid-semester, in order to give the participants useful feedback on their progress and on the sort of writing skills that would be expected for the major project (see below). The essay has turned out to be a quite discriminating part of the assessment mix, with submissions that ranged from superficial collections of facts to highly coherent arguments, but again most participants do very well.

2.5 Project

The participants were required to carry out a major project requirement which counted for 30% of the total mark. We feel that it is important to provide a range of project choices that will complement the backgrounds and interests of the participants. The intention was to provide challenging, open-ended project work to reward higher-order learning skills such as the ability to research a project in depth or plan and carry out a major observational project, together with the ability to clearly and succinctly communicate the results of that research or set of observations. The project is also intended to provide an assessment task which participants would find personally useful and rewarding.

The list of possible project choices at the time of writing included over 50 suggested topics, and participants were also allowed to suggest their own. Approximately one third of the suggested topics are amateur observing projects, ranging from unaided eye to binocular to telescope projects. The list of suggested topics includes ‘Internet research’ projects for the armchair astronomers and curriculum development projects for the educators. It also includes software projects, where participants with programming expertise can write software simulations of astronomy concepts, and even a science fiction project, where participants can write a short story based on astronomy principles then provide a critical review of how those principles were used (or broken!) in the story.

The Project assignment produces a spectacular array of submissions. While a few are rushed and superficial, the vast majority of participants combine their workplace skills and astronomy interests to produce reports of a standard far exceeding that usually received in comparable conventional astronomy courses.

2.6 Assessing the more advanced subjects

In the more advanced subjects we will incorporate problem solving & extended-answer questions where appropriate into the computer-managed tests, in order to assess a wider range of skills than that possible with multiple-choice testing. We are also extending the scope of the project work to emphasize project topics more closely tied to the course content of the advanced subjects, while continuing to use the newsgroups as a central part of the course communications and assessment.

We are exploring ways to introduce optional cooperative work, mainly to reduce the sense of isolation that distance education students sometimes comment on despite the aid newsgroup communications. However we are also mindful that our participants are busy people who tend to do their course-work in ‘bursts’, not necessarily in step with an assigned partner. We are likely to trial cooperative work in the more advanced subjects first, as the students in these subjects are already experienced at organising their time effectively in online education and therefore likely to be more successful than novice online students in organising cooperative study over the Internet.

2.7 Examinations

Our present assessment mix does not include traditional examinations. Traditional examinations could in principle be held at designated distance-education venues, though they may not be practical for some of our participants who are located in remote places like the Cook and Canary Islands. However traditional examinations would not be an appropriate means to assess the learning outcomes for the course (see 2. The Assessment Mix above).

3. Feedback and Evaluation

Obtaining useful and reliable feedback is a challenge in any form of distance education. The assessment results themselves are useful feedback on outcomes, but do not provide much information about other problems which may inhibit learning – for example, difficulties with the technology or in carrying out practical observing projects, problems with the workload, and so on.

3.1 Formal evaluation

The Swinburne Quality Education Unit conducted an email survey of participants in Swinburne Astronomy Online in the second half of Semester 1, ‘99. The responses from the 50% of the participants in the two subjects running at that time who replied to the survey were extremely positive. On all 31 questions asked, on a scale of 0 (lowest) to 5 (highest), the participants ranked the course at 4.5 on average. In particular, they ranked the Assessment at 4.2 on average for clear assessment requirements, fair grading techniques, manageable workload and useful feedback from instructors. Table 1 shows a breakdown of the responses of the participants to the Assessment section of that survey.

While the results of this formal evaluation were very positive, it is also true that only 50% of the participants returned the surveys. While this is regarded as a good result for a distance education course, it still leaves us wondering how the other 50% would have responded. Work needs to be done to develop more effective evaluation techniques in online education (and distance education generally). Data on whether the participants who respond to these surveys are a representative sample would be very valuable. At present the Quality Education Unit does not perform any correlations on, for example, survey response with final course grade.

3.2 Informal Feedback

One of the advantages of teaching online is that one has an electronic copy of almost all feedback received from students, either as emails or as newsgroup entries. We received a great deal of enthusiastic informal feedback from participants in Swinburne Astronomy Online, including comments like:

"I'm finding the coursework group better than a standard session in class: you get more time to mull things over and to do a bit of research before contributing to the discussions. … It's more like the classic Greek form of tuition, with the course tutors and students sitting round a forum discussing the subject. It's live tuition."

"Their [the instructors in the newsgroup discussions] approach of letting us teach ourselves and then only stepping in to guide us back on track should we drift off it, was inspirational. I really like the way they did not interfere but rather collaborated with us."

We have not had the resources and time so far to quantify the percentage of, for example, favourable to unfavourable comments. This would be an interesting activity to carry out in the future, but might be better associated with the formal evaluation surveys, as in an informal setting one is arguably likely to receive more feedback emails from enthusiastic students than from apathetic or alienated students.

3.3 Assessment Results and Retention Rates

Not surprisingly, universities are particularly interested in practical feedback in terms of assessment results and retention rates. In Semester 1 ’99 participants in Swinburne Astronomy Online averaged a creditable 69% as their overall assessment mark.

It is too early to be definite about retention rates for the course. Approximately 60% of the participants in Semester 1 took more subjects in Semester 2, but by far the majority of the other 40% took leave of absence in Semester 2 with the declared intention of returning in later semesters. We will have a better idea of our real retention rates by mid 2000.

4. Conclusion

We have carried out the first year of teaching in Swinburne Astronomy Online, an online Graduate Certificate course in astronomy designed particularly for groups such as amateur astronomers, educators, and individuals working in astronomy-related fields.

We use a range of assessment tools meant to encourage active learning and to reward skills like effective communication of astronomy concepts, overall conceptual understanding, grasp of detail, and ability to research a topic in depth. Participants are offered a range of project work designed to complement their backgrounds and interests. The response from participants to the course and assessment mix, and especially to the use of newsgroup discussions as a teaching and assessment tool, has been exceptionally positive. We are now looking at adapting the assessment mix used with our introductory-level subjects to suit more advanced subjects presently being developed as part of Swinburne Astronomy Online.

References

Hufnagel B., Slater T., Deming G., Adams, J., "Astronomy Education for the New Millenium", ASA-sponsored conference at University of Western Nepean, Sydney, 11 – 13 July 1999.

Mazzolini, M. "Targeting Misconceptions in an Introductory Astronomy Course", paper presented at the Astronomical Society of Australia AGM, University of New South Wales, July 1997 (unpublished)

Sadler, P.M., "The Initial Knowledge State of High School Astronomy Students", Thesis Harvard University 1992

Table

 

1 Strongly Disagree

2 Disagree

3 Neither Agree or Disagree

4 Agree

5 Strongly Agree

Standard Deviation

Mean

The assessment requirements were clear.

   

2 (8%)

6 (23%)

18 (69%)

0.64

4.6

Evaluative and grading techniques were fair.

1 (4%)

 

5 (19%)

5 (19%)

11 (42%)

1.08

4.1

The assessment workload is manageable.

   

5 (19%)

9 (35%)

12 (46%)

0.78

4.3

I obtained useful feedback from the online instructors, which assisted me to judge my progress in this subject.

 

3 (12%)

8 (31%)

4 (15%)

11 (42%)

1.11

3.9

 

Table 1
"ASSESSMENT (please rate questions on a scale from 1=Strongly Disagree to 5=Strongly Agree by typing the appropriate number at the end of each statement).

ASKAP
Public