Edition 17 of the Astronomy On-line Newspaper

Evaluation - the Questionnaire

The evaluation of Astronomy On-Line is an important part of the exercise, since it will help to ensure that the further development is done in an optimal way. The organisers are thankful to all those participants who have sent in comments - please be ensured that they will all be taken into account. It is good to note that most are very positive, expressing obvious satisfaction with the project and its implementation. Nevertheless, some suggest possible improvements in various areas. This is of course what had to be expected in connection with such a complex undertaking.

As you may remember, a Questionnaire was placed on the Astronomy On-Line Website towards the end of the Hot Phase. From this, the organisers hoped to obtain a clearer picture of the strengths and weaknesses of this Programme, according to the corresponding input by the participants.

It would appear that in the end comparatively few participants decided to answer the Questionnaire and the following, still provisional report from Jari Multisilta who undertook this work may, therefore, not fully reflect opinions of all participants. Nevertheless, it contains important findings and is in any case a very valuable contribution to the discussion about the future of Astronomy On-Line.


Provisional report from Jari Multisilta

Foreword

Here is the text of my provisional report about Astronomy On-Line. Once it has been completed, I am considering to publish this in the Proceedings of a small workshop here in Finland and to further develop it to a full article for the scientific journal Education & Information Technologies. It is published by IFIP, the International Federation of Information Processing.

Jari Multisilta
Tampere University of Technology
Information Technology
FIN-28601 Pori
FINLAND
Tel. +358-(0)2-627 2747, +358-(0)400-935 359
Fax +358-(02)-627 2727
email: Jari.Multisilta@pori.tut.fi
http://www.pori.tut.fi/~multisil/

Abstract

In this article, the Astronomy On-Line (henceforth AOL) learning experiment on the World Wide Web is described. As a part of the AOL, an educational research project was carried out. The purpose of this study was to measure how students experience the new medium in their learning and how they utilise the communication facilities (group email, realtime discussions) available through the AOL website. This study describes also the computational facilities available among AOL participants all around the Europe.

Introduction

Astronomy On-Line (AOL) was the world's biggest astronomy event on the WWW; it was organized by European Association for Astronomy Education (EAAE) and the European Southern Observatory (ESO). Its aim was to bring together students from all over the Europe to explore astronomical questions using modern learning tools.

The AOL started in October 1996 and ended on November 22, 1996. In total, 720 groups consisting of over 5000 students from 39 countries participated in AOL. There were students from all age groups starting from under 12 years old primary school students to over 20 years old university students. As a part of the AOL an educational study was set up to evaluate the learning process in AOL.

AOL activities

A main activity during AOL was a series of collaborative projects with astronomical observations. These collaborative projects were developed in three 'phases'. The first phase was used to experiment with the WWW. The second phase involved more activities and the third phase, actually the last week of the AOL, was the 'hot' phase that involved a lot of intensive work. The first collaborative project was to observe the lunar eclipse on September 27, 1996. Based on the observations the longitude of the observer's place could be determined.

The Second project was to observe the partial solar eclipse on October 12. Based on the observations it was possible to measure the Moon's distance. This was probably the first time that the Moon's distance was measured by the means of a solar eclipse by amateurs and students.

Yet another collaborative project was to study the Sun and the terrestrial aurorae. Groups in Central and Southern Europe observed the Sun, looked for sunspots, maybe flares and made a daily sketch of the solar surface as seen on a projected telescope image. At the same time, Groups in Northern Europe made observations of variations in the Earth's magnetic field and of the beautiful aurorae which are often visible there.

Together, these observations made it possible for students to study actively the relationship between the Sun and the Earth. For example, the changes on the Sun may later be registered at the Earth, when the fast particles of the solar wind have travelled the distance.

Other AOL activities included treasure hunts using WWW navigational techniques, talks with professionals using real-time discussion software, the use of astronomical databases and software, group communications using email and bulletin boards.

Network Based Learning Environment

The elements used in AOL constitutes of an ideal network based learning environment. In this section we try to describe network based learning environments from the theoretical point of view.

Generally, a learning environment is a collection of topics. A topic is a collection of educational activities, such as reading a piece of text, looking at a picture, playing with a computer-based interactive device or searching information from the library. The general definition of the learning environment is based on the concept of learning-by-doing.

From a more technical point of view the network based learning environment is characterised by the elements that are needed in the network based learning environment. In order to learn by doing educational activities it is nessessary that network based learning environment consists of learning material as hypertext, problem solving tools (cognitive tools), communication tools and external information resources. The type of the network conncetions on the user side and on the server side can considerably limit the type of the activities that can be delivered to the remote student. For example, the use of video material may be impossible in slow modem connections, but acceptable in ISDN connections.

There has not yet been many implementations of learning environments that can be described as being network based. For example, the survey of learning environments in mathematics in [Multi96] shows that many learning environments do not yet utilise communication technologies. The network based learning environments differ from local learning environments by the fact that in network based learning environments the information is loaded at least partly from the network and not from the local hard disk or CD-ROM. This fact must be remembered when designing and implementing network based learning environments.

In particular, the three key factors for using networked information are:

1. Availability: information may be available and still be impossible to use (slow connection prevents the use of large materials).

2. Reliability: how to be sure about the correctness of the information.

3. Usability: how to find relevant information (searching, filtering) and how to find information that can be understood with my backgound.

Research Objectives

As a part of the AOL an educational study was set up to evaluate the learning process in AOL. The main research objectives were the following:

- How do students experience the new medium in their learning?

- What is the level of international collaboration and the usage of communication tools in the collaboration?

- What was the role of astronomical observations in the learning process?

- What was the role of experts (talks to professionals) in the learning process?

- What was the availability of local resources (computers, networks, time)?

- How much did the students previous knowledge influence their learning outcome?

- What were the motivational factors to learn astronomy in AOL?

The research was done by implementing an on-line questionnaire on WWW as a WWW form and a corresponding cgi-script that automatically processed the answers in the questionnaire to the data matrix. The questionnaire was available to the students during the last days of the Hot Phase of the AOL.

The questionnary contained of 28 statements that studens were asked to evaluate on scale 1 to 5 (totally agree, totally disagree). In addition, there were 7 questions where students had to characterise in a five point scale themselves as a learner (active or passive, theoretical or practical, quick or slow, explorer or listener), the WWW as a learning environment (active or passive, information database or a learning tool) and the navigation in AOL web site (easy or difficult). There were also 7 questions that described the equipments (place where they used the AOL web site) and the background of the students (age, reference group, gender). Finally there were 5 questions where the studenst could answer verbally to explain theri expectations from AOL, the most positive and negative experience from AOL and the main problems they had during the AOL.

Altogether, 89 responses were received before the deadline, a few days after the termination of the Hot Phase. This must be considered as a small number of responses when compared to the number of all students participating in AOL. There are many possible explanations why we did not succeed to get more responses. However, from the feedback available we can say that many students had problems with slow communication lines, strict timetables and language. In the next chapter we try to summarise the collected data.

Some Provisional Results

In this section the collected data are summarised. The data is analysed using only descriptive statistics. For deeper analysis, the collected amount of data may not be sufficient.

Characteristics of the responses

The answers are distributed geographically quite heterogeniously all over Europe. Most responses were recorded from Norway and Denmark which had both 17 percent of all responses.

Only 12 percent of all respondents were under 16 years old - the rest were older. This could be explained by the language problems. The english questionnary was propably too difficult for the younger (non-English) participants. There were also a couple of verbal responses that asked for German or French versions of the WWW pages.

15 percent of the respondents were members of astronomy clubs, 32 percent were secondary school students, 18 percent were college students, 10 percent were university students. The responses from teachers were about 15 percent of all the responses. There were about 15 percent female respondents.

Connections and Website navigation

It seems that the most popular place where the AOL was mainly used was home (30 percent of all responses). This can also be seen from the most popular connection type that was fast modems: 52 percent of all respondents informed that they used a 9600 bit/s or higher modem. 10 percent of all respondents did not know the connection type at all. They probably used a computer in a public library or on another school than their own.

However, in the verbal responses quite a few mentioned that the main problem in AOL was slow connection lines. On one hand, AOL Web pages had much graphics that takes a long time to load over some modem lines. On the other hand, because of the graphics the navigation in AOL Web site were found to have been easy (65 percent of all responses).

Usability and reliability

65 percent of all respondents reported that they found easily activities that corresponded their level of knowledge.

The reliability of the information in AOL was not a problem because it was known that the activities and information pages were designed by professional astronomers and teachers. This is an important factor for the continuation of the project. Some groups informed that they had not had enough time to do all the collaborative projects during the AOL, but instead they had printed the projects and will do them later by themselves.

Communications

Many respondents expected to get in contact with other students interested in astronomy. In general, the communication were considered to be the most positive feature and also a weakness of AOL: there was the real-time discussion channel but it opened too late.

One of the problems in AOL was the cloudy skies. During AOL there were many projects that required to do observations on a certain night at a certain time (for example eclipses). However, the problems was at least partially solved by providing color pictures in AOL Website for example from the lunar eclipse.

Learning

Interestingly, it appeared that females do not find solving scientific problems as interesting as males. Local resources were all seen as important factors for learning. It seems that the role of the local teacher was extremely important because the local teacher could integrate the AOL to the other work at school and to the school's curricula.

The activities were generally not seen as too theoretical and the theory behind the activities was understood very well. In this study computers were seen as a tools for learning (they make the studying easier and efficient) and they are not used too much in learning. What is more AOL couraged quite a many to study more about Internet.

About the author

Dr. Tech. Jari Multisilta is currently a senior researcher at Tampere University of Technology at the department of Information Technology, Pori. His research interests include modern learning environments, hypermedia and multimedia.

Acknowledgments: I am grateful to Mr. Franck Pettersen and Mrs. Josee Sert from EAAE who made many improvements to the questionnary and to Mr. Richard West for coordinating Astronomy On-Line and making it possible to do the AOL evaluation project.


Astronomy On-line | EAAE | ESO | Help | Search
Last update: Jan 24, 1997