CRITERIA FOR ANALYSING AND EVALUATION OF DISTANCE LEARNING ENVIRONMENTS: A LITERATURE REVIEW

Mehmet Karahan, Mehmet Tikici, Kaan Diyarbakırlıoğlu Inönü Universty

mkarahan@inonu.edu.tr ÖZET

Bu çalışma “uzaktan öğretim ortamlarını değerlendirme kriterlerimiz neler olmalıdır?” sorusuna cevap ararken literatürde karşımıza çıkan değerlendirme ölçütleridir. Yaptığımız geniş literatür taramasında, bu konuda değişik ülkelerde yapılan uygulamalar ve bu çalışmalarda göz önünde tutulan, genel kabul gören birtakım kriterler bulunduğunu belirledik. Yapacağımız araştırmalarda yararlanmak düşüncesiyle, elde ettiğimiz bu kriterler ve araştırmacı görüşlerini mümkün olduğunca kısaca özetlemeye çalıştık.

Yaptığımız literatür taraması sonucu tespit ettiğimiz bu kriterler; “iyi bir öğretim uygulamasının yedi prensibi”, “web planlamada on temel hata”, “online öğrenme ortamları arayüz tasarımı”, “dağıtık uzaktan öğretim hizmet araçları” başlıkları altında, literatürdeki gibi düzenlenmiştir. Bu bilgilerin uzaktan öğretim ortamlarının değerlendirilmesi çalışmalarında konuyla ilgili araştırma yapan tüm araştırmacılara yararlı olacağını ümit ederiz

Anahtar Kelimeler: Online öğretim ortamlarının analizi, arayüz tasarımı, uzaktan öğretim hizmet araçları ABSTRACT

This study is the evaluation criteria we faced in the literature while we are looking for an answer to the question of “what are to be our criterions to evaluation of distance learning environments?” during the analysis of distance learning environments. In the extensive literature search, we have determined some generally accepted criteria that are taken into consideration in different countries. We tried to arrange these criteria and researchers’ views shortly in order to use them in our study. The criteria we have determined at the end of literature search of distance learning environment evaluation can be classified as follows: “Seven Principles of Good Teaching Practice”, “Top Ten Web Design Mistakes”, “Interface Design for Learning Environments”, “Distributed Distance Learning Delivery Tools”. Thus we aimed at following these criteria in our studies. We hope that these findings will be helpful for all the researchers who study on the evaluation of distance learning environments.

Keywords: Online learning environments, instructional design, interface design, distance learning delivery tools INTRODUCTION

While making a literature search to look for the answer of the question “What must be our criteria when evaluating distance learning environments?” in analyzing the distance learning environments, we have faced with some evaluation criteria. By taking the applications and studies carried out in different countries, we tried to summarize the generally accepted criteria in order to use in the further studies. These criteria will be a guide for us in the study of distance learning environments’ evaluation in preparing the questionnaire. Moreover, we involved the findings and comments of the researchers from different countries who studied on the same subject.

Gorsky, P. & Caspi, A. (2005),as an analytical tool, elements of distance education systems may be viewed in terms of

dialogues and supporting resources. Discursive activities amongst instructors and students are viewed as interpersonal dialogue (subject matter oriented or social) whilst individual study is intrapersonal dialogue. Resources may be viewed as supporting one or both dialogue modes. This compact frame of reference imposes an order on the wide variety of seemingly unrelated variables that comprise distance education systems.

Illustrations follow:

1. A student reads a self-instruction text. The text is a structural resource that enables intrapersonal dialogue.

2. Student X seeks help in solving an assigned exercise by posting a message in an asynchronous discussion group. Student Y responds and helps. The discussion group is a structural resource that enables interpersonal dialogue, both subject matter oriented and social. A subject matter-oriented dialogue occurred between Students X and Y.

3. A student calls another student on the telephone. They discuss the weather and make plans to meet at a pub. Having telephone numbers of fellow students is a structural resource for interpersonal dialogue whilst the fellow student is, of course, a human resource. In this case, they engaged in social dialogue.

4. Assume a face-to-face tutorial session led by an instructor who encouraged discussion and question asking. The tutorial is a structural resource that enables both intrapersonal and interpersonal dialogue.

5. An instructor lectures and there is no accompanying question and answer for discussion. The lecture is a structural resource that enables intrapersonal dialogue. The verbal presentation is equivalent to a video cassette or a written printout of the lecture.

Robertson, J.S., Grant, M. & Jackson, L. (2005), the seven principles are abstractions from successful teacher practice and represent an aggregate of teaching strategies and methods. For example, “encourages active learning” can subsume many constructivist methods and learner-centered strategies, such as project-based learning, problem-based learning, case-based learning, open-ended learning environments, self-regulated learning and self-directed learning. However, the implementation of these strategies neither guarantees successful teaching and learning nor assures student satisfaction.

IETC

Bangert, (2004) asserts that constructivist models are recommended and employed for online instruction. Therefore,

evaluation instruments that do not specifically reflect these learner-centered strategies or methods are suspect. Moreover,

there is an assumption that providing for all of the seven principles will ensure successful learning and student satisfaction. This is not the case. Specifically, some online students may be ill equipped to handle the self-direction necessary to succeed (e.g., Bangert, (2004) and some students’ lack of face-to-face contact may prevent student satisfaction or self-efficacy. In March 1987, the American Association of Higher Education first published "Seven Principles of Good Practice in Undergraduate Education." These seven principles are a meta-analysis of 50 years of research on good teaching principles by Arthur Chickering and Zelda Gamson. These principles apply to teaching and learning in any environment.

A. Seven Principles of Good Teaching Practice(Online, 2005) 1. Good practice encourages interaction between students and faculty.

Frequent student-faculty contact in and out of class is a most important factor in student motivation and involvement. Faculty concern helps students get through rough times and keep on working. It also enhances students' intellectual commitment and encourages them to think about their own values and plans.

WebCT Tip: Use the Discussion Board, Mail, Chat & Whiteboard to interact with your students. 2. Good practice encourages interaction and collaboration between students.

Learning is enhanced when it is more like a team effort than a solo race. Good learning, like good work, is collaborative and social, not competitive and isolated. Working with others often increases involvement in learning. Sharing one's ideas and responding to others improves thinking and deepens understanding.

WebCT Tip: Encourage students to ask questions of each other, try & answer each others' inquiries, or react to each others work (via the Discussion Board, in Mail, or Chat, for example). Set up a "buddy" system or groups for collaboration (you can assign private Discussion Topic areas to specific groups of students, assign Chat rooms, and set up a public Student Presentation area where each group can share their group work).

3. Good practice uses active learning techniques.

Learning is not a spectator sport. Students do not learn much just sitting in classes listening to teachers, memorizing prepackaged assignments, and spitting out answers. They must talk about what they are learning, write reflectively about it, relate it to past experiences, and apply it to their daily lives. They must make what the learn part of themselves.

WebCT Tip: Design Assignments to encourage students to apply the concepts learned in the course. Use the Self-Test tool for active review of materials. The Discussion board is a good place to have students reflect publicly on issues & respond to each other's reflections.

4. Good practice gives prompt feedback.

Knowing what you know and don't know focuses your learning. In getting started, students need help in assessing their existing knowledge and competence. Then, in classes, students need frequent opportunities to perform and receive feedback on their performance. At various points during college, and at its end, students need chances to reflect on what they have learned, what they still need to know, and how they might assess themselves.

WebCT Tip: Let students know what type of feedback to expect from you & how often it will be provided. Likewise, be clear as to what type of feedback you'd like from them. At various times during the course, allow the students a forum for providing feedback about the course itself. Note that the Quiz & the Self-Test tools are a good ways of providing instant feedback.

5. Good practice emphasizes time on task.

Time plus energy equals learning. Learning to use one's time well is critical for students and professionals alike. Allocating realistic amounts of time means effective learning for students and effective teaching for faculty.

WebCT Tip: Use the Calendar tool as one way to keep students on-task. Timed quizzes emphasize time-on-task, as well. 6. Good practice communicates high expectations.

Expect more and you will get it. High expectations are important for everyone - for the poorly prepared, for those unwilling to exert themselves, and for the bright and well motivated. Expecting students to perform well becomes a self-fulfilling prophecy.

WebCT Tip: Provide students examples of "A"-quality work. Release statistics along with grades, so that students can see how they are performing as compared to the rest of the class (stats can give the mean grade and/or the frequencies). Use the "Selective Release" feature to release course info only as students achieve a certain level of success on a test.

7. Good practice respects diversity --- talents, experience, and ways of learning.

Many roads lead to learning. Different students bring different talents and styles to college. Brilliant students in a seminar might be all thumbs in a lab or studio; students rich in hands-on experience may not do so well with theory. Students need opportunities to show their talents and learn in ways that work for them. Then they can be pushed to learn in new ways that do not come so easily.

WebCT Tip: Stick to a template for course page design *but* vary the types of exercises and assignments. B. Top Ten Web Design Mistakes

Nielsen’s Top Ten Web Design Mistakes are adapted from Nielsen's Alertbox (Nielsen, 1996). These design guides will be used to evaluate the user interface of the WBI(Henke, 1997).The following is a succinct explanation of each item.

1. Using Frames. Frames are not to be used as frames make it hard to correctly set a bookmark.

2. Gratuitous Use of Bleeding-Edge Technology. Like avoiding to many fonts, new technology, such as three-dimensional designs should only be used if the content dictates use, such as a web site for maps.

3. Scrolling Text, Marquees, and Constantly Running Animations. Do not include any element that constantly is moving because this distracts the reader.

IETC

4. Complex User Resource Locations (URLs). Users may want to type in a URL directly within their browser or may use the URL as bookmark. Therefore URL should make sense and use lower case characters to make entering the data easier. 5. Orphan Pages. Each page should have a link to the home page (index page). A page that does not allow you to navigate from the page is considered orphaned. Or a page that causes the Web Browser to display a message stating the link is no longer valid.

6. Long Scrolling Pages. Nielsen (1996) states that "Only 10 percent of users scroll beyond the information that is visible on the screen when a page comes up" Therefore, if a page must be longer than one page, then the most important information must be displayed at the top of the page.

7. Lack of Navigation Support. A map must be provided of how to access information on a web site and also each page should include a link to get to the next page or the previous page. Additionally, a search engine should be provided to enable users to search any page on a web site.

8. Non-Standard Link Colors. Nielsen (1996) states that: "Links to pages that have not been seen by the user are blue; links to previously seen pages are purple or red". This is typical throughout the World Wide Web community.

9. Outdated Information. Information must be kept updated as information on the World Wide Web is often outdated, within days. Information considered 'old' by users may lead the users to believe all of the information is outdated.

10. Overly Long Download Times. Nielsen (1996) states that: "Traditional human factors guidelines indicate 10 seconds as the maximum response time before users lose interest. On the web, users have been trained to endure so much suffering that it may be acceptable to increase this limit to 15 seconds for a few pages."

C. Interface Design for Learning Environments

Interface Design for Learning Environments by Jones and Okey, (1995). The Jones and Okey Interface Design includes a set of User Interface Design Guidelines that incorporate five concepts from literature in the fields of computer based instruction, computer based learning environments, and human-computer interaction. Each concept represents sub-concepts such as for the concept browsing, there are sub-concepts such as closure, progressive closure, changes in state, and selection indicators. Each concept includes a number of items used to evaluate the WBI design. For instance, under browsing, an item to evaluate is “provide maps so that users can find where they are and allow provisions to jump to other information of interest from the map.”

The five concepts include: browsing, media integration, metaphors, information access, and unfamiliar territory. The Interface Design for Learning Environments is not a proven set of metrics; instead it is a set of metrics based on Human Computer Interaction research and is meant to be used as a guideline for evaluating the design of Computer-based Learning Environments.

D. Distributed Distance Learning Delivery Tools

The differences between traditional in-class courses and distributed/distance learning (DDL) courses create several factors which need to be addressed in DDL course design. First, traditional courses are face-to-face synchronous courses. This means that learning occurs with the instructor in front of the learner and instruction and learning occur simultaneously and in the same place. This personal interaction between the two gives the instructor the opportunity to provide feedback, direction, and to observe learning activities. It gives the learner the opportunity to solicit feedback and receive responses and directions in real time. Second, traditional courses place the learner with other learners. Learners are able to draw from each others experiences and interact in groups. Third, the requirement to attend class creates a responsibility for the learner and provides a source of accountability, possibly increasing his/her motivation to perform the required tasks (Baker and Papp, 2003). To develop procedures for addressing these factors, an understanding of the tools available for distance-learning instructional methods and course design is needed. A variety of web-based tools and course design strategies including the following, are available (Ferguson, 2001; Baker, 2003):

1. Syllabus / course outline posting

While syllabus posting is available for both DDL and non-DDL courses, the absence of an instructor to review the syllabus with the learner requires an easily understandable and comprehensive syllabus.

2. Video classroom

Streaming video has replaced video tapes as the tool of choice for lecture delivery in DDL courses. Video must be interesting, engaging, and worthwhile. Videos should not simply restate what is available in written materials. A primary consideration for streaming video is the bandwidth available to the learner. While high speed connections are usually available on campus, Students who rely on dial-up connections will be at a significant disadvantage when using streaming video. Provisions for downloadable or CD-Rom video files should be made to prevent this problem.

3. Course Notes

Course notes can be posted to the web to supplement video lectures and required readings.

4. Course Reference Materials, Readings, Cases Supplemental reading materials can be posted to the web for students to download. Links to other websites are frequently provided in references.

5. Chat Rooms

Structured chat rooms conducted by the instructor provide group discussion on course activities and assignments. Using real- time chat, the instructor can ask questions during in a similar manner to the traditional classroom. Audio chat, a web-based tool that functions similar to teleconferencing, using a tool like Microsoft’s NetMeeting ® has an advantage in that it is more spontaneous than text-based chat and not dependent on the learners keyboard speed. Chat also allows the instructor to provide immediate feedback to learner questions, evaluate learner participation, and take attendance.

IETC

6. Email

Email allows students to asynchronously communicate with their instructor. Learners can ask questions and send assignments to the instructor. The instructor can use email to send evaluated assignments back to the learner

7. Bulletin Boards, Group Discussion Boards, Digital Drop Boxes

These tools allow the learners to collaborate on projects, exchange ideas and participate in group activities. 8. Online Testing

Online testing procedures allow the instructor to design evaluation instruments comparable to any form of paper-based instrument. Multiple choice, true or false, matching, and fill-in-the-blank questions can be automatically graded and posted. Long or short answer essay questions can also be used. However, essay questions must be graded by the instructor.

9. Interactive Activities

Interactive activities provide a method of having the students practice desired behaviors. Click and drag techniques can be used to assemble components online (for example atoms into a molecule).

10. Feedback

Specific provisions must be provided to insure students receive sufficient feedback. A frequent criticism of DL is its disembodied nature restricts feedback leaving learners feeling abandoned. Instructors must be trained to promptly respond to emails. Virtual office hours can be held using chat. Computer graded exams should have provisions for giving the student correct answers to the questions answered incorrectly. Provisions for both asynchronous and synchronous feedback should be provided in course design. Asynchronous feedback occurs when the individual requesting the feedback (the student) experiences a time delay before feedback is received. Synchronous feedback occurs when the feedback response immediately follows the question or request with no time delay.

11. Virtual Classroom

An online, interactive class session between students and instructor. Simulates much of the interaction found in traditional face-to-face classrooms. Frequently incorporates other web-based tools including audio chat, video classroom, whiteboarding, etc.

12. Whiteboarding

The ability to write and draw on an electronic board during a virtual classroom session. REFERENCES

Anderson, M., Jackson, D. (2000). Computer Systems For Distributed and Distance Learning. Journal of Computer Assisted

Learning (2000) 16, 213-228.

Arthur W. Bangert, A.W. (2004). The Seven Principles of Good Practice: A framework for evaluating on-line teaching.

Internet and Higher Education, 7 (2004) 217–232.

Baker, Russell K., 2003. A Framework for Design and Evaluation of Internet-Based Distance Learning Courses Phase One - Framework Justification, Design and Evaluation. Online Journal of Distance Learning Administration, Volume VI, Number II, Summer 2003.

Gorsky, P. & Caspi, A. (2005). Dialogue: a theoretical framework for distance education instructional systems. British

Journal of Educational Technology Vol 36 No 2 2005, 137–144.

Henke, H. (1997). Evaluating Web-Based Instructional Design. Published online. [On-line], http://www.chartula.com Henke, H. (2001). Factors Influencing Attrition Rates in a Corporate Distance Education Program. Originally published by

the United States Distance Learning Association, November, 2000, in Education at a Distance Journal, Volume 14, Number 11. Updated June, 2001 and published online.

Insung, Jung (2001). Building a theoretical framework of web-based instruction in the context of distance education. British