Internet Learning Volume 5, Number 1, Fall 2016/Winter 2017 | Page 11

Internet Learning
form that organized evaluation items
according to the eight categories. However,
the items were ordered randomly
without identifying the categories before
being presented to the students in
order to reduce potential bias in their
responses for the titles of the categories.
The peer instructors evaluated
courses without any time constraint.
During the evaluation process, they
examined course syllabi, the structure
of course sites, course materials, and
assignments. Students responded to
the evaluation items more on the basis
of their experiences in the course. Students
also had no time constraints.
Participation
Students who took the online courses in
the adult education graduate program
were recruited for this study. Among 89
students who were invited to the study,
a total of 24 students from five courses
participated in the evaluation. Three
peer instructors from the same department
were recruited to evaluate courses
based on the QM rubrics. Two peer
instructors independently evaluated
courses that they did not teach.
Analysis
Survey responses were analyzed descriptively.
In order to reveal general
evaluations of the courses, the average
ratings of categories and individual
items were calculated per course.
Correlation analysis was carried out to
explore the inter-rater reliability of the
evaluations between students and peer
instructors.
Results
Overview of Evaluation
Overall, the evaluation revealed that
the courses abided by the QM standards
quite well (on average, 4 out of
5). Although there were not big differences
between courses (range between
3.7 and 4.2), we found some variation
across the 8 categories (see Table 1). For
example, between category 5 (course
activities and learner interaction) and
category 7 (learner support), the mean
Table 1. Overview of Evaluation Conducted by Students and Peer Instructors
Note: Category 1, Course Overview Introduction; Category 2, Learning Objectives; Category
3, Assessment and Measurement; Category 4, Instructional Materials; Category 5,
Course Activities and Learner Interaction; Category 6, Course Technology; Category 7,
Learner Support; and Category 8, Accessibility and Usability.
10