Background

Rovai (2003) argued in this research article that, in order for distance education to be as successful as possible, evaluations must be done and they must be thorough. He proposed a framework following a review of the literature surrounding online education evaluation and provides specific areas where the foci should be for distance education evaluation. Further, Rovai (2003) offered questions that cover four areas of evaluation for stakeholders and the methods needed to answer the questions. Those four areas of evaluation highlighted were:

• Input
• Output
• Process
• Impact

The framework and suggestion the author made relied heavily on open-systems theory, which supports a flexible set of processes to match what would be most appropriate for the program (Rovai, 2003). Evaluations concluded with the analysis of gathered information. This process was modeled in Figure 1. All opinions were considered for inclusion in the final report and the final comments included the perspectives that best represented the interests of the audience and stakeholders. Finally, programs were judged based on their norms (program standards and expectations) or benchmarks were put in place for the final judgment to take place (Rovai, 2003). It is significant to note that benchmarks need to be determined prior to the evaluation process beginning so as to avoid end-of-evaluation bias or changing standards to reflect the evaluation, meaning norms need to be a significant part of the initial conversation with stakeholders (Rovai, 2003).

Key points

• While there are a variety of evaluation categorizations to choose from (formative, summative, CIPP, etc), the author argued that the systems model approach was the most effective for distance education given its ability to evaluate multifaceted program elements (Rovai, 2003).
• Program evaluations should include:
  • Student performance
  • Program cost and effectiveness
  • Course design and instruction
  • Teacher and student satisfaction
  • Quality of technology and support services
• Specific questions and methodologies should be employed to meet the needs of the stakeholders. Qualitative and/or quantitative methodology should be used to answer the evaluation questions. Four evaluation types were provided:
  • Process
  • Impact
  • Output
  • Input
• Within the specific evaluation types, there are specific orientation strategies that reflect stakeholder involvement to consider. These strategies can either be used independently or combined with one another to build the most accurate evaluation:
  • Objectives-oriented—Determining whether outcomes and program goals were met
  • Management-oriented—Determining funding (reallocation, especially)
  • Consumer-oriented—Developing information on the product for the consumer
  • Expertise-oriented—Professional judges an educational program (heavily used for accreditation purposes)
  • Adversary-oriented—Incorporates both positive and negative feedback into the evaluation
  • Participant-oriented—All stakeholders are involved qualitatively in the evaluation
  • Evaluations should be chosen based on the needs of the program and the desires of the stakeholders

Discussion

Program evaluation for distance education needs to be a requirement and something that is done with every developed course. The evaluation type that was the most effective in the eyes of Rovai (2003) was the systems model approach. This approach allowed evaluators to not only compare distance education to traditional but gauge effectiveness of the distance education and whether the tools used were beneficial. Further, implementing an open-systems model, the program, organization where the program is housed, and its stakeholders can alter procedures to match the individualism of the program for a more effective program and thorough evaluation. This can be done by using an input, output, process, or process evaluation, based on the needs of the program being reviewed. Following choosing the evaluation type, the approach to the evaluation can then be selected. This decision is made after determining the combined interests of all stakeholder groups that have asked for the evaluation (Rovai, 2003). All evaluation types allow for areas of weakness and strength to be highlighted in the review of the distance education and varying levels of hands-on involvement from stakeholders. Both qualitative and quantitative (or a mix of both) can be used throughout all evaluation types, upholding the systems model approach that embraces flexibility to meet the needs of the evaluation.

Questions

1. Rovai (2003) proposes four areas or program evaluation foci. Do you agree with these four areas? What would you change? Explain.
2. Were there one or two specific evaluation strategies that resonated with you as a way you could see yourself evaluating future instructional designs? Provide the strategy/strategies and why they resonated with you. How would you implement them?
3. Rovai (2003) notes that one of the purposes of evaluation is to “drive down costs” (p. 122). What are the pros and cons of having financial incentive while evaluating educational programming
4. What do you believe the biggest benefit to ongoing program evaluation and assessment is? What would you struggle with? Explain.

Additional resources

• Open Systems Model. (n.d.). Reflect & learn: Learning together about organizational assessment. Retrieved from https://www.reflectlearn.org/discover/open-systems-model
• Roddy, C., Amiet, D.L., Chung, J., Hold, C., Shaw, L., McKenzie, S., … Mundy, M.E. (2017). Applying best practice online learning, teaching, and support to intensive online environments: An integrative review. Frontiers in Education, 2(59), 1–10. https://doi.org/10.3389/feduc.2017.00059.
• Rovai, A.P., Ponton, M.K., & Baker, J.D. (2008). Distance learning in higher education: A programmatic approach to planning, design, instruction, evaluation, and accreditation. New York, NY: Teachers College Press.

Additional references

• Rossi, P.H., Freeman, H.E., & Lipsey, M.W. (1999). Evaluation: A systematic approach. Newbury Park, CA: Sage.
• Worthen, B.R., Sanders, J.R., & Fitzpatrick, J.L. (1997). Program evaluation: Alternative approaches and practical guidelines (2nd eds.). New York, NY: Longman.