Critical Issues in Evaluating the Effectiveness of Technology

By Mary McNabb, Mark Hawkes, and Ullik Rouk

Reaction Paper

By Sandy Nightingale

CET720

At the Secretary’s Conference on Educational Technology on July 12-13, 1999 in Washington, D.C., evaluating the effectiveness of technology was a topic of discussion. The article stated that in recent years there has been a change in schools’ focus on technology, shifting away from building and implementing a technology infrastructure towards evaluating the effectiveness of its use in schools and classrooms. It also went on to say that if funding is to continue, the effectiveness of technology’s use in schools and classrooms must be evaluated. I agree that we not only need summative evaluations for the state policymakers, but also a more useful formative evaluation for school practitioners. My school district seems to be stuck between the two phases of implementing technology and evaluating its effectiveness.

Seven critical issues were identified at the conference, and I will react to each issue and compare them to the technology processes currently being used in my district, although I must emphasize again that our evaluation process is at somewhat of a standstill.

Critical Issue 1: “The effectiveness of technology is embedded in the effectiveness of other school improvement efforts.”

As the article stated, “Developing ways to isolate the effects of technology within a dynamic environment where so many elements work together is one of evaluation’s most challenging issues.” I believe that it is very difficult to produce measurable results that portray the effects of technology. I agree that evaluation designs need to take the “localness” of evaluation into account.

My school district’s Technology Plan (White Lake School District 1-3) contains a very brief section on evaluation. This is the only place that I was able to find any information on technology evaluation for my school. It isn’t very specific, but contains four questions that will be evaluated, without stating how the evaluation will be conducted. They are very broad questions about teachers’ skills, student benefits through technology, community benefits through technology, and future needs for improvement. The evaluation section is very vague as to when it will be done, and it doesn’t contain a clear direction for the evaluation to take place, whether through benchmarks, surveys, interviews, test scores, or a mixture of these. It doesn’t state how technology can be separated from other content areas that it has been integrated with. I am not aware of any technology evaluation even taking place, so this is an area that definitely needs work.

Critical Issue 2: “Current practices for evaluating the impact of technology in education need broadening.”

I agree with the authors that finding ways to identify and measure skills and knowledge students gain from using technology is a very difficult task. It is compounded by the fact that policymakers need summative evaluations showing concrete improvements that students gain from using technology, while educators need formative evaluations to align technology uses with the learning goals and content standards of the curriculum, curriculum integration methods, and learning processes in order to make decisions for continual improvements. The constant change in technology and the struggle to keep current also complicates this issue. I don’t believe that my district is even ready for this step, because we really haven’t even started the process of evaluating technology. I think that keeping Critical Issue 2 in mind will help us when developing our technology evaluation plan, hopefully making our plan broad enough from the start.

Critical Issue 3: “Standardized test scores offer limited formative information with which to drive the development of a school’s technology program. Most schools are looking for additional means for collecting useful data for this purpose.”

To improve technology’s effectiveness in schools, they need formative evaluations that tell what works, under what conditions, and with which students. To me that sounds like a very difficult task, but the authors’ suggestions in the article for using existing data would be considerably helpful, as well as using observations, questionnaires, and standardized tests. I don’t believe that my district has even started collecting data, but the suggestions offered in the article to look for evidence of technology effectiveness through fewer disciplinary referrals, students’ completing more complex homework assignments, a new robustness in student performances, students taking more difficult electives, etc. will be a great place to start collecting useful data for this purpose.

Critical Issue 4: “Schools must document and report their evaluation findings in ways that satisfy diverse stakeholders’ need to know.”

Parents, teachers, administrators, funders, policymakers, and taxpayers each have different interests in the effectiveness of technology. As the article stated, “The best hope of closing this gap lies in helping all stakeholders to see (1) the importance of technology as an effective component of the educational system, (2) how technology is and isn’t capable of making a difference in curriculum and instruction, and (3) how innovative practices of teaching and learning with technology require multiple measures in order to verify its impact.” Comparing instructional practices and learning opportunities that students have with technology to instruction without technology might be helpful. Interpreting and reporting data in understandable terms for a variety of audiences could influence the way people think about technology. In my opinion, this is a major point we will need to consider when reporting data, especially to parents. It could mean the difference between their support for technology or not.

Critical Issue 5: “In order for evaluation efforts to provide stakeholders with answers to their questions about the effectiveness of technology in education, everyone must agree on a common language and standards of practice for measuring how schools achieve that end.”

Dialog among stakeholders is very important in evaluating technology. Therefore, there is a need to come to a consensus on the terms and language of the evaluation process. Different terms elicit different meanings from different educators, which could harm evaluation efforts.

Critical Issue 6: “The role of the teacher is crucial in evaluating the effectiveness of technology in schools, but the burden of proof is not solely theirs.”

The teacher is a great source of data about how technology impacts student learning. What teachers know about their students helps them to make decisions about using technology on a daily basis. I agree that evaluators need to trust the teacher’s ability to determine and describe technology effects. I believe that teachers should act as partners with researchers and also help with measuring and documenting changes in student learning because of technology. I also believe that most of the teachers in my district would be willing to help in the documentation process.

Critical Issue 7: “Because today’s classrooms are expected to be technologically up to date, policies need to be updated continuously to support best practices in technology use.”

As I read over this section of the article, two issues stuck in my mind. The first is “What is a school’s responsibility for out-of-school computer access?” As more and more teachers become computer literate, they are assigning more work requiring the use of computers. Many of our students do not have access to computers/Internet at home, so the work must be completed in school. We do have open labs one night a week, and students could make arrangements with teachers to work after school in the labs. The other issue deals with using email communications with parents. I agree that frequent communications with parents is positive, but I also worry that some parents may over-use this feature, making it very time consuming for the teacher.

Summary: I found the issues in this article to be very realistic and identifiable. I hadn’t realized until now that my school district didn’t have a very thorough or effective evaluation plan. The article had many feasible starting points to help us when creating our technology evaluation plan. I can see that creating the plan will not be an easy task, but using some of the suggestions laid out in the article should prove to be very helpful.