JSET ejournal
Research
& Practice
Associate Editor Column
Dave Edyburn
Measuring Assistive Technology Outcomes in Reading
Given the importance of the topic of assistive technology outcomes
to both researchers and practitioners, a series of Research and
Practice columns have been devoted to the topic of measuring
assistive technology outcomes in specific academic domains. Interested
readers may wish to review previous columns in the series: overview
of key concepts associated with measuring assistive technology
outcomes (18-1), assessing assistive technology outcomes in writing
(18-2), and assessing assistive technology outcomes in mathematics
(18-4).
In the final column of this series, we examine issues associated
with measuring assistive technology outcomes in reading. Despite
the critical importance of learning how to read, little attention
has focused on how technology can be used to augment the performance
of struggling readers (National Reading Panel, 2000). Likewise,
much remains to be learned about measuring the outcomes associated
using technology to compensate for poor reading performance.
Reading and Technology
Learning to read is the predominant focus of instruction in grades
1-3. Every school year, hundreds of thousands of young children
celebrate the developmental milestone associated with learning
to read independently. While the research literature describing
the process of reading is extensive (Schallert, Fairbanks, Worthy,
Maloch, & Hoffman, 2002; Schmidt, Rozendal, & Greenman,
2002; Stahl & Hayes, 1997), children generally progress through
developmental processes involving letter and word recognition,
decoding, comprehension, and fluency to become skilled readers.
The current accountability climate associated with implementation
of the No Child Left Behind Act of 2001 (NCLB) (P.L. 107-110),
outlines a national educational goal that all children will be
able to read by the end of grade 3. A key component of NCLB is
known as, Reading First, a $900 million grant program that promotes
the use of scientifically based research to provide high-quality
reading instruction for grades K-3, in order to help every student
in every state become a successful reader (http://www.nochildleftbehind.gov/start/facts/readingfirst.html).
This federal mandate has raised awareness about the importance
of helping all children learn to read and the value of interventions
that make a difference.
In grade 4 and beyond, problems in reading are magnified given
that the focus of instruction shifts from learning to read to
reading to learn. The predominant instructional model is based
on learning from print (Hynd, 1998; Sorrells & Britton, 1998).
As a result, how can students demonstrate appropriate academic
achievement when the instructional model expects reading proficiency
at grade level and their disability reflects skills at a much
lower level?
McCormick (1999) indicates that the most useful discrepancies
for determining whether a reading delay is serious enough to
warrant instruction in a special program are: primary grades,
1 year; intermediate grades, 2 years; secondary level, 3 years.
If these criteria are used, about 15% of the students in the
United States need remedial, clinical, or LD reading services,
and about 3% of all students have severe reading problems. Many
labels are used to describe students with reading difficulties:
delayed reader, struggling reader, disabled reader, dyslexic,
print disabled, and learning disabled. One of the leading reasons
for referral to special education involves reading difficulties.
Estimates suggest that 80% of students with learning disabilities
receive services for a reading disability (Bryant, Young, &
Dickson, 2001). The knowledge base documenting the array of skill
deficits and specific instructional and remedial reading interventions
for students with disabilities is extensive (Bos & Vaughn,
2001; Meese, 2000; Miller, 2001).
Despite the critical importance of learning how to read, the
knowledge base on beginning and struggling readers is disproportionately
focused on instruction and remediation. That is, helping students
acquire the necessary skills to read independently. However,
if remedial approaches always worked, we would never see high
school students that couldn't read independently beyond the second
or third grade level.
What happens when a students fails to learn to read? Historically,
educators search for different instructional methods or materials.
Seldom do they raise the question: Are there other ways of performing
the task? Routine failure to attain appropriate levels of academic
performance should trigger assistive technology consideration.
That is, compensatory strategies that use technology to enhance
performance. To-date, little attention has focused on systemic
decision-making concerning the selection and use of instructional
and assistive technology interventions that make it possible
for students to learn from text when the intrinsic nature of
their disability negatively impacts their decoding, fluency,
and comprehension skills (Edyburn, 2003a).
How Does Technology Enhance Reading Performance?
Reading educators have had a long-standing interest in the use
of technology for helping emerging and struggling readers (Reinking,
1987; Reinking, McKenna, Labbo, & Kieffer, 1998). In recent
years, a variety of instructional interventions that utilize
technology for enhancing reading performance have gained widespread
acceptance for engaging beginning and struggling readers in developmentally
appropriate reading activities.
Some examples of technology enhanced reading interventions include:
concept mapping software like Inspiration used as a tool for
facilitating reading comprehension (http://www.inspiration.com),
language translation tools like Bable Fish for converting texts
from one language to another for English Second Language Learners
(http://world.altavista.com), tiered reading materials provide
similar information at different interest and ability levels
(http://www.windows.ucar.edu), Start-to-Finish books are available
in print, audio, and CDROM formats to support readers with different
learning styles and needs (http://www.donjohnston.com), and Read
180 intimately links reading instruction with assessment (http://www.scholastic.com).
In addition, a variety of efforts have been directed at engaging
students in reading through hypermedia-based reading materials
(Anderson-Inman & Reinking, 1998; Boone, Higgins, Falba,
& Langley, 1993; Pang & Kamil, 2002). While the anecdotal
and descriptive evidence about the value, utility, and promise
of these interventions are adequate to convince many practitioners,
in general, the emerging research base in the area of technology
and reading does not rise to meet the NCLB standard of scientifically-based
research.
Assistive Technology and Reading
The application of technology as assistive technology for individuals
with difficulties in reading has largely been overlooked in the
literature. One leading assistive technology textbook devotes
only one-half page to the topic of assistive technology and reading
(Cook and Hussey, 2002). Since reading is primarily a cognitive
function, this gap in the literature is consistent with the profession's
developmental delay in recognizing applications of assistive
technology for disabilities, which are primarily cognitive in
nature, rather than the historical evolution of assistive technology,
which has been driven in response to physical and sensory disabilities
(Edyburn, 2003b, 2000).
Current efforts to harness the potential of technology for struggling
readers are responses based on marketplace developments. The
abundance of marketplace tools that involve text-to-speech (e.g.,
CAST eReader, Kurzweil 3000, PDF Aloud, Reading Bar, ReadPlease,
TextHelp, WordQ, Write:Outloud, WYNN) hold particular promise
that sometime soon, everyone will be able to click on words or
select portions of text and have the information read to them.
While the literature describes a number of applications of text-to-speech
for struggling readers (Cavanaugh, 2002; Lankutis, 2001; Poftak,
2001), the research is promising but so far inconclusive that
these tools enhance reading performance and subsequent educational
achievement (Boyle, Washburn, Rosenberg, Connelly, Brinckerhoff,
& Banerjee, 2002; Dawson, Venn, & Gunter, 2000; Willis,
Koul, & Paschall, 2000).
Measuring Outcomes When Technology is Used
to Enhance Reading Performance
Interest in the measurement of assistive technology outcomes
is a relatively recent phenomenon. In contrast, there is a considerable
literature on measuring the reading abilities of children. In
general, little information is available to inform decision-making
about assistive technology for learning. To-date, we have identified
limited information in the literature regarding measuring the
outcomes of assistive technology as it could be used to enhance
reading performance and educational achievement. In practical
terms, this means the profession does not have a consistent response
to critical questions, such as: If a child has repeatedly failed
to read and understand printed text, how much failure data do
we need before we have enough evidence that the child can't perform
the task? When do we intervene? And, what do we do about it?
For the purpose of this discussion, let's consider one example
of assistive technology for a student with a significant reading
disability. The IEP team has documented decoding, reading rate,
and fluency skills significantly below grade level. However,
the student displays good comprehension when information is read
to him. After trying a variety of text-to-speech products, the
team selects a system that allows the teacher to scan portions
of the textbook into the computer and the student is able to
listen to the material as it is read to him. Given that the student's
performance indicates substandard performance in reading, and
the IEP team has identified what it feels is appropriate assistive
technology, how does one measure the outcome of the assistive
technology?
As a member of the Assistive Technology Outcomes Measurement
System (ATOMS) Project (http://www.atoms.uwm.edu), we have identified
the following design, measurement, analysis, and decision-making
factors that will need to be addressed in the process of creating
outcome systems for measuring the impact of assistive technology:
o Conceptual foundations
o Research designs
o Standardization of the performance task
o Standardization of the data collection and coding process
o Analyze results using standardized metrics and benchmarks
o Decision-making
In the sections that follow, each component is described in the
context of trying to answer the question: How do you measure
the outcomes of assistive technology in reading?
Conceptual Foundations
Historically, not being able to read meant someone had to read
everything for you. Personal readers and books on tapes are examples
of the limited palette of compensatory strategies that have been
made available to individuals with a reading disability. The
concept of using technology to read everything for you is not
an idea that has been extensively consideration in the reading
literature. Rather, such ideas are more common in science fiction.
As a result, there is an urgent need for theoretical foundations
to guide the development and use of reading assistive technologies.
Two recent works provide important frameworks for filling a void
in the area of assistive technology for struggling readers. Dyck
and Pemberton (2002) advanced a model for making decisions about
text adaptations and outlined the theoretical rationale for five
types of text adaptations: bypass reading, decrease reading,
support reading, organize reading with graphic organizers, and
guide reading. Inspired by the Dyck and Pemberton model but disappointed
that assistive technology was not critical to the interventions,
Edyburn (2003a) created a taxonomy of text modification strategies
that highlighted both instructional and assistive technology
interventions. These preliminary efforts are encouraging developments
for both research and practice in understanding the application
of assistive technology in reading.
Research Designs
Central to the definition of assistive technology is the expectation
of enhanced performance. Smith (2000) outlines a theoretical
view known as Time Series Concurrent and Differential (TSCD)
Approach which involves a series of performance measures of an
individual when s/he is completing a specific task, with AT,
and without AT. Ideally, the results reflect a pattern that shows
growth in improved performance in both conditions, however, the
performance with AT is significantly greater than the performance
without AT. The differences between the two measurements isolates
the specific impact of AT and provides evidence of the impact
and outcome over time.
The general utility of this approach for research seeking to
measure the outcomes of assistive technology for reading is unknown.
While the approach is potentially useful and practical for measuring
component skills of the reading process (e.g., reading rate,
comprehension questions answered correctly), the challenges of
measuring less discrete components of the reading process (e.g.,
reading enjoyment) is problematic.
Alternatively, A-B-A single-subject research designs will reveal
interesting patterns of performance for an individual during
baseline (without AT), intervention (with AT), and revert to
baseline concerning any number of variables (e.g., minutes spent
reading, comprehension questions answered correctly, etc.). This
methodology is essential for making decisions about the effectiveness
of specific assistive technology for an individual student but
has limitations for informing general professional practice.
The importance of identifying high-quality research methodology
for reading assistive technology research is critical. The ongoing
controversy surrounding the National Reading Panel (2000) report,
the subsequent debates and critiques of research methodology
for defining best practice (Block & Pressley, 2002; Kamil,
Mosenthal, Pearson, & Barr, 2002), concerns about what types
of data count as evidence of reading achievement (Murphy, Shannon,
Johnston, & Hansen, 1998), and the use and misuse of research
in policymaking (Pressley, 2002; Smith, 2003) are instructive
for our embryonic efforts.
Standardize the Performance Task
Reading performance is often assessed through informal reading
inventories, curriculum-based assessments, and standardized tests.
Typically, assessments of reading involve a variety of skills
(letter and word recognition, vocabulary, reading rate, and comprehension
(e.g., literal, inferential, critical). While there are a number
of common standardized measures of reading achievement, no single
test is a recognized standard. As a result, measuring the outcomes
of reading assistive technology will be more problematic than
assessment of outcomes of assistive technology for math and writing
which have standardized representative tasks to facilitate comparison
of performance between students.
Standardize the Data Collection and Coding Process
A routine measure of reading comprehension involves answering
assorted questions about the material a person has read. While
the nature of the comprehension questions will vary from assignment
to assignment, comprehension question taxonomies illustrate a
fairly standard set of questions (e.g., main idea, fact retrieval,
inferential). Typically, data is collected concerning the percentage
of comprehension questions answered correctly. As a result, standardizing
comprehension question sets in multiples of 5 or 10 to permit
conversion to percentage appears desirable.
Two recent advances in the measurement of reading performance
offer significant potential for standardizing the data collection
and coding process associated with measuring the outcomes of
reading assistive technologies. The Dynamic Indicators of Basic
Early Literacy Skills (DIBELS) are a set of standardized, individually
administered measures of early literacy development (http://dibels.uoregon.edu/).
DIBELS are one-minute fluency assessments that allow teachers
to gather important curriculum-based assessment data on students'
pre-reading and early reading performance. Another innovation
in the area of measuring reading performance is the development
of the Lexile Framework for Reading (http://www.lexile.com/).
Lexiles are a computation metric used to assess the difficulty
of text and the skill of the reader. The purpose is to predict
the difficulty an individual reader will have with a given text
in order to match the reader with appropriate materials for reading
instruction.
Analyze Results using Standardized Metrics and Benchmarks
Exceptional performance in reading is often characterized by
high rates of fluency and comprehension. Scores on standardized
reading assessments are often transformed into percentiles to
illustrate how an individual's performance compares with others
similar in age. While grade equivalents are sometimes used, often
they have been abandoned due to their technical inadequacy and
significant potential for misuse.
The Matthew Effect (Stanovich, 1986) is a well-documented effect
in reading education. Based on a Biblical metaphor about the
rich getting richer, it means that while young children may display
small differences in reading ability, over time, the differences
become much larger such that effective readers exponentially
become more proficient and learn more while poor readers fall
farther behind. Understanding the long-term net effect of a reading
disability has significant implications for analyzing the results
of data collected concerning the use of reading assistive technology.
That is, while text-to-speech technology may provide short-term
improvement in reading comprehension, will such gains be adequate
for closing the achievement gap with non-handicapped peers? If
not, what does this mean for measuring the outcome of the assistive
technology? If reading assistive technology shows promising potential
in short-term gains, should this trigger intensive assistive
technology-based remedial interventions to try and close the
multi-year gap? Questions like these illustrate the urgent need
for significant philosophical and theoretical work regarding
the nature of assistive technology for enhancing reading performance.
Decision-making
Reading education is concerned with two primary skills sets:
learning to read (grades K-3) and reading to learn (grades 4
and beyond). Students with disabilities often experience developmental
delays that limit the benefit they receive from typical reading
instruction in the early grades and then are penalized throughout
the rest of their academic career because their reading skill
sets are not at grade level as the curriculum utilizes a one-size-fits-all
reading-to-learn model in grades 4 and beyond. At the present
time, little is known about how much failure data needs to accumulate
before educators recognize that a child is unable to read and
in need of reading assistive technologies. Edyburn (2003a) has
argued that this issue is probably not either/or, but rather,
what percentage of time/effort should be devoted to instruction
and what percentage of time/effort should reading compensation
technologies be provided so that the student can have access
to the information.
As reading assistive technology performance data is collected,
analysis of the student performance data should reveal several
factors that will inform decision-making. First, does the graph
indicate that reading performance with assistive technology is
higher than performance without assistive technology? If so,
the case can be made that the assistive technology is an effective
intervention for enhancing performance. If not, the data suggest
the need additional training or a different intervention.
Second, do the data reflect that the student is able to meet
the performance standard (i.e., 80% comprehension)? If so, the
case can be made that the reading assistive technology effectively
compensates for the person's disability. If the performance standard
is not met, the IEP team needs to explore whether additional
time is needed for developing mastery, whether additional interventions
must be applied concomitantly, or whether a different intervention
is needed.
Finally, can high levels of performance be maintained over time?
That is, will the routine use of the assistive technology result
in consistent high-quality performance in reading? Is there any
evidence that the assistive technology is closing the achievement
gap known as the Matthew Effect (Stanovich, 1986)?
Summary
The purpose of this article was to provide an introduction to
the measurement issues associated with measuring assistive technology
outcomes in reading. While the research and pedagogical knowledge
base which informs current instructional practice concerning
students with disabilities and reading is considerable, much
more work needs to be undertaken to determine the kinds of assistive
technology that enhance reading performance. In contrast to other
academic applications of assistive technology in writing and
math, the area of reading appears to lag significantly behind
in the development of measurement tools and procedures that will
enable the profession to make definitive statements about outcomes
of technology enhanced performance in reading.
References
Anderson-Inman, L., & Reinking, D. (1998). Learning from
text in a post-typographic world. In C.R. Hynd, C.R. (Ed.). Learning
from text across conceptual domains. Mahwah, NJ: Lawrence
Erlbaum, pp. 165-191.
Block, C.C., & Pressley, M. (2002). Comprehension instruction:
Research-based best practices. NY: Guilford Press.
Boone, R., Higgins, K., Falba, C., & Langley, W. (1993).
Cooperative text: Reading and writing in a hypermedia environment.
LD Forum, 19(1), 28-37.
Bos, C.S., & Vaughn, S. (2001). Strategies for teaching
students with learning and behavior problems (5th ed.). Boston:
Allyn & Bacon.
Boyle, E.A., Washburn, S.G., Rosenberg, M., Connelly, V.J., Brinckerhoff,
L.C., & Banerjee, M. (2002). Reading's SLICK with new audio
texts and strategies. Teaching Exceptional Children, 35(2),
50-55.
Bryant, D.P., Young, S., & Dickson, C. (2001). Secondary
level students with reading disabilities: No time to waste. [WWW
document] URL http://www.ldao.on.ca/articles/spec_ed_secondary.html.
Cavanaugh, T. (2002). eBooks and accommodations: Is this the
future of print accommodations? Teaching Exceptional Children,
35(2), 56-61.
Cook A.M., & Hussey, S.M. (2002). Assistive technology:
Principles and practices (2nd ed.). St. Louis, MO: Mosby.
Dawson, L., Venn, M.L., & Gunter, P.L. (2000). The effects
of teacher versus computer reading models. Behavioral Disorders,
25(2), 105-113.
Dyck, N., & Pemberton, J.B. (2002). A model for making decisions
about text adaptations. Intevention in School and Clinic,
38(1), 28-35.
Edyburn, D.L. (2003a). Learning from text. Special Education
Technology Practice, 5(2), 16-27.
Edyburn, D.L. (2003b). Rethinking assistive technology. Special
Education Technology Practice, 5(4), 16-22.
Edyburn, D.L. (2000). Assistive technology and students with
mild disabilities. Focus on Exceptional Children, 32(9),
1-24.
Hynd, C.R. (1998). Learning from text across conceptual domains.
Mahwah, NJ: Lawrence Erlbaum.
Kamil, M.L., Mosenthal, P.B., Pearson, P.D., & Barr, R. (2002).
Methods of literacy research. Mahwah, NJ: Lawrence Erlbaum.
Lankutis, T. (2001). Reaching the struggling reader. Technology
and Learning, 21(10), 24-32.
McCormick, S. (1999). Instructing students who have literacy
problems (3rd ed.). Columbus, OH: Merrill, p. 15.
Meese, R.L. (2000). Teaching learners with mild disabilities:
Integrating research and practice (2nd ed.). Belmont, CA:
Wadsworth.
Miller, S.P. (2001). Validated practices for teaching students
with diverse needs and abilities. Boston: Allyn & Bacon.
Murphy, S. Shannon, P., Johnston, P., & Hansen, J. (1998).
Fragile evidence: A critique of reading assessment. Mahwah,
NJ: Lawrence Erlbaum.
National Reading Panel. (2000). Teaching children to read:
An evidence-based assessment of the scientific research literature
on reading and its implications for reading instruction.
Washington, DC: U.S. Department of Health and Human Services.
Pang, E.S., & Kamil, M.L. (2002). Reading hypertext: Comprehension
and strategies of third grade students. In D.L. Schallert, C.M.
Fairbanks, J. Worthy, B. Maloch, & J.V. Hoffman, (Eds.).
51st yearbook of the National Reading Conference. Oak
Creek, WI: National Reading Conference, pp. 333-343.
Poftak, A. (2001). Getting a read on e-books. Technology and
Learning, 21(9), 22-36.
Pressley, M. (2002). What I have learned up until now about research
methods in reading education. In D.L. Schallert, C.M. Fairbanks,
J. Worthy, B. Maloch, & J.V. Hoffman, (Eds.). 51st yearbook
of the National Reading Conference. Oak Creek, WI: National
Reading Conference, pp. 33-45.
Reinking, D. (1987). Reading and computers: Issues for theory
and practice. NY: Teachers College Press.
Reinking, D., McKenna, M.C., Labbo, L.D., & Kieffer, R.D.
(1998). Handbook of literacy and technology. Transformations
in a post-typographic world. Mahwah, NJ: Lawrence Erlbaum.
Schallert, D.L., Fairbanks, C.M., Worthy, J., Maloch, B., &
Hoffman, J.V. (2002). 51st yearbook of the National Reading
Conference. Oak Creek, WI: National Reading Conference.
Schmidt, R.J., Rozendal, M.S., & Greenman, G.G. (2002). Reading
instruction in the inclusion classroom: Research-based practices.
Remedial and Special Education, 23(3), 130-140.
Smith, F. (2003). Unspeakable acts, unnatural practices: Flaws
and fallacies in "scientific" reading instruction.
Portsmouth, NH: Heinemann.
Smith, R.O. (2000). Measuring assistive technology outcomes in
education. Diagnostique, 25, 273-290.
Sorrells, R.C., & Britton, B.K. (1998). What is the point?
Tests of a quick and clean method for improving instructional
text. In C.R. Hynd (Ed.) Learning from text across conceptual
domains, (pp. 95-115). Mahwah, NJ: Lawrence Erlbaum Associates.
Spires, H.A., & Estes, T.H. (2002). Reading in web-based
learning environments. In C.C. Block, & M. Pressley, (Eds).
Comprehension instruction: Research-based best practices.
NY: Guilford Press, pp. 115-125.
Stahl, S.A., & Hayes, D.A. (1997). Instructional models
in reading. Mahwah, NJ: Lawrence Erlbaum.
Stanovich, K.E. (1986). Matthew effects on reading: Some consequences
of individual differences in the acquisition of literacy. Reading
Research Quarterly, 21, 360-407.
Willis, L.H., Koul, R.K., & Paschall, D.D. (2000). Discourse
comprehension of synthetic speech by individuals with mental
retardation. Education and training in mental retardation
and developmental disabilities, 35(1), 106-114.