JSET ejournal
Research
& Practice
Associate Editor Column
Dave Edyburn
Measusing Assistive Technology Outcomes in Mathematics
Given the importance of the topic of assistive technology
outcomes to both researchers and practitioners, the Research
and Practice Column in JSET has been devoted to a series of articles
on measuring assistive technology outcomes in specific academic
domains. In previous columns I provided an overview of key concepts
associated with measuring assistive technology outcomes (18-1)
and assessing assistive technology outcomes in writing (18-2).
In the final column of this series (19-1), I will examine issues
associated with measuring assistive technology outcomes in reading.
In this column (18-4), I focus on issues of assessing assistive
technology outcomes in mathematics. Compared with the areas of
reading and writing, one might assume that quantifying the outcomes
of assistive technology on math performance is easier. Perhaps,
it is in some respects. However, given the historical reluctance
to use the calculator as a cognitive prostheses for students
who struggle, the role of assistive technology in mathematics
may be a useful case study for understanding the general acceptance
of assistive technology for enhancing learning and performance.
Mathematics and Technology
As special education teachers prepare IEP objectives, they
are well aware of the many students that struggle with in math
with a variety of tasks:
Basic Skills
Inability to learn basic facts
Difficulties using multi-step algorithms (i.e., long division)
Functional Applications
Counting
Telling time
Making change
Applied Applications
Solving real-world problems
Difficulties in mathematics are a common characteristic of students
with disabilities. A considerable research knowledge base documents
an extensive list of skill deficits and details specific instructional
and remedial interventions (Bley, & Thorton, 2000; Bos &
Vaughn, 2001; Butler, Miller, Lee, & Pierce, 2001; Fuchs
& Fuchs, 2001; Maccini & Cagnon 2000; Meese, 2000; Miller,
2001; Woodward, Baxter, & Robinson, 1999; Xin, & Jitendra,
1999).
Despite the breadth of what is known, given the number of students
who have failed to master basic mathematical skills, questions
must be raised concerning whether we know all that we need to
in order to help every child achieve high levels of mathematical
proficiency. Over the past 25 years, the literature reveals a
profound interest in the application of instructional and assistive
technology to the challenges of helping students acquire mathematical
knowledge and skills.
How Does Technology Enhance Mathematical Performance?
Researchers have used a variety of strategies to harness
the power of technology in ways that enhance students' skills
and knowledge in mathematics. Among the more recent applications
found in the literature: instructional strategies designed into
software (Irish, 2002), advanced interactive groupware that evaluates
students' real-time performance (Shin, Deno, Robinson, &
Marston, 2000); video-based problem solving environments (Bottege,
2002, 2001, 1999), calculator use (Horton, Lovitt, & White,
1992), and hand-held computing devices (Bauer & Ulrich, 2002;
Woodward, & Montague, 2002).
As is often the case with technology, advances in the marketplace
provide a rich array of possibilities for practioners to explore.
Recent applications of technology in math for students with disabilities
have explored: adaptive calculators (Pellerito, 2001); drill
and practice via the web (Wissick, 2000); problem solving via
the web (Christle, Hess, & Hasselbring, 2001; Miller, Brown,
& Robinson, 2002); spreadsheets for calculating and problem
solving (Anderson & Anderson, 1999); software tools for visualizing
data (Anderson & Anderson, 2001); virtual manipulatives (Riley
& Beard, 2002); and applications of universally design software
in math (Stanger, Symington, Miller, & Johns, 2000).
Assistive Technology and Mathematics
The primary focus of mathematics instruction is helping
students acquire the cognitive abilities to manipulate numbers,
using specific algorithms, and to solve routine and novel problems.
Mathematics is a cognitive discipline. While computers are routinely
used to solve problems that are too complex for humans, K-12
education has been reluctant to recognize and embrace the use
of tools that enhance cognitive performance. The rationale for
this position is based on the belief that calculating aids undermine
acquisition of the mental discipline necessary for learning basic
facts, operations, and algorithms.
Paper and pencil are used simply as an aid to the internal problem
solving. The use of manipulatives is a relatively recent addition
to the mathematics instructional toolkit. Manipulatives are viewed
as a transitional strategy that initially makes learning more
meaningful by engaging students in a visual and interactive learning
environment. Ultimately, manipulatives are abandoned as students
gain fluency and automaticity in problem solving. Calculators
and computers are sometimes simply considered to be another form
of math manipulative. However, this view severely underestimates
their distinct value for calculating and problem solving. As
a result, when is it appropriate to blur the distinction between
the use of technology as a manipulative vs. technology as a cognitive
prosthesis?
One problem that prevents the field from rethinking mathematics
instruction and the role of technology in calculating and problem
solving is the general ban of calculators in high-stakes testing.
Teachers rationalize since calculators cannot be used on the
tests, they should not be used in instruction. The amount of
recent work in the area of test accommodations and their impact
on math achievement (Calhoon, Fuchs, & Hamlett, 2000; Helwig,
Rozek-Tedesco, & Tindal, 2002; Helwig, Anderson, & Tindal,
2002; Hollenbeck, Rozek-Tedesco, Tindal, & Glasgow, 2000;
Johnson, 2000) is encouraging but a patchwork of policies and
practices are currently in-place. Embedded in this discussion
is a bias that the only acceptable performance is that which
you can do without assistance of any type (i.e., raw power, naked
independence). Given the extensive use of technology in advanced
math and science, inadequate attention has been given to the
interaction of student, tool, and performance in K-12 mathematics
instruction. This factor may explain the general reluctance to
allow students with disabilities to use calculators or other
forms of assistive technology as a routine part of math instruction.
An interesting, but conflicting finding, concerning calculator
use by students was recently reported using data from the 2000
Mathematics Assessment of the National Assessment of Educational
Progress. At grade 4 more frequent calculator use was associated
with lower scores, while at grades 8 and 12 the opposite was
generally true students who said they used calculators more
often tended to score higher than their peers who reported using
calculators less frequently (National Center for Education Statistics,
2003).
Measuring Outcomes When Technology is Used
to Enhance Mathematical Performance
Interest in the measurement of assistive technology outcomes
is a relatively recent phenomenon. While there is considerable
literature on measuring mathematics abilities of children, there
is little in the literature regarding assistive technologies
that enhance mathematical performance and achievement.
For the purpose of this discussion, let's consider a few examples
of assistive technology. Obviously, calculators can be used by
any student struggling to master basic math facts. For students
with disabilities where physical limitations prevent manipulation
of the calculator keys, onscreen calculators like the Big Calc
(Don Johnston, Inc.) are a useful alternative (Pellerito, 2001).
A variety of software products offer support in teaching students
about specific algorithms and assistance in the calculating aspects
of problem solving through simplified spreadsheets (Anderson
& Anderson, 1999). Finally, specialized devices like the
Coin-U-Lator (Attainment Co.) serve as a specialized calculator
for problems in totaling costs and making change. When a student's
performance indicates substandard performance and the IEP identifies
what it feels is appropriate assistive technology, how does one
measure the outcome of the assistive technology?
As a member of the Assistive Technology Outcomes Measurement
System (ATOMS) Project, we have identified the following design,
measurement, analysis, and decision-making factors that will
need to be addressed in the process of creating an outcomes systems
for measuring the impact of assistive technology:
Repeated measures research design
Standardize the performance task
Standardize the data collection and coding process
Analyze results using standardized metrics and benchmarks
Decision-making
In the sections that follow, I will describe each component in
the context of trying to answer the question: How do you measure
the outcomes of assistive technology in math?
Repeated Measures Research Design
Central to the definition of assistive technology is the
expectation of enhanced performance. Smith (2000) outlines a
theoretical view known as Time Series Concurrent and Differential
(TSCD) Approach which involves a series of performance measures
of an individual when s/he is completing a specific task, with
AT, and without AT. Ideally, the results reflect a pattern that
shows growth in improved performance in both conditions, however,
the performance with AT is significantly greater than the performance
without AT. The differences between the two measurements isolates
the specific impact of AT and provides evidence of the impact
and outcome over time.
The general utility of this approach for research seeking to
measure the outcomes of assistive technology for math is potentially
useful and practical. Given that teachers routinely assign problem
sets, data can be gathered on a daily, weekly, or monthly basis
that will provide measures sensitive enough to demonstrate change.
In addition, the research design will permit the collection of
data from students without disabilities, which is valuable for
subsequent performance benchmarking efforts.
Standardize the Performance Task
In contrast to other academic tasks, the mathematics curriculum
has a high degree of standardization concerning the types of
tasks believed to reflect an accurate sample of desired math
outcomes.
Standardize the Data Collection and Coding Process
In math, daily math assignments provide a ready-access form
of student performance data. Each problem is scored correct or
incorrect. The number of correct problems is divided by the total
number of problems to produce a percentage of the number of problems
accurately completed. This procedure is common and robust across
topics within the curriculum. As a result, the math curriculum
provides teachers with considerable opportunity and experience
in standardized data collection and coding processes.
Analyze Results using Standardized Metrics and Benchmarks
Exceptional performance in math is often characterized by
speed and accuracy. As a result, teachers and curriculum developers
often set benchmarks involving the percentage of problems completed
correctly divided by the time required to complete to yield a
single score of correct problems completed per minute. While
the expectation may vary across grade level, this metric can
be calculated across the math curriculum to provide a standard
performance benchmark for making comparisons.
Decision-making
Two outcomes of mathematics instruction are often discussed:
the ability to calculate accurately and the ability to use one's
mathematical knowledge to solve problems correctly. Students
with disabilities often fail on both accounts because they lack
the basic skills necessary to perform math calculations with
appropriate fluency and accuracy and they cannot apply their
knowledge and skills to solve problems. Hence, the ultimate question
involved in the assessment of assistive technology outcomes in
math focuses on the student's ability to independently solve
routine and novel problems. Analysis of the student performance
data should reveal several factors that will inform decision-making.
First, does the graph indicate that performance with AT is higher
than performance without AT? If so, the case can be made that
the AT is an effective intervention for enhancing performance.
If not, the data suggest the need additional training or a for
a different intervention.
Second, do the data reflect that the student is able to meet
the performance standard (i.e., 100% accuracy)? If so, the case
can be made that the AT effectively compensates for the person's
disability. If the performance standard is not met, the IEP team
needs to explore whether additional time is needed for developing
mastery, whether additional interventions must be applied concomitantly,
or whether a different intervention is needed.
Finally, can high levels of performance be maintained over time?
That is, will the routine use of the assistive technology result
in consistent high-quality performance in math?
Summary
The purpose of this column was to provide an introduction
to the measurement issues associated with measuring assistive
technology outcomes in math. While the research and pedagogical
knowledge base which informs current instructional practice concerning
students with disabilities and math is considerable, much more
work needs to be undertaken to determine the kinds of assistive
technology that enhance math performance. In contrast to other
academic applications of assistive technology, the field of math
appears to offer considerable measurement tools and procedures
that will enable the profession to make definitive statements
about performance outcomes in math.
References
Anderson, K.M., & Anderson, C.L. (2001). Using software to
help visualize mathematical processes. Special Education Technology
Practice, 3(2), 13-18.
Anderson, C., & Anderson, K. (1999). Integrating spreadsheets
into mathematics for students with disabilities. Special Education
Technology Practice, 1(4), 37-39.
Bauer, A.M., & Ulrich, M. E.. (2002). "I've Got a Palm
in My Pocket:" Using handheld computers in an inclusive
classroom. Teaching Exceptional Children, 35(2), 18-22.
Bley, N.S. & Thorton, C.A. (2000). Teaching mathematics
to students with learning disabilities. Austin, TX: Pro-Ed.
Bos, C.S., & Vaughn, S. (2001). Strategies for teaching
students with learning and behavior problems (5th ed.). Boston:
Allyn & Bacon.
Bottge, B.A., Heinrichs, M., Mehta, Z.D., & Hung, Y. (2002).
Weighing the benefits of anchored math instruction for students
with disabilities in general education classes. The Journal
of Special Education, 36(4), 186-200.
Bottge, B.A. (2001). Building ramps and hovercraftsand improving
math skills. Teaching Exceptional Children, 34(1), 16-23.
Bottge, B.A. (1999). Effects of contextualized math instruction
on problem solving of average and below-average achieving students.
Journal of Special Education, 33, 81-92.
Butler, F.M., Miller, S.P., Lee, K., & Pierce, T. (2001).
Teaching mathematics to students with mild-to-moderate mental
retardation: A review of the literature. Mental Retardation,
39, 20-31.
Calhoon, M.B., Fuchs, L.S., & Hamlett, C.L. (2000). Effects
of computer-based test accommodations on mathematics performance
assessments for secondary students with learning disabilities.
Learning Disabilities Quarterly, 23(4), 271-282.
Christle, C.A., Hess, J.M., & Hasselbring, T.S. (2001). Technology
research in practice: Taking a virtual trip to the mall to learn
math. Special Education Technology Practice, 3(2), 23-31.
Fuchs, L.S., & Fuchs, D. (2001). Principles for the prevention
and intervention of mathematics difficulties. Learning Disabilities
Research and Practice, 16, 85-95.
Helwig, R., Anderson, L., & Tindal, G. (2002). Using a concept-grounded,
curriculum-based measure in mathematics to predict statewide
test scores for middle school students with LD. The Journal
of Special Education, 36(2), 102-112.
Helwig, R., Rozek-Tedesco, M.A., & Tindal, G. (2002). An
oral versus a standard administration of a large-scale mathematics
test. The Journal of Special Education, 36(1), 39-47.
Hollenbeck, K., Rozek-Tedesco, M.A., Tindal, G., & Glasgow,
A. (2000). An exploratory study of student-paced versus teacher-paced
accommodations for large-scale math tests. Journal of Special
Education Technology, 15(2), 27-36.
Horton, S., Lovitt, T., & White, O. (1992). Teaching mathematics
to adolescents classified as educable mentally handicapped; Using
calculators to remove the computational onus. Remedial and
Special Education, 13(3), 36-60.
Irish, C. (2002). Using peg- and keyword mnemonics and computer-assisted
instruction to enhance basic multiplication performance in elementary
students with learning and cognitive disabilities. Journal
of Special Education Technology, 17(4), 29-40.
Johnson, E.S. (2000). The effects of accommodations on performance
assessments. Remedial and Special Education, 21(5), 261-267.
Maccini, P., & Cagnon, J.C. (2000). Best practices for teaching
mathematics to secondary students with special needs. Focus
on Exceptional Children, 32(5), 1-22.
Meese, R.L. (2000). Teaching learners with mild disabilities:
Integrating research and practice (2nd ed.). Belmont, CA:
Wadsworth.
Miller, D., Brown, A., & Robinson, L. (2002). Widgets on
the Web: Using computer-based learning tools. Teaching Exceptional
Children, 35(2), 24-28.
Miller, S.P. (2001). Validated practices for teaching students
with diverse needs and abilities. Boston: Allyn & Bacon.
National Center for Education Statistics. (2003). National Assessment
of Educational Progress (NAEP), 2000 Mathematics Assessment.
Available online: http://nces.ed.gov/nationsreportcard/mathematics/results/calculator.asp
Pellerito, F. (2001). Go figure: Calculator options for students
with special needs. Special Education Technology Practice,
3(2), 19-22.
Riley, G.W., & Beard, L.A. (2002). On-screen math manipulatives:
Virtual access for students unable to handle traditional manipulatives.
Special Education Technology Practice, 4(2), 40-42.
Shin, J., Deno, S.L., Robinson, S.L., & Marston, D. (2000).
Predicting classroom achievement from active responding on a
computer-based groupware system. Remedial and Special Education,
21(1), 53-60.
Smith, R.O. (2000). Measuring assistive technology outcomes in
education. Diagnostique, 25, 273-290.
Stanger, C., Symington, L., Miller, H., & Johns, S. (2000).
Teaching number concepts to ALL students. Teaching Exceptional
Children, 33(1), 65-69.
Wissick, C.A. (2000). Drill and practice: Web style. Special
Education Technology Practice, 2(3), 37-39.
Woodward, J., & Montague, M. (2002). Meeting the challenge
of mathematics reform for students with LD. The Journal of
Special Education, 36(2), 89-101.
Woodward, J., Baxter, J., & Robinson, R. (1999). Rules and
reasons: Decimal instruction for academically low achieving students.
Learning Disabilities Research and Practice, 14, 15-24.
Xin, Y. P., & Jitendra, A. K. (1999). The effects of instruction
in solving mathematical word problems for students with learning
problems: A meta-analysis. The Journal of Special Education,
32(4), 207-225.