'The Only Bad Test Is...': Understanding When and What to Test

I have this large, square basket in my living room that sits on top of my bookshelf just gathering dust now, but back in the day that thing was a workhorse.  Two feet square and six inches deep, I could load that thing up with a week’s worth of grading and know what I would be doing all day Saturday and Sunday.  For any of my former students who are reading these, let me just say right now that I am sorry.  I was an idiot.  I know better now. 

See, beside that basket, I also had a very large cylinder about a foot in diameter and two feet deep.  You would call this a trash can, but I called it the round file.  The round file was an excellent place to put paper clipped stacks of worksheets that had sat in the bottom of the basket for too long and were no longer relevant.  Hindsight being 20/20, the moment that those stacks of papers made it IN to the basket, they were most likely no longer relevant.  If I could wait for the weekend to grade Monday’s assignment, then what was its value?  What was its purpose?  Class had moved on.  We didn’t learn about something on Monday to talk about the mistakes that were made the following Monday. 

Eventually I broke that habit and the round file got used less and less.  I learned how to assign the right questions – not all the questions.  I learned how to track incorrect answers across the papers to find mistakes in the learning.  I learned how to provide more immediate feedback.  And I also learned how to use a pretest correctly to guide my instruction. 

Pretests seem like easy things to use, right?  Give a student this magic little test before you start teaching a chapter, and then you give them a posttest after all the teaching, and then you can see if they learned what you taught.  Right? WRONG!  Pretests, if they are well designed, aren’t going to tell you what the students need to learn.  They are going to tell you what the students might not know that they should know before starting to learn.  Getting ready to do multi-digit multiplication? Do they know the multiplication facts? Going to diagram sentences? Can they identify parts of speech?  A well-designed pretest should tell you what skills you might need to review as you are teaching the new ones.  Once I understood that was how they should work, I was golden. 

Luckily, this philosophy will also scale!  You can do a pretest for a chapter, a unit, a semester, and even a school year.  Depending upon the purpose, these tests will cover the entire range of assessment – from formative to benchmark to summative.  The purpose is the key here.  Let me walk you through what I mean with a benchmark assessment.  Benchmark assessments can be used for multiple reasons including measuring mastery and measuring growth.  A good benchmark can do both.  A bad benchmark cannot.  Let me tell you about my experience with a bad benchmark. 

"Know your need, know your purpose, know your test and then get that information."

A long time ago in a school down near the coast, I had a group of 4th graders that were pretty awesome.  Like the rest of the teachers in the school and district, I had to give my 4th graders a benchmark test at the end of every nine weeks.  The first one came and went with little fanfare.  After the scores arrived, I was called to the office to review the scores with the other teachers in the grade level.  Mine were as expected so nothing traumatic.  (Hang on to that word.) The next benchmark comes along, and the same process happens.  I was called to the office to again review the scores.  This time, though, was traumatic.  The principal had highlighted, underlined, outlined, indented, circled, and boldfaced every single one of my scores.  This principal had then gone on to cross-reference, cross-categorize, over-analyze, and cross-compare those scores with the previous benchmark.  And I was now expected to explain to this principal in front of all the other teachers why my class had dropped 5 percentage points in math instead of growing by 10 like the rest of the grade. 

Before we go any further – let’s talk about what this test actually was.  It was a test consisting of questions written by someone at the district office looking at the county pacing guide that was to assess whether the students had learned the topics that had been taught in the previous nine weeks.  This test did not break down how students did on each standard.  It did not use items that had been field tested.  It did not change items year over year.  It was a knowledge-based assessment with little actual application that only provided a percent correct and really should have been tossed out the door years before I got there.  But it hadn’t, and I was about to be the one tossed out the door.  So, let’s go back to the principal’s office. 

While I was not prepared for this onslaught of academic over-analysis, I did know my standards and my pacing guide.  I was able to point out that the first nine weeks had been focused on basic math concepts with a good deal of review from the previous year, and that the second nine weeks was dealing with multi-digit multiplication and long-division.  The lower average score on this new content was to be expected with the more difficult skills. Also, these benchmark assessments were not designed to be compared for growth.  They provided only a percent correct which could be used to measure growth if the standards covered were the same for each test, along with the difficulty levels and thinking skills.  The principal* was not willing to admit defeat, but I was never asked to explain why I did or did not have growth between two benchmark assessments.  I took it as a win and kept on teaching.   

I learned then that benchmark assessments have multiple purposes, and that if I wanted one that was able to both measure growth and measure skills that I wasn’t going to get it by using one that was made locally with no way of developing data behind the items.  These local benchmarks are ok if the need is for something that will tell you about standard mastery but be careful thinking about growth or comparing to other classrooms. 

This year, there is a deeper need for a pretest on a larger scale.  I talked about this in a previous post and said then that you are going to have to test at the beginning of the school year.  You have some options, though, and I wanted to give you some food for thought as you consider what you need for the start of school. 

If you filled out the Classroom Scorecard, or you gathered info from the student records, then you will have an idea of where you are missing data points.  The fastest and most accurate way to replace that information will be with an assessment that covers the need. 

There really is no such thing as a bad test.  Every test when used as it is designed will generate some data.  Where things turn bad is when we use tests for purposes other than for what they are intended, like that benchmark experience I had.  Know your need, know your purpose, know your test and then get that information.  Use this at the start for the school year and you’ll be on track for measuring success throughout the year hopefully while not filling up your round file. 

*The principal really was a very good one after all and I have him to thank for where I am today.  He saw that I was interested in all things assessment and set me up to be an assessment trainer for the school. It allowed me to find and pursue my passion.