Developing a Simple Test Blueprint/SpecificationPosted: June 25, 2009 | Author: Jamie Madigan | Filed under: Uncategorized | 1 Comment »
I’ve been doing some research lately on developing employment test content from scratch, just like Grandma used to make. One of the steps in this process that’s pretty widely recommended is the creation of a test blueprint (also sometimes called a test specification). Basically it’s a short document that maps out what the test is about, what content areas it will tap, and some other information.
At first I glossed over this because it seemed like an example of over documentation and a likely candidate for omission from a project if you’re in a hurry. As I got further closer and closer to the point of actually writing test content, though, I realized that this is really an invaluable step. Mainly because it forces you to map things out and think carefully about what your test should look like. It will also likely keep you from wasting the time of item writers –if you need twice as many vocabulary items as math, you can appropriately direct the activities of your writers.
So this post is all about creating a simple test blueprint for an employment test. It’s not meant to be definitive, and if you have suggestions or pointers to other resources, please share them in the comments section.
First, though, we’ll need a job. Let’s just pick one totally at random, like say, Human Cannonball, which is Code # 159.347-018 in the Dictionary of Occupational Titles. Just picked that one totally at random. Honest. Now, of course, any test development activity should be preceded by a thorough job analysis, so let’s assume you did that and identified these knowledge, skills, and abilities to include in your selection program:
- Spatial Orientation
- Mathematical Reasoning
- Knowledge of the English language
Right. Okay. Let’s look at what sections to include in your test blueprint:
Purpose of the Test
No need to get fancy. This can just be a sentence or two about how you plan to use the test. “The test will be used as a preliminary screen when selecting candidates for Human Cannonball positions at any of our regional locations.”
Content Areas and Test Length
Here’s the meat of it. In this section you want to break out the different content areas of the test, which will probably map directly on to the KSAs you chose to use. You may, however, want to also break out sub-areas (e.g., breaking arithmetic down into addition, subtraction, multiplication, and division).
You also want to mention the total length of the test (probably by number of questions), then break out the number of questions per content area. Now, let’s assume that your job analysis included some ratings of the criticality of each of these KSAs, so that you rank order them in terms of importance. This is important, since the amount of test content for each KSA should positively correlate to its relative importance. If Spatial Orientation is twice as important as Knowledge of the English Language for Human Cannonballs, then you probably want to have twice as many test items covering it. This is particularly true if you are pursuing a content validity strategy for your test and trying to directly map the content of the test onto the content of the job.
This all can be communicated pretty effectively by a simple table, like this:
Here you should list the question format(s) that you will use. I found that it’s even helpful to identify what formats will be used to tap what content areas, since it gives you a plan to start developing item templates and the items themselves.
Possible question formats include:
- Oral examination
- Fill in the blank
- Alternate choice
- Multiple true/false
- Traditional multiple choice
- Context-dependent multiple choice
- Complex multiple choice
- Extended matching
And probably a bunch more I’m forgetting. But you get the idea.
Desired Psychometric Properties
Depending on the item formats, you might skip this, but generally it’s wise to think about what kinds of psychometric properties you want for your test and place a stake in the ground so you’ll know when you’re on target. You probably want to mention some test-level statistics here (e.g., reliability), but you should also establish your thresholds for item level statistics like difficulty and item discrimination.
Overview of Test Administration and Scoring
Finally, you want to pound out a few words about the test administration and scoring process. The information in this section should answer these questions:
- In what format(s) (e.g., paper, computer) will the test be administered?
- Who will administer the test? Will any certification be required?
- What should be the max number of people to take the test in one session?
- What physical conditions will be required to administer the test?
- What technologies, tools, or other materials will be required to administer the test?
- Will test takers be allowed to use aids such as calculators or dictionaries?
- What will be the time limit
- How will the test be scored? Will items/areas be unit weighted or not
- What will be the retest policy?
So, that’s it. Even something as quick and dirty as what’s outlined above should help get you thinking along the right lines and should make it easier to move on to other steps in the test development process, like item writing and pilot testing. Again, if you have suggestions or pointers, put ’em in the comments!