THE BOOKMARK STANDARDS SETTING PROCEDURE

CTB has used the procedure designed by Lewis, Mitzel, and Green with more than 20 committees on at least 8 other occasions. Panelists work in small groups with a table leader facilitators who also participate. The work of the panel is a research activity. Panelists will be expected to handle copyrighted, secure test items, respect panel confidentiality, and must participate in the entire process from beginning to end. Table leaders meet prior to large group for facilitator training.

Steps in the Proficiency Standard Setting process:

1. Participants discuss the knowledge, skills, and abilities they expect of students in each of the performance levels:
   Advanced, Proficient, Partially Proficient, Minimal.
   The descriptions are based on what all students should know and be able to do relative to the content of the test, not some idealized curriculum.
   
   
2. All participants take the portion of the test that they will rate in order to be familiar with the questions in the same order as the students answer them.
   
   
3. Using an ordered item booklet assembled from easiest to hardest, small groups review and discuss "ordered booklets" by Level/Content area.
   
   
4. Round 1: First rating of proficiency cut scores
   
   
   • Proficiency setting using ordered booklets, individuals in level/content groups, each judge moves through the ordered test booklet and places three bookmarks at items where the level of item difficulty for the category is surpassed (based on the judges' standard and the initial descriptions).
     
     
   • One bookmark is placed for each of the required cut points (between Proficient/Advanced, Partially Proficient/Proficient, Minimal/Partially Proficient). Items preceding the mark reflect content that all students at the given performance level should be "expected" to know and be able to do (defined as "with at least a 2/3 likelihood").
   
   
   
5. Round 2: Second rating of proficiency cut scores
   
   
   • Level/content groups systematically review and discuss individual ratings from round 1
     
     
   • Individual make second ratings of proficiency cut scores.
   
   
   
6. Round 3: Third rating of proficiency cut scores
   
   
   • Show impact data of round 2 ratings to large group
     
     
   • Discuss impact data across levels.
     
     
   • Individuals make third ratings of proficiency cut scores.
   
   
   
7. Present results of round 3 with impact data to groups.
   
   • Sessions continue in rounds until work is completed. Panelists may work into the evening.

For questions about this information, contact oeamail@dpi.wi.gov

Last updated on 2/26/2008 10:42:50 AM