Bookmark Standards Setting

Overview of Standard Setting

States typically set cut scores on their assessments that characterize a student's performance on the assessment as at or above one of 3-4 performance levels (for example, basic, proficient and advanced). The method used to set these cut scores is called Standard Setting. There are several acceptable ways to set standards. Each depends on having participants in the standard setting who are very knowledgeable about the state's content standards and willing to help the state define the level of knowledge and skill expected of a student at each performance level articulated by the cut scores. An overview of standard setting and in particular CTB's Bookmark Standard Setting method is provided below.

Standard Setting

Standard Setting is a method of determining cut scores that correspond to performance levels. The standard setting process is usually implemented with a committee of educators since educators understand the state's content standards and the performance of students at each grade level and in each subject area. The cut scores that are determined during the Standard Setting procedure demarcate one performance level from another as is demonstrated in Figure 1. Figure 1 shows a state's performance levels separated by cut scores. During the Standard Setting, participants will set these cut scores by engaging in a structured conversation that includes discussion of content standards, performance levels, the test, and expectations for students.

Figure 1. State Performance Levels Separated by Cut Scores

For educators, this is a variation of a routine task. Teachers, for example, regularly separate students into performance levels, such as A-Student, B-Student, C-Student, D-Student and F-Student. The teacher assigns cut scores to each performance level; typical cut scores are 90% for A-Student, 80% for B-Student, 70% for C-Student, and 60% for D-Student. While these types of cut scores may be appropriate for classroom tests, their arbitrary and test-specific nature is not useful for a large-scale assessment. For a large-scale assessment, such as a state's subject area assessments, cut scores must take into consideration the difficulty of the test, as well as the knowledge, skills, and abilities expected of students to answer each test question.

Bookmark Standard Setting Procedure

To set valid, meaningful cut scores, a Standard Setting must be held. The Bookmark Standard Setting Procedure is a research-based method, recommended by CTB that can be used by committees of educators to establish cut scores on a state's assessments.

The Bookmark Standard Setting Procedure is a groundbreaking process developed by CTB/McGraw-Hill. Since its inception in 1996, over 28 states have implemented Bookmark to set cutscores on their large-scale assessments. The Bookmark Procedure typically includes training, 3 rounds of activities and discussion, and description writing.

Bookmark Materials: The Ordered Item Booklet (OIB) and its associated item map are key to the Bookmark Procedure. The OIB is constructed using items from the test. The items are ordered in terms of difficulty where the easiest item appears first and the hardest item appears last; this ordering is determined by student performance on the test. The Item Map gives detailed information about each item in the OIB and gives standard setting participants a place to record their thoughts about the items.

At the Bookmark Standard Setting Workshop

Over a three-day period, Bookmark standard setting participants engage in training, three rounds of structured discussions and ratings where they set cut scores on an assessment, and a writing session where they write descriptions of the recommended performance levels.

Round 1 of Discussion and Bookmark Placement: In small groups - typically tables of six to eight people - participants examine each item in the OIB, discussing what each item measures and what makes the item harder than those before it. After this discussion, each participant determines a cut score by placing a bookmark in the OIB according to his or her own judgement of what, for example, Proficient students should know and be able to do.

Rounds 2 and 3 of Discussion and Bookmark Placement: Participants then engage in two more rounds of bookmark placements. In Round 2, participants discuss the rationale behind their original bookmark placement with other participants at their table. In Round 3, participants at all tables discuss their bookmark placements together. After each round of discussion, participants may adjust or maintain their bookmark placements. Impact data, that is the percentage of students in that state that would fall below each Bookmark, is introduced to participants during the third round. After the final round of bookmark placement, CTB Research calculates the group's recommended cut score by taking the median of all bookmark placements in the final round.

Description Writing: To complete the standard setting, participants write performance-level descriptors that reflect the final recommended cut scores. In small groups, participants examine the items before the bookmark and synthesize the content measured by those items. After two rounds of revisions, the performance-level descriptors represent a summary of the knowledge, skills, and abilities students must be able to demonstrate to enter each performance level.

Second Method Validation: For some states, a second method of standard-setting can be introduced at the end of the Bookmark standard-setting workshop in order to validate the recommended cut scores. CTB recommends using the contrasted groups method where teachers who have set the cut scores using the Bookmark methods then assign each of the students in their class rosters to a given performance level. CTB can check the actual performance of that student against the teacher's judgment. With cut scores being used for important decisions (e.g., WKCE is one factor in the promotion decision for students in Wisconsin), a second method gives the state more information upon which to set the final cut scores.

Final Decision on Cut Scores: The final determination of a state's cut scores is made by the individual or group with education policy-making authority within the state. The State Superintendent and/or the State Board of Education considers the recommendations of the standard setting committee and, oftentimes, review of those recommendations by a Technical Advisory Committee when establishing the final performance levels for a state's assessments. The more information the policymakers have when making this decision, the more defensible the final decision will be if the cut scores are challenged by a school, student or parent.

STATES THAT HAVE USED THE BOOKMARK STANDARD-SETTING APPROACH

STATE	ADDITIONAL INFORMATION
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut	New CTB State - Bookmark being considered
Delaware	1999
District of Columbia
Florida
Illinois	2nd Grade Title I Test
Indiana
Kentucky	Bookmark, Contrasted Groups, Jaeger-Mills
Louisiana
Maryland	2003
Michigan
Minnesota	Re-set HSGT standards
Mississippi	September 2002
Missouri
Nevada	November 2002
New Mexico
New York	Grades 4 & 8
North Dakota
Oklahoma	1999
Pennsylvania
South Carolina
Tennessee
Vermont	1998
Washington
Wisconsin
West Virginia	November 2002

Compiled by Dr. Karla Egan, Research Scientist, CTB/McGraw-Hill
September, 2002

For questions about this information, contact oeamail@dpi.wi.gov

Last updated on 2/26/2008 10:42:51 AM