How to Score Process and Results Items

Introduction:

While the Criteria are used to initially guide development of Examiner comments, the scoring guidelines are used to assess the overall maturity of the response. Individual scoring factors, listed below, serve as the link between comments and the score and can be used in addition to further refine and clarify comments.

Use of the scoring factors:

The following four process [scoring] factors are addressed in each Process scoring range. As the scoring ranges build in point value, each scoring guideline `bullet' relating to that factor describes a greater level of maturity.

Process: The scoring factors of approach, deployment, learning, and integration are used to assess the maturity of processes. See the A-D-L-I: Process Evaluation Factors Resource sheet for further information and pertinent questions to ask when evaluating an applicant response. Use the `Process' portion of the scoring guidelines for scoring Categories 1-6.

The following four results factors are addressed in each Results scoring range. As the scoring ranges build in point value, each of the factors describe a greater level of maturity.

Results: The four [results scoring] factors used to evaluate results are performance levels, rate and breadth of improvement (trends), performance relative to appropriate comparisons and/or benchmarks (comparisons), and linkage to important customer, product and service, market, process, and action plan performance requirements identified in the Organizational Profile and in Process Items. See the L-T-C-Li: Results Evaluation Factors Resource sheet for further information on these [results evaluation] factors. Use the `Results' portion of the scoring guidelines for scoring Category 7.

Basic, overall, and multiple requirements:

The first bullet in the process scoring guidelines indicates a level of maturity related to a scoring factor and the extent to which the applicant has responded to the Criteria Items. The terms describing Criteria requirements are:

n       Basic Item requirements = Item Heading (e.g., Senior Leadership). These represent the central concept or fundamental theme of the Item.

n       Example: If the applicant describes a basic approach for the Item, but does not respond to each of the Areas to Address, the applicant is responding only to the basic Item requirements.

n       Overall Item requirements = Bold black text at top of the Item. These requirements are the most significant_, features of the Item.

n       Example: The applicant provides a general approach for each `Area to Address' designated by lettered Item subtitles but does not respond to any of the questions listed after the numbers (e.g., 1.1a[3]).

n       Multiple Item requirements = Individual questions listed in each Area to Address. Responses to multiple questions within a single Area to Address may be grouped. The multiple questions serve as a guide in understanding the full meaning of the information being requested, so should not be used as a checklist when evaluating an applicant response.

Steps in the scoring process:

                Use the following steps to score an Item

Additional scoring concepts:

Balance of comments: In general, higher scores should be reflected by a greater number of strengths or ++'s, lower scores reflected by a greater number of OFIs or - - 's. In addition, some comments are more important than others because they discuss aspects of the applicant^'s performance management system that have a significant impact on the applicant's score. Such comments often support Key Themes.

Most descriptive of the organization's achievement level: This is based on a holistic view of either the four process factors or four results factors in aggregate and not on a tallying or averaging of independent assessments against each of the four factors. Assigning the actual score within the chosen range requires evaluating whether the item response is closer to the statements in the next higher or next lower scoring range. Not all `bullets' in a range must be correct for the range to be most descriptive.

Importance: This is related to the importance of the processes and results that are related to key business factors. The areas of greatest importance should be identified in the Organizational Profile and in Items such as Item 2.1, Item 2.2, Item 3.1, Item 5.1, and Item 6.1. Key customer requirements, competitive environment, key strategic objectives, and action plans are particularly important.