3.5. Criterion maintenance

A criterion translates an IT objective into something that can be measured.

For example, you may have the objective that IT systems have high quality tests. A criterion lets you define precisely what you mean by this, and award a score to a system for the quality of its tests.

Criterion maintenance takes place during a number of processes:

During roll out (waterfall) or roll out (iterative), to initially define the criteria to be used by the organisation.
During the annual review to revise criteria to reflect experience and changing priorities.
During the comparison and evaluation process to define special criteria to use for a system selection.
During a compliance audit to define criteria to reflect new requirements outside of the normal annual review cycle.

	Criterion maintenance in Metrici Advisor
	Criterion definition and maintenance can be carried out within the Metrici Advisor tool. Please check with your Metrici representative for an up-to-date view of the functions and capabilities of Metrici Advisor.

Criterion definition

Each criterion has:

Name: A unique name for the criterion.
Description: A description of the criterion.
Significance: The significance of the criterion to the broader business.
Question: The words you would use to find out about the criterion. This is used during assessment to prompt for the textual response.
Answer criteria: What to look for in the answer to make sure that the question has been properly answered.

Each criterion has a set of grades. These characterise different responses. For example, if the criterion is about test quality, one of the grades could be “no formal testing is carried out”. Each grade has:

Name

A name or short sentence that sums up the grade.

Description

A fuller description of when the grade should be awarded.

Score

How “good” the grade is, as a percentage. As a guide, base the score on the following scale:

100%: Perfect response, no improvement possible.
75%: Good response, more than merely acceptable, some room for improvement.
50%: Acceptable, but only just acceptable.
25%: Not acceptable, but shows some characteristics of an acceptable solution.
0%: Entirely unacceptable.

Make sure that the grades cover every eventuality. Most criteria need a “not applicable” grade, which explains conditions when the criterion does not apply. “Not applicable” should have a score of 100%.

Criteria are arranged in groups, loosely classifying them into different areas. Within Metrici Advisor, the groups are: information, use, service, risks, development and technology.

A collection of criterion groups that will be used for assessment is termed a template.

Weighting

Each criterion has a weighting which represents its relative importance.

To assign weightings, give a weighting to each group and then scale these so that the weightings across the groups add up to 100%. Then, group at a time, give a weighting to each criterion in the group and then scale these so that they add up to the group's weighting.

This method of assigning weights to groups and then to criteria makes sure that important groups with few criteria are not overpowered by less important groups with many criteria.

Variants

Some simple criteria do not require a textual response, just a series of grades. (Most criteria require a textual response so that the grades can be checked.)

Conversely, some criteria are only documenting background information and do not have any grades. These are known as information only criteria. Information only criteria always have a weighting of 0.

Some criteria with grades have a weighting of 0. This means that the grades may be important for triggering rules (see below), but that otherwise there are no “good” or “bad” answers to the criterion. For example, a criterion about data sensitivity is important because it alters rules about system security, but there is nothing inherently good or bad about storing sensitive data.

Selecting criteria

To be efficient, focussed and effective, system governance should use a relatively small number of criteria. At the time of writing, the standard system governance template within Metrici Advisor has about 30 criteria with grades, plus about 10 information-only criteria. Aim for about this number of your own criteria.

Define criteria that represent your priority areas. Do not create criteria for everything you can think of.
Define criteria that will help you make decisions.
Define criteria that have a big impact on costs.
Define criteria that reflect major aspects of systems, such as overall usability and technology viability.
Define criteria that apply across all systems, not just some.
Define criteria that are significant. Be suspicious of any criteria with a weighting less than 1%.

Issues and rules

An issue is a point of interest that can be derived from the grading of one criterion, or the combination of gradings from multiple criteria.

Each issue has three parts:

Finding: A description of the issue.
Recommendation: What should be done to address the issue.
Priority: The relative priority of the issue.

Rules are defined which trigger issues using if-then-else logic on the grades. See issue detection for examples of these rules.

Rules are defined for the standard set of criteria within Metrici Advisor. These detect issues covering many of the main IT management recommendations.

Previous: 3.4. Analysis | Next: Chapter 4. System governance processes | Up | Contents