3.2. Assessment

Assessment involves investigating one system and responding to all the criteria for that system.

Assessment takes place as part of a number of different processes.

During the initiation of systems governance, some initial assessments are carried out to familiarise the organisation with system governance.
During the roll out of systems governance, existing systems are assessed for the first time. Assessment of each system involves responding to every criterion. See Section 4.4, Roll out (waterfall) and Section 4.5, Roll out (iterative).
Once system governance is established, new and changed systems are assessed as part of the systems development process. See Section 4.8, Project review.
Once a year, systems that have not been changed are reassessed. The reassessment involves checking and updating existing assessments, and responding to new criteria. See Section 4.9, System review.
During a system selection, assessment is used to compare and evaluate competing options. This is similar to a first time selection, but uses a modified set of criteria to include any specific requirements of the selection. See Section 4.10, Comparison and evaluation.
Occasionally it may necessary to assess all the systems for a new requirement, as part of a compliance audit. The focus of the assessment is the new and changed criteria that represent the requirement, not a system-by-system assessment.

The assessment activity is similar across all these processes.

The need for information

Assessment requires knowledge of the system. Assessment is much easier if systems have effective owners, who can find all the required information easily. If the system is not well documented, first time assessment will be difficult, but then the information captured as part of the assessment will be of extra value to the organisation as a consolidated source of documentation.

Approaches to assessment

Metrici Advisor lets you carry out assessments in a number of ways:

An assessor can use Metrici Advisor to interview a system owner to respond to all the questions, and fill in the responses on-line. If necessary, the questions and gradings can be printed out, and entered into Metrici Advisor after the interview. This approach provides good assessments, quickly, but requires an effective owner who has the information to hand.
Metrici Advisor's permissions structure allows the assessment of a system to be delegated directly to the system owner, and for them to fill in responses themselves. This self-assessment approach takes some of the pressure off a central assessment team, but can lead to more work in the validation stage.
The interview and self-assessment approaches can be combined. The owner initially responds themselves, which gives them the opportunity to investigate areas that are not currently known. An assessor then reviews the information with the owner, and helps complete any outstanding questions.
Assessment of specialist subjects, such as networks and databases, can be delegated to an appropriate expert. Metrici Advisor's permission structure could, for example, allow a database administrator to respond to database criteria across all systems, but not permit them to respond to any other criteria. This can be useful where knowledge is split by subject area more than by system.

In practice, a mixture of approaches will be required. Some systems will be easy to assess, and others will require significant investigation, at least for first-time assessments.

Responding to criteria

Most criteria have two parts: a carefully-worded question which requires a textual response, and a carefully-worded grading scheme. The textual response is important because it makes the assessment readable and justifies the grading. Read the question and the grading scheme very carefully, and respond fully and concisely. One-line answers and a quick tick in the box will not provide the quality of information needed to drive improvements in IT.

Check the response and the grading with the system owner, or other authority. Ensure that they agree that the response is factually correct, and ensure that gradings are fully justified by the textual response.

In some cases, such as system selection, it might not be appropriate to discuss the grading at the same time as gathering responses. In this case, ensure that the system owner or other authority (such as the vendor) agrees with the textual response, and then grade the response separately.

Escalation procedure

It should always be possible to write a textual response that the system owner agrees with. Occasionally, however, system owners or other authorities might question the grading. Usually these disagreements can be resolved by reading the wording of the gradings carefully.

Sometimes, however, it is not possible to agree a grading. This might be because the criterion has not recognised the situation that applies to the system, or because the system owner wishes, despite the evidence, to claim a higher or lower grade. If agreement can not be reached, do not grade the response. Write a comment against the response that it has not been graded, and refer the grading to the system governance committee in the next interim review. If the committee can not agree, refer the issue to the sponsor. Do not let a disagreement on one criterion undermine the general progress of the assessments.

Remember that system governance is focused on providing the organisation as a whole with a tool for improving IT. Occasionally this will mean establishing gradings that are not popular with system owners, either because they imply the systems could be improved, or because they imply that the systems do not need additional investment. The system governance sponsor has final responsibility for navigating any issues that this may cause.

Coverage

It is not usually possible to complete assessments for all systems. When planning assessments, set a target level of coverage, such as 95% coverage on 80% of systems, and 75% coverage on the remaining 20% of systems. This ensures that the overall assessment processes can complete even if some information is outstanding on some systems.

Metrici Advisor can report on the coverage, and analyses allow for missing data. Metrici Advisor provides a comment field on every response, which allows you to document issues with finding out the information.

Previous: 3.1. Metrici Advisor | Next: 3.3. Validation | Up | Contents