3.4. Analysis

During assessment, every system is graded against every criterion. Each grade has a score. The outputs to assessment, which are the inputs to analysis, can be thought of as a table in which the rows are systems, columns are criteria, and individual cells are scores.

The table below shows a simplified example of what assessment delivers.

Give missing assessments a score of 0.

Each criterion has a weighting. The total weighting is 100%.

To calculate a score for each system, multiply the scores for each criterion by the criterion weight, and then sum these weighted scores for each system.

You can use the system scores to identify the “good” systems and the “bad” systems. Perhaps more importantly, you can use the system scores to shows which systems are improving, and which are declining, and to set and check targets.

Criteria are arranged in groups which represent different areas of IT management, such as support, technology and usage. You can repeat this analysis one group at a time, to find out in which areas systems are strong or weak, improving or declining.

Not all systems are of equal size and complexity. A small problem on a big system can be more serious than a big problem on a small system. You need to take account of this when calculating an overall score across all your systems.

Estimate a relative size for each of your systems. You can use any scale you like, as long as it is consistent. Use the relative sizes to calculate a weighted average of all the system scores, to give an overall score across the entire system portfolio.

To calculate a weighted average, multiply the size of each system by its score, and then add up this figure for all systems. Divide this total by the sum of the sizes to give the weighted average.

The overall score across all systems is a measure of the IT organisation's overall performance at looking after systems. It is a good measure to use for systems that are outsourced.

Improvement points are a relative scale that measures how valuable improvements are to the organisation. You can calculate improvement points for an individual criterion on an individual system, for one criterion across all systems, for all the criteria in one system, or for all criteria across all systems.

To find improvement points, calculate the difference between the assessment score and 100%, and multiply this by criterion weighting and system size.

For example, the improvement points for service availability on the HR system is (100 – 70) × 25% × 2 = 15

If you arrange the individual improvement points in a table as before, then the improvement points for individual systems are the row totals, and the improvement points for individual criteria are the column totals. You can use these improvement points to assess the priority for improvements in different systems and across different criteria.

The improvement point total (482 in our example) is important because it can be used as the basis for calculating a notional value for improvements.

The improvement points measure the relative value to the organisation of improving the qualities and capabilities of systems. This can be converted to a notional value by multiplying it by a monetary figure, which we will term the improvement point value (IPV).

You can calculate an IPV based on a complete assessment of your systems. To do this you need to put a value on fully meeting all criteria, and then divide this by the improvement point total.

One way of estimating the value of fully meeting all the criteria is to estimate the annual cost saving that this would bring, and then multiplying this by a payback period. For example, you might estimate that meeting all the criteria would save £560,000 per year (in reduced user and support costs, reduced downtime, and reduced project costs). Your organisation might require a 2.5 year payback period on investments, in which case it is worth spending £560,000 × 2.5 = £1.4m to achieve this saving.

Another, perhaps light hearted, way of estimating the value of fully meeting all the criteria is to imagine a miracle that transformed all your systems so that they met all the criteria perfectly. How much would you pay for that miracle?

In our example above, you might decide that meeting all the criteria is worth £1.4m. The IPV is therefore £1.4m ÷ 482 = £2,906.

You do not have to use the IPV that you calculate. Agree a figure a little below the calculated IPV, to allow for inaccuracies in weighting, scoring and sizing. Having calculated an IPV of £2,906, you might decide to use £2,000.

The table below shows a notional value for improvements in each area, by multiplying the improvement points by 2,000.

These figures estimate a notional value for an improvement in the systems' fit to criteria assuming that the systems then meet criteria fully. Of course, the changes might not aim to meet criteria fully, and the value of the changes will be less.

The accuracy of the notional values depends on the criterion weightings, scoring scheme and system size. If you set these parameters to reflect their financial value to your organisation, then the calculated notional values will be a reasonable input to your planning. You should, of course, treat this figure as only indicative, and not rely on it solely in your work planning and justification.

Some notes on IPVs:

As part of criterion definition, rules are defined which specify combinations of grades that trigger issues. Issue detection involves running these rules. This issue detection is normally carried out within Metrici Advisor. Some simplified illustrations of rule processing are given below.

Some rules are triggered by individual grades.

IF system can not be recovered after a hardware failure
THEN
    risk of permanent system loss

Each issue (such as “risk of permanent system loss”) is associated with a recommendation which outlines what needs to be done to overcome the issue.

Many rules are more complicated and involve a combination of grades.

IF system holds sensitive data
  AND system security is weak
THEN
    risk of compromising sensitive data

Issues have a priority. The priority scheme used by Metrici Advisor is:

Some issues have a fixed priority. For example, the “risk of compromising sensitive data” issue always has a high priority, irrespective of the system in which it occurs. Many issues are more context dependent and for these the priority can be adjusted up and down.

IF system is business critical
  OR system is very large
THEN
    increase priority

IF system is not business critical
  AND system is very small
THEN
    decrease priority

Issues are calculated one system at a time. Issues across multiple systems can be collated.

Issues can be independent of the scoring scheme. Criteria with a weighting of 0 can still trigger issues. This lets you identify things that you must do even if there is no specific payback, such as a legal requirement.

In most cases, the issues do reflect the scoring scheme. The recommendation suggests what needs to be done to improve the score. These recommendations can be used with the notional values to determine what needs to be done, and to put a value on it.