Questions

Once you understand the purpose, evaluands, respondents, outputs and processes, you can consider the questions for the assessment. It's important to get the other things sorted before you try putting together the questions, or you will end up with questions that will not meet your needs.

You need to think of:

  • Which questions to include.
  • What patterns the questions follow.
  • How the questions are structured on questionnaires.
  • Written style for the questions.
  • Supporting content

Selecting questions

The assessment should contain the minimum number of questions that you need to meet your objectives. You need questions that:

  • Provide the information you need to generate outputs.
  • Gather background information to support the outputs.
  • Give the respondent somewhere to "get things off their chest".

Most assessments make the mistake of having too many questions. Be ruthless and remove any questions that you can.

As a guide, a short question on a simple survey will take 15 seconds to complete. A question that asks for a textual answer will take at least 2 minutes to complete, and could take 5 or more minutes if it is complicated. Calculate how long it will take the respondents to complete the assessment, and ask whether the respondents will consider that reasonable. Think how many assessments are being completed in total, and consider whether the organisation will commit the total time. Every question that you add will add to the time that the respondents need to give to the assessment, and is likely to reduce the completion rate and the time they spend per question. You will get a much better result if you have a short questionnaire that contains only the most important questions than if you have a long questionnaire that contains everything you can think of.

Question patterns

Metrici can support a great variety of question patterns. It can support different types of text and number input and selection fields, lists, document upload, and so forth. However, most assessments boil down to a small number of question patterns.

The most basic pattern is to associate a question with a set of distinct answers or grades. These can be:

  • The same for each question, such as "strongly disagree", "disagree", "agree" and "strongly agree".
  • Follow a similar progression of badness to goodness (or goodness to badness) for every question, but be worded differently. For example, every question might have four grades which mean "very bad", "bad", "ok" and "good", but written to reflect the question.
  • Offer completely different sorts of grades for each question, with no specific progression of badness to goodness.

Questions can be associated with a comment area, to allow people to explain or justify their answers. Although they are not used in the scoring, reading through comments can be a great way of finding out what's really going on, especially in survey sent to multiple people.

The other pattern used widely in assessments is to ask an open question, and then ask the respondent to grade their answer, as in the following example:

Question with text and grades

This style of question is useful in deeper assessments, as it encourges the user to write more. It also allows the grades to be checked for consistency against the text.

Questions with different sorts of grades, and the longer question pattern above, make people think more. This can be a good thing if you want an expert opinion but not so good in a quick survey.

Within an assessment, stick to as few patterns of question as possible, as it makes the assessment easier to complete. It also makes it easier to explain the connection between the assessment and the outputs.

Whichever style of question you use, you need to think of:

  • A title for each question.
  • The question itself.
  • A set of grades. These can have just short names, or long descriptions as in the example above.

Question headings and question numbers are both optional. The default is to have question headings but not question numbers, but this can be varied:

  • Use question numbers for long assessments where the user may need to jump around the assessment or return to the assessment.
  • Omit question headings for simple surveys where only the question needs to be shown.

Structure

Simple assessments will have only one questionnaire. For more complex assessments, where different things are assessed by different people, use your analysis of evaluands and respondents to break down the overall assessment into multiple questionnaires.

If you have more than ten questions on your questionnaire, consider splitting the questionnaire down further so that it is presented to the respondent in a more digestible way, and, if appropriate, to guide the respondent through different aspects. To split a questionnaire down, you can:

  • Split the questionnaire into two or more pages, presented to the respondent on different tabs.
  • Within a page, split the questionnaire into sections that the user can open and close. This is good for detailed assessments with a lot of text, but should not be used for simple surveys because it hides the questions from the user.
  • Split the questionnaire so that it is two or more questionnaires.

Written style

The table below summarises the most important guidelines for writing questions.

General

Write in a single language, e.g. British English, International English, or another language. Spell and punctuate correctly. Be concise.

Review, edit and trial the questions.

Target audience

Remember that your audience do not know as much about the subject as you.

For an assessment targeted at a business, write everything for a general business audience. Think of a relatively junior person in a business department, and write for them.

For an assessment targeted at the public, think of a well-educated thirteen year old, and write for them.

Do not assume any technical knowledge. Be very careful not to use jargon.

Question wording

Either:

  • Write the question as a question, ending in a question mark.
  • Write the question as an instruction asking the respondent to provide information or select the most appropriate grade.

The table below illustrates appropriate and inappropiate question wording.

How does your organisation ensure IT is aligned with business strategy?

OK

Organisation alignment with business strategy.

Not a question or an instruction.

Which of the following best describes the ISMS in your organisation?

Requires further explanation. What is an ISMS?

The Information Security Management System (ISMS) is a set of policies and procedures for managing information security.

Which of the following best describes the ISMS in your organisation?

OK

The Information Security Management System (ISMS) is a set of policies and procedures for managing information security.

Pick the item below that best describes the ISMS in your organisation.

 OK
Grade wording

Decide whether you want to use just short names for your grades, or grade names and longer descriptions.

Grade names should be short phrases that sum up the situation.

Grade descriptions, if used, should be longer explanations that provide further detail. The screen shot above shows the use of both grade names (in bold) and grade descriptions.

Abbreviations

Define abbreviations on their first use. For example:

The Information Security Management System (ISMS) ...

Not

The ISMS (Information Security Management System) ...

Do not define abbreviations on grades if they have been explained on the question.

If two or more questions that follow each other use the same abbreviation, omit explaining the abbreviation on the following questions if it is clearer to omit it.

Always re-define abbreviations on findings and recommendations.

Use abbreviations in the names of questions and grades.

Addressing the respondent

Always direct the questions as if you were asking the respondent personally, using "you" and "your" as appropriate:

How responsive has the help desk been to your requests?

If you want to refer to the organisation, business, company, etc, use the company name or, if the assessment is intended for multiple organisations, use "the organisation":

Does the organisation have effective business recovery plans?

Negative questions

Never write questions in the negative, as this leads to double negative grades that are hard to interpret and lead to inaccurate gradings.

To illustrate, consider the following questions and grades. It is much more likely that respondents will interpret the second one correctly.

Are projects run without business approval?

  • Always

  • Usually

  • Sometimes

  • Never

Are projects run with business approval?

  • Never

  • Sometimes

  • Usually

  • Always

Grades in question

Avoid using grade-like word in the question. For example, the following is unanswerable:

Are projects always run with business approval?

  • Never

  • Sometimes

  • Usually

  • Always

Judgements

Avoid adding unnecessary judgements to questions and grades, because this will skew people away from giving factual answers.

To illustrate, consider the following. It will be easier to use without the words written in bold.

How responsive is the IT service desk to your requests?

  • The IT service desk are excellent and always responds to my requests quickly and effectively.

  • The IT service desk usually responds well to my requests, but sometimes there are worrying issues.

  • The IT service is rubbish and does not respond effectively to my requests.

Supporting content

As well as the questions themselves, questionnaires can contain or link to other content. This can include blocks of text, pop-up help, links to guidance such as glossaries, images and videos. These can be arranged as part of the questions, between questions, as part of the output, or as stand-alone pages that the respondents are directed to.

Consider what additional content the assessment would benefit from.