Anyone in a leadership position will inevitably come across some metric or, going further, a performance indicator. If there is an operational level of measurement for a given metric, the performance indicator delivers a strategic value to the leader, with a perspective oriented towards a specific end. It is the metrics that will lead to the performance indicators. Then, they will, in a structured way, provide the scenario for achieving the established goals. Explaining it this way, there is a logical and important chain for decision-making and strategy changes. The danger lies when different ways of measuring data begin to emerge.
Recently, Jeff Sauro raised a very relevant question for leaders: “how to assess the quality of a measure ?” According to the author, every year it seems like a new form of measurement or questionnaire is introduced. Imagine that you are in the office and every day an infomercial appears announcing the measurement of the year or an innovative chile whatsapp data research methodology, the future of NPS. This is more or less what the expert was feeling. He is constantly responsible for reviewing new questionnaires that are published in journals and conference proceedings. In general, questionnaires are applied in a survey and simply cited as a form of measurement. But how do you know which ones are really suitable? Which ones can be considered “good”?
If new measurements are being introduced every day, you need to know which ones will have a positive impact on your assessments. After all, as we said above, metrics are what will build performance indicators and it is with them that the leader will know how far or close they are to achieving their goal. It is common to hear arguments for and against different types of metrics, such as, for example, the parameters used in digital marketing. Only then do we include a series of actions and goals to be measured and evaluated, such as social networks, inbound marketing and the consequences of each of them.
You may have heard a similar discussion, but I have often heard people talk about how vanity metrics are harmful. And, given this, we should ignore them. But is that really true? Even though they do not help in strategic decision-making, it may be valuable to have them there, being observed, it may be something interesting. It is the famous case of it all depends. However, the issue is not defending whether they are good or bad, whether they should be ignored or not. It is about not generating doubt, as I just did previously. And, for that to happen, the argument for or against must be based on data, not guesswork.
How to know the quality of the questionnaire or the way of measuring
Weak ratios are not a good basis for measurements or questionnaires. There is much to evaluate about quality. Therefore, I brought some of the considerations made by Jeff Sauro to apply in a new questionnaire or measurement.
Is it reliable? In a questionnaire, the questions and answer options must be consistent and clear, for example. To measure reliability, it is possible to measure two different points at the same time, evaluate different versions or make correlation between items.
Is it valid? You need to describe content validity and be able to predict results.
Are you using existing questionnaires? There is no need to invent something that already exists and works well. No need to reinvent the wheel. You can check whether it is based on previous research.
Can you spot differences? Assess whether you can spot differences, even small ones.
Are there reference scores? Knowing whether 5 is good or what 77 represents, for example. Reference scores are used to find out whether 5 is great, ok or bad.
Has it been tested? It is interesting to check whether the measure has been used in more than one data set and in different contexts.
Is it too long? The more items, the more reliable. But outside of an official setting, such as schools or governments, it is common to want shorter questionnaires.