Temporal Stability of Psychometric Instruments
One important measure of reliability for psychometric instruments is that of temporal stability, often known as 'test-retest'. For an instrument to be useful, it is important for it to have a reasonable level of temporal stability that can be related to the defining measures of the constructs.
The Team Management Profile Questionnaire
As an example, consider the Margerison-McCann Team Management Profile (TMP) which determines a person's team role preference and reports it as a location on the Team Management Wheel. The instrument measures people's responses across four work preference scales as follows:
If a respondent scored strongly as an extrovert on a particular day and then, say, two weeks later scored strongly as an introvert we may begin to question whether this instrument is measuring anything useful. That is why all instrument designers should publish their test-retest data so that users know the level of stability they can rely on.
'Test-retest' involves administering the same test to people over a time span. Intervals can vary from a short-term interval of, say, two to four weeks to longer-term ones of several months or years.
Test-retest reliability can tap sources of error external to the actual instrument. Issues such as mood swings, health, motivational factors and recording errors are just some of the aspects that can cause response variations over time. The length of time between testings is crucial here as well. Short periods will be subject to bias from 'remembering' effects and longer periods will be subject to the fact that people will change if the time duration is long enough.
It is possible that a change in scores does represent a legitimate change in the true score rather than random error. For example in the Team Management Profile Questionnaire constructs there is ample evidence that many people can move from a more introverted relationship preference at work to a more extroverted one, as they develop confidence in their ability to communicate.
Test-Retest of the Team Management Profile Questionnaire
An example of a test-retest study on the Team Management Profile is given in the Team Management Systems Research Manual. A sample of 100 people were asked to complete the profile questionnaire over a time period of between one and six years. The breakdown of time intervals between administrations was as follows:
- 1 year: 4 people
- 2 years: 39 people
- 3 years: 20 people
- 4 years: 23 people
- 5 years: 13 people
- 6 years: 1 person
The data were analyzed and reported in two ways. Table 1 shows the correlation matrix between administration at time #1 and time #2.
Table 1. Correlation table of Administration 1 and Administration 2: Individual scales
As can be seen all the correlations are above the acceptable level of 0.70- 0.75
Table 2 shows the stability of role preferences over one to six years reported as the percentage of respondents who changed their score across the poles of the four work preference measures. For example someone who scores E: 6 at time 1 and I: 5 at time two would be judged as having changed their preference from extroversion to introversion.
Table 2. Stability of preferences over one to six years
As can be seen, 64% of respondents reported no change in their overall team role preference and a further 27% changed their scores across one of the work preference measures only. In other words 91% of respondents had no change or only one change in the poles of the four work preference measures.
From this, it is possible to conclude that the Team Management Profile gives excellent stability on an individual's work preferences, certainly more than enough for it to be used reliably in personal and team development interventions.
Table 1 also gives useful information on which of the work preferences are more likely to change over time. The data suggests that for the sample surveyed, respondents recorded a greater change on their Structured-Flexible (S-F) scores than any of the other three. From interviews with many participants in our worldwide programs it seems that this change is likely to be encouraged by many organizations. We have many examples of people starting off their careers with a 'flexible' score on the organization work preference and then changing to be more structured as management development programs encourage them to work in a way that is planned, time sensitive and converges on desired outputs.
The QO2™
The QO2 ™ - the Opportunities-Obstacles Quotient: this profile questionnaire is based on a fifth scale which measures different aspects from those defined by the Team Management Profile Questionnaire. The fifth scale is probably the most difficult to use in a work context, although it is widely used in the realm of clinical psychology where it helps distinguishes between normal and abnormal behavior. The scale is often used to identify such characteristics as depression, anxiety and neuroticism.
Five subscales form the basis of the QO2™ methodology and comprise the Risk-Orientation Model. All scales have undergone a rigorous research program and have high face validity and utility for managers and their teams. The five subscales are:
- MTG Energy - how much energy you put into 'Moving Towards your Goals'.
- Multi-Pathways - the extent to which you find ways around obstacles
- Fault-Finding - how good you are at seeing potential obstacles
- Optimism - the extent to which you expect positive outcomes
- Time-Focus - a measure of your psychological time and your orientation to the past, the present or the future
Test-Retest of the QO2™
The fifth dimension is an interesting scale as it contains measures (optimism for example) which are known to be influenced by mood swings. The four scales of the Team Management Profile are, by comparison, much more stable. In fact one of the learning concepts used in the QO2™ workshops is that of MIPs - mood induction procedures - which can be used to shift a person's position up or down the QO2™ scale.
The interesting question is how much variability might there be on an instrument of this type? For the QO2™ to be of use we need to be confident that it has sufficient stability in personal and team development applications, for which it was designed.
A sample of 165 people was analyzed with a time interval of re-administration ranging from one to over twelve months. The age range was 19-60 years with 46% men and 54% women.
The test-retest correlation coefficients were 0.80 for the Opportunities items and 0.74 for the Obstacles items. While the Obstacles scale correlation coefficient is only just below the value of 0.75 we usually look forward, it is perfectly adequate for a scale known to have some variation with time.
The results showed a quarter of the sample with a QO2™ score variation of less than 9%. Half the sample showed a variation of less than 17% and three-quarters had a variation of less than 30%.
Copyright © Dr Dick McCann, Team Management Systems. All rights reserved.