What is it?
Testing the intervention (or part of it) with target audiences before it is implemented. It might involve testing new services, products or programmes (or elements of them), communication messages, materials and brand names. This is likely to involve the final customer, but may also involve other stakeholders such as those responsible for service delivery. You may also want to test your data collection and evaluation methods to ensure there are no difficulties once the intervention is launched
Why do this?
There are two main reasons for pre-testing:
To refine the approach. More >
The ideas designed in the scoping stage are now further developed and require testing with target audiences so that they can be refined and made more specific to their needs.
To change the approach. More >
Costly mistakes can be avoided by pre-testing. Errors could have been made in interpreting research data and/or interpreting the motivations and needs of the target audience. Additionally, other factors may change over time, including environmental trends. Small scale testing can highlight problems before you commit substantial resources.
How might you do this?
There are a number of decisions for the steering group to make at this stage, including:
What should be pre-tested?. More >
It is important to assess which elements will require pre-testing. Typically these will be new to the team and/or the target audience, so there is little knowledge or experience on which to base decisions or assess impact. The greater the potential risk involved, the greater the need for a pre-test. Members of the steering group may vary in their opinions as to what needs to be pre-tested. It is important to record why decisions to pre-test (or not) were made as this is likely to be a factor to consider at the evaluation stage.
The nature of the pre-test. More >
There are three main approaches to consider, although they do overlap to a certain extent. These are:
- Test delivery
- Communications testing
- Research approaches
Test delivery may be required and involves limited delivery of the intervention. For example, new clinic services could be delivered at one clinic rather than across the whole PCT; new exercise regimes could be tested at one school; or new approaches to recycling in one geographical area. More >
The following are the main steps:
- Select a representative area/location/group of people to involve in the test. You may also want to consider a control group – evaluating the representative area with an area where you have not carried out the test.
- Establish preliminary objectives for the test market - to assess the level of take-up of a new service; identify barriers and incentives to take-up; assess the quality of service and impact/usefulness of communication materials.
- Involve service employees and other relevant stakeholders in a meeting (or a series of meetings) to discuss the issues and gain consensus about the purpose of the pre-test. It is particularly important to agree on what information to collect and what action will be taken on the basis of the test results. Often, negative test results are ignored because by this time there may be a considerable amount of ‘buy-in’ as well as time and other resources devoted to the proposed campaign.
- Select a representative area/location/group of people to involve in the test.
- Prepare materials, facilities and necessary training to allow the intervention to be delivered on a limited scale
Decide on a time frame. This will vary according to your objectives and what is to be measured. If, for example, you were to assess take-up of exercise classes by obese school children, the pilot would have to be long enough to assess the time frame for drop out.
- Decide how the pre-test is to be evaluated, for example, through observation, records and research approaches such as interviewing the target audience. Also decide who is to evaluate the pre-test. It might be better to involve an independent person who can take an objective view
Implement the pre-test.
- Collect and analyse the information.
- Involve stakeholders in a further meeting to decide whether the proposed programme will go ahead as originally planned; will require some modification in line with the pre-test results; or will require major changes or be dropped.
When testing proposals, there are a number of methods available. More >
Pre-testing method: Proposition concept testing
When to use it: The objective of this testing is to obtain consumer reactions to the proposition, if you cannot afford to run the intervention to test the ideas or propositions with your target audience.
Pre-testing method: Ad concept testing
When to use it: Ad positioning statement testing The objective of this testing is to obtain consumer reactions to positioning statements such as ‘Condom. Essential Wear’, which was used by DH in 2008 to promote condom use, or ‘It's not for girls’, which HYPERLINK "http://en.wikipedia.org/wiki/Nestl%C3%A9" o "Nestlé" Nestlé used to promote its Yorkie bar in the same year. In the process of designing major advertising campaigns, it is important that statements are received positively by the target audience and initiate the desired behaviour.
This usually follows positioning statement testing. At this point, preliminary versions of the printed materials have been developed and you may be testing concepts for print work, radio or television advertisements. A variety of methods can be used, but face-to-face data collection procedures (usually focus groups) are the norm.
Pre-testing method: Readability testing
When to use it: Readability is an attempt to match the reading level of written material to the ‘reading with understanding’ level of the reader. Readability tests are designed to give a statistical analysis of the difficulty of a text. While any attempt to reduce language (which is inherently creative), to statistics can be criticised, readability tests can be used to give an approximate indication of the ease with which it can be understood.
Pre-testing method: Expert and field review
When to use it: This will enable you to obtain ‘buy-in’ from key stakeholders. Expert and field reviews can be done either by sending experts/ key stakeholders information on the intervention and a self-administered questionnaire, or by setting up a focus group or individual interviews. While this method is mostly employed for pre-testing promotional materials, it can also be used for the pre-testing of new products and services. (Reference: McDaniel and Gates, 1999; Sigel and Doner, 2004).
Many research approaches can be involved in pre-testing, for example individual interviews and observation.More >
Focus groups are one of many useful method to assess new ideas, products and services. They are also very good forums for discussion of communication messages and media. A focus group usually consists of 8 – 10 people (smaller groups are also possible) who meet together with a moderator. The benefits and problems of focus groups are outlined below.
Benefits:
- As there are a number of people involved, more views and opinions can be gathered in a shorter time than through individual interviews.
- Participants are likely to trigger ideas and opinions in others. A good dynamic in a group can create an exciting discussion which can raise valuable insights.
- Individuals may feel safer and more secure in expressing their opinions when in the company of others.
- Quality control can be enhanced by including additional observers, recording and/or videoing.
- Flexibility can be built into the process so that novel ideas and opinions can be explored.
Problems:
- This is a qualitative technique involving a few people and cannot therefore be assumed to be representative of the views and opinions of the whole target audience.
- A skilled moderator is essential as unskilled moderators can bias outcomes.
- This is a difficult process to manage. People may not show up for the meeting. When they do they often try to talk at the same time. It is therefore often difficult to analyse and interpret focus group data.
- Findings are open to subjective interpretation. It is important to have a structured approach to data analysis.
Tips
Take care when extending the results of a test market (or any research method) to the target audience as a whole. More >
We need to ask how representative are the people involved in the test/research, what was the sample size, and are there any other factors which question the relevance of the test to the whole audience.
Pre-testing can take place in the field and laboratory, so be aware of the pros and cons associated with each. More >
Field approaches involve the target audience in a real life situation whereas laboratory tests involve a simulated approach. Laboratory tests are likely to be cheaper. In addition, they may help to avoid some of the ethical problems highlighted in the next point.
Remember that the public and other stakeholders may perceive a pre-test as unfair and unethical, for example if services are being developed in some areas and not others.
Be aware that focus groups are often maligned because their findings are used in an inappropriate way. More >
This method can provide valuable, qualitative insights into the target audience’s perceptions, attitudes, motivations and behaviour. Due to the small numbers involved they cannot be representative and the views can therefore not be expressed as those of the target audience as a whole.
Output
An initial pre-testing and piloting plan.
Reports (internal or commissioned) for field test or other research based evaluations.
Intended Outcome
A better understanding of the reactions of the target audience to the proposed programme/campaign (or elements of it).
A decision as to whether to modify, drop or go ahead to full launch.