aes17

David Roberts (RobertsBrown)

A major issue for evaluators is the validity of the answers people give when questioned on topics of interest to evaluators. Cognitive and psychological research show that people generally do NOT search all their memory or use rational processes to answer the questions we ask. Indeed, what people 'Say' in interviews often bears little relation to actual behaviour in the past or in future. Instead, implicit cognitive processes throw up a large number of possible answers, most of which never reach conscious awareness.  One such answer may emerge in our awareness as ‘the’ answer.  Sometimes, two or three answers emerge to awareness as 'probable' answers for consideration.   Generally, we then use a 'best fit' heuristic to choose between the 'probable' answers generated by our implicit processes.

While the cognitive research is relatively clear, there is very little research into how we might apply the lessons to our own evaluation practice.  The research does suggest that the closer the interview context is to the context of action the greater the validity of the responses.  So techniques that are able to recreate the context of action are more likely to generate valid answers.

This interactive session explores the lessons from research and how those lessons might be applied to improve our research techniques. Participants will be asked to participate in a question-answering session and then to explore the cognitive bases of their responses.  We will then discuss the cognitive research to see how it applies to the experience of the group.  If time allows, the final part of the session will involve participants working together to develop ideas about how to apply the research to developing questions