What is “Impact” and how do you measure it?

by Anneliese Poetz, KT Manager, NeuroDevNet

For NeuroDevNet, impacts of research and training are achieved when children with neurodevelopmental disorders:
• Are diagnosed sooner
• Receive validated interventions as soon as possible
• And their families are supported through the life span

Related to these, impact is achieved when we make a difference – changes to existing policies or the implementation of new ones, or changes in the way caregivers and/or health practitioners approach their work with children and families. Impact is also helping improve the quality of life for children and families in unexpected ways. In the field of Knowledge Translation (KT), there is still ambiguity about how to measure and report on ‘impacts’ of KT. NeuroDevNet frames its Knowledge & Technology Exchange and Exploitation (KTEE) activities using Phipps’ Co-Produced Pathway to Impact evaluation framework which encompasses the Network’s KT activities as well. If impact is what we are trying to achieve, then KT is one of the means to help us achieve it.

Phipps' Co-Produced Pathway to Impact Evaluation Framework
When thinking about KT in terms of evaluation and reporting on KT activities, several quantitative measures easily come to mind: # of peer-reviewed publications, # of citations of one’s research publications, # of conference presentations, # of KT Products created, etc. However, these measures do not go far enough – notice that these are all indicators in the ‘dissemination’ phase of the CPPI. Typically, this is where KT activities ‘stop’ – it is the point of departure for researchers move onto the next research project. But stopping at the “dissemination” (otherwise referred to as end-of-grant KT) stage doesn’t help you measure the impact of your research.

In order to find out whether your research has been considered useful (in practice, or policy, or otherwise) you have to go and ask the people 1) who you engaged in your research process (integrated Knowledge Translation), and 2) who you imagined would find your research useful once it was completed even if they did not directly participate in informing your research questions or process. Yes, qualitative interviews!

The KT Core conducts qualitative interviews with its researchers, trainees and most importantly its collaborators and partners. There is a lot of good work going on in the Network, and these interviews are for the purpose of discovering stories about how NeuroDevNet’s research and training have made a difference. Some of them might not have otherwise been discovered and/or reported on. The first interview is always with the researcher or trainee. Then, we ask them who their collaborators/partners were, and whether they would be willing to broker an invitation for an interview so we can ask questions about the impact of NeuroDevNet’s work from their perspective. How have we changed things for them in their organization? Their practice? For the children and families they serve?

An example of a story we discovered was through one of our trainees, Angelina Paolozza, in the FASD program of research. Angelina was invited to present at Adopt Ontario after someone from that organization saw her present her research at a local hospital. Angelina was able to adapt her presentation style to be compatible with an audience of prospective parents. After her presentation (the 2 times she has been invited) the audience had the same response – many parents said that now that they understood FASD after hearing her describe her research they would revisit the files they had reviewed on children with FASD. Talk about impact – a child in a stable home has a much improved life trajectory and quality of life. The basic underpinning of any effective KT activity is relationships – and this impact was achieved through the relationship built between NeuroDevNet and Adopt Ontario.

Getting these stories is not just useful for reporting purposes, but it is also valuable for us as a Network to learn what works and what needs more attention/improvement in terms of our collective KT activities. By learning how we can best achieve impact, we can maximize the chances that we can repeat and scale our efforts.

If you are a NeuroDevNet researcher, trainee or collaborator/partner and you have a success story you would like to share, please contact the KT Core and we can help draft it into a formal ‘success story’ to be placed on the NeuroDevNet website as part of a series.

One thought on “What is “Impact” and how do you measure it?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s