Embedded KT Support within Project Teams – working the “Co-Produced Pathway to Impact” for NeuroDevNet NCE

by Anneliese Poetz, KT Manager, NeuroDevNet

In a recent blog post we told you about the development of a “Hybrid” tool that combines aspects of KT Planning with principles of project management. While this tool is appropriate for use by any researcher or trainee, we primarily developed it for use by NeuroDevNet’s 4 High Impact Projects (HIPs). The HIPs were chosen from existing NeuroDevNet research projects after the Research Management Committee directed that NeuroDevNet focus on 4-5 projects that had the “highest potential for creating impacts on diagnostics, interventions and services” during Cycle II. The idea is that by working closely with a small number of projects we can maximize the chances we will have specific examples of how we have achieved impact during Cycle II, to best position NeuroDevNet for Cycle III renewal. The process took several months, and began with a call to PIs to ask them to select projects within their programs that would fit the criteria to become a HIP. The KT Core made recommendations and the final High Impact Projects were approved by Dan Goldowitz, Scientific Director for NeuroDevNet.

Social ABCs (ASD) High Impact Project Team members meeting with KT Core.

We provided the 4 HIPs with the Hybrid tool to use for drafting their KT plans for the next 5 years (Cycle II for NeuroDevNet). Over the course of this summer, David Phipps (KT Lead, NeuroDevNet) and I met with 3 out of the 4 HIP project teams for 1.5-2 days in-person to review their KT Plan with them and collaboratively refine it as needed.

This is exciting for NeuroDevNet’s KT Core, because it is the first time there have been KT practitioners (KT supports within an organization) embedded within project teams in this way. The information gathered within the Hybrid KT Planning tool will provide the information needed to manage the timelines and milestones for these KT plans during Cycle II. The aim is to work with project teams so that by the time we write our application for Cycle III funding we will have concrete examples of impact.

Screening & Intervention (FASD) High Impact Project team members working on KT Plan with NeuroDevNet’s KT Core

The first meeting we attended was for a project in the ASD program called Social ABCs, the next was for the FASD program’s Screening & Intervention project, and finally the CP Program’s Exergame project. The first two were more focused on KT activities toward achieving uptake and implementation of their respective interventions into programs that serve children and families affected by ASD and/or FASD while the latter is more focused on commercialization of the Exergame technology and games for home use. In all meetings, the project team members (researchers, research support staff, practitioners/partners etc.) were fully engaged and commented afterwards about how useful this process has been for them: both the tool we provided and the in-person meetings.

As part of the process we are listening to project team members for their feedback on the Hybrid KT planning tool in order to inform future iterations. After the in-person meetings the KT Core continues to work with the HIPs to further refine and finalize their KT plan, and also to determine the best ways in which we can integrate with project teams and support their KT goals for Cycle II. We view this as an iterative process, and we will review these KT plans on an annual basis with follow up in-person meetings with project teams.

These HIPs are pilot projects – the KT Core remains available to help all NeuroDevNet projects with KT Planning and other KT services.

If you are a NeuroDevNet researcher or trainee and would like help with your KT plan for a grant application or for your already-funded NeuroDevNet project(s) contact the KT Core to find out how we can help.

What does Program Science have to do with Knowledge Translation?

by Anneliese Poetz, KT Manager, NeuroDevNet

Program science “is the systematic application of scientific knowledge to improve the design, implementation and evaluation of programs”.

The NCE Program in Canada strives to facilitate the achievement of socio-economic impact for Canada through cutting-edge research and innovation.  One social benefit of this Federally-funded research includes implementing innovations into programs that serve Canadians.  NeuroDevNet NCE strives to achieve impact for Canadians affected by neurodevelopmental disorders such as Cerebral Palsy, Fetal Alcohol Spectrum Disorders and Autism Spectrum Disorders.  One avenue is to achieve implementation of evidence-informed innovations into programs.  For example, after refining the exergame prototype, an innovation funded by NeuroDevNet and GRAND NCEs, this technology can expand the physical therapy options for youth with CP.  Integrating the exergame bike into physical therapy programs would be what we refer to as ‘implementation’ (see Figure below).

Phipps’ Co-Produced Pathway to Impact can be used within a Program Science approach toward impact of evidence-based interventions.

NeuroDevNet NCE has adopted Phipps’ Co-Produced Pathway to Impact framework which focuses on stakeholder engagement throughout all stages of the research process, and pushes the boundaries of traditional end-of-grant KT beyond dissemination toward uptake, implementation of new evidence into practice and policy, and evaluation of subsequent impact(s).  Consider that programs are governed by managers and the policies they develop, and the program’s services are delivered by practitioners.  Impact is measured by evaluating both quantitatively and qualitatively, how the program has made a difference for those they serve – in our case, NeuroDevNet is concerned with measuring how research that has been implemented into programs and policies has improved the lives of children and families affected by neurodevelopmental disorders.  So, after the exergame bike has been implemented into a program, we would follow up and evaluate.

Program science has become important for HIV program development because it provides information about what programs work, for which individuals.  The field of program science asks questions that relate to aspects of a program including strategic planning, program implementation (mix of interventions, synergy across interventions) and program management (sustaining effective interventions and modifying programs as new knowledge and interventions emerge, quality improvement processes).

NeuroDevNet supports the Chair in Autism Spectrum Disorders Treatment and Care Research, Jonathan Weiss.  Jonathan specializes in stakeholder consultation with diverse stakeholders including program planners/managers for informing his research, and researching interventions in order to make an evidence-base available for uptake.  This evidence-base can then be used by practitioners and policymakers to inform their decisions with respect to which interventions they choose to provide within programs and organizations (such as schools), and how these interventions are delivered.

NeuroDevNet’s FASD program is embarking on a new project this year to develop and test program materials for frontline workers in Children’s Aid Societies to improve their practice. This project is being done with the full involvement of frontline practitioners and program managers throughout the research process.  When there is an evidence base for these tools and training to show its effectiveness at the study sites, program science can inform the scaling up of the research findings toward improving practice across Canada.

“One aspect of program science is implementation research, which is concerned with the development and implementation of evidence-based interventions…it can also provide information about how interventions can be adapted to new situations or communities…program science typically involves an ongoing process of engagement between researchers, policy makers, program planners, frontline workers and communities through which research is embedded into the design, implementation and continuous improvement of the overall program. Because the focus is on how an entire program impacts a population, program science typically involves consideration of overall health systems” – CATIE

It appears that we (NeuroDevNet, the NCE program, and many Knowledge Translation practitioners) may, in fact, be using a program science approach without knowing it.

How do you think you might already be using a program science approach to your work?

How can the principles of program science that are used for informing HIV programs, be translated into programs for children and families affected by neurodevelpmental disorders?

What can the field of KT learn or adapt from the field of program science? 

Do you think a program science approach can help you (as a researcher, as a KT professional) scale your proven interventions to other health systems, cultures, programs, or geographic regions across Canada and internationally?

Sustainability and Knowledge Translation: sessions at Canadian Knowledge Mobilization Forum 2015

by Anneliese Poetz, KT Manager, NeuroDevNet

This past Thursday May 14, 2015 and Friday May 15, 2015 the 2015 Canadian Knowledge Mobilization Forum took place at the Grand Bibliotheque in Montreal, QC. The Canadian Knowledge Mobilization forum is the national conversation on KT/KMb practice and an excellent way not only to build our own skills but to brand NeuroDevNet as a leading KT organization.  In fact, it was during this event that David Phipps (NeuroDevNet KT Lead) received the “2015 President’s Award for innovation” in “recognition of his extraordinary contribution to the field and practice of Knowledge Mobilization in Canada and internationally”.

Peter Levesque presents Knowledge Translation award to David Phipps at CKF 15
Photo credit: Pedro Ruiz

The overall theme of the conference was “Creativity as Practice: Mobilizing Diverse Ways of Thinking”. I both learned from other presenters, and shared my own knowledge.

In the workshop “Narratives, video and smartphones as KT tools for youth” (by Sean Muir) I learned that the ‘formula’ for maximizing effectiveness of KT with youth is: grab their attention with a shocking image or story, present your content/message, and then end with something positive. Sean used examples of videos and posters to illustrate this point. In the workshop on “Mobilizing your message through documentary video: research findings as cinematic narrative” (Callista Haggis et al.) the takeaways for creating KT videos were “done is better than perfect”, “show don’t tell” and “think about what you want your target audience to think, feel, do”. In this case, the documentary was both to present research findings in an alternative format, as well as to inspire discussion about the issues presented in the video toward possible infrastructure changes to accommodate the needs of an aging population.

NeuroDevNet’s KT Core Lead, David Phipps, participated in leading 2 sessions. One session was with Purnima Sundar (Ontario Centre of Excellence for Child and Youth Mental Health) and Renee Leduc (NCE Secretariat).

The audience gained insight into the 3 common reasons why research grant applications fail: 1) lack of meaningful end user engagement, 2) unclear pathway to impact, and 3) poor evaluation of KM (Knowledge Mobilization) and of impact. The NCE Secretariat provided tips on how to prepare a successful research funding application, and held an interactive session asking for the audience’s ideas for what the NCE Secretariat could do to help applicants be more successful. Ideas included: successful applicants’ mentoring of new applicants, creation of how-to videos to accompany written grant application instructions, and provision of examples.

David moderated the session on “the paths of sustainability for KMb” in which I was one of the 4 presenters. I presented on the KT Core’s evaluation framework, indicators, and 3 factors relating to sustainability: relevance (how does what we’re doing fit with our priorities), leadership (who is responsible for ensuring outcomes are met), and financial (can cost-effective strategies be used).   The presentations were 10 minutes each. When the presentations were over, each presenter took their discussion question to a corner of the room and invited attendees to join their group (depending on which question most interested them) and discuss it further in terms of their own context.

The questions were:

– How are people attempting to influence sustainability across diverse settings with the use of tools?
– How can we sustain KT implementation through strategic planning?
– How can team capacity and culture be shaped over time to best meet the needs of knowledge users?

And my question was:

– What factors should be considered with respect to sustainability?

I had about 12 people in my breakout discussion group. Although I had a discussion question prepared, I received several questions about what NeuroDevNet’s KT Core does in terms of evaluation and also about database design and development. After the breakout discussions we returned to the large group and each presenter did a ‘report back’ about what their group discussed.

“Anneliese provided a great overview of the process she developed to measure the relevance and impact of knowledge translation products. Her experience was very relevant as our organization is currently exploring different methods of evaluating our work. We look forward to learning more about Anneliese’s indicators and database.”
– Sheena Gereghty, Canadian Centre on Substance Abuse

If you are a NeuroDevNet researcher or trainee and would like help with KT videos, advice on event evaluations and/or evaluation of your other KT activities and products, contact the KT Core to find out how we can help.

Research partners, research users and research impact

By: David Phipps, KT Lead, NeuroDevNet

“If you want your research to have an influence on early childhood literacy practice you’d better not be partnering with the fire department”

David Phipps leads discussion during workshop for research administrators in the UK

On April 15 I led a workshop for the UK Association of Research Managers and Administrators. This workshop was for research administrators (university staff managing research applications among other things) who were implementing the Research Excellence Framework. The REF 2014 was a research assessment exercise that assessed both research excellence and the impacts of research. For REF impact was defined as:

“an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”

– (see page 26, REF Assessment Framework and Guidance on Submissions)

The REF officers and other research administrators interested in research impact gathered for a one day ARMA workshop to look beyond REF 2014. This included looking towards REF 2020 as well as beyond the narrowly construed REF frameworks including university research expertise (faculty and graduate students) that is engaged beyond the academy.

I used Melanie Barwick’s KT Planning Guide (click the link and enter your e-mail address to get access to the tool) as a tool to help the UK impact officers look beyond REF reporting on past impacts and start to create the conditions to enable future impacts. This planning guide asks researchers to consider 13 elements of a KT framework. Working through those 13 elements provides the raw material to then craft the KT strategy.

Melanie Barwick’s KT Planning Tool

The KT planning guide (elements 1-3) asks the researcher to consider the types and roles of partners in the research. Partners are the individuals/organizations who are along for the ride. They are co-producers of research. They help disseminate research results. They co-supervise students. They provide cash and in-kind (space, data, populations, equipment) resources to the research project.

The KT planning guide also asks the researcher to consider types of research users (element 5). These are individuals/organizations that take up the research evidence and use that evidence to inform decisions about public policy, professional practice and social services. The NCE Secretariat calls them “receptors” or “knowledge users (KUs)”. Both partners and receptors/users are critically important to the research to impact process. The co-produced pathway to impact outlines the pathway from research to impact on the lives of children with neurodevelopmental disorders and their families. Partners collaborate throughout but receptors only become involved after dissemination.

Phipps’ Co-Produced Pathway to Impact, the evaluation framework adopted by NeuroDevNet NCE

Research partners will likely be research users but research users are not always research partners.

In the ARMA impact workshop one Impact Officer was convinced that research partners and research users were the same. After I explained the difference she remained unconvinced. That’s when I said, “If you want your research to have an influence on early childhood literacy practice you’d better not be partnering with the fire department”. Research users need to be coherent with research partners because one informs and/or has access to the other.

For NeuroDevNet’s social ABC’s intervention led by Dr. Jessica Brian from Holland Bloorview as part of the Autism Discovery Program, the research partner is Humber College which has two full-time community-based childcare settings. Humber College’s practitioners-in-training will help develop and evaluate the intervention. The knowledge users will be early childhood centres and day care centres across Canada who will put the research evidence into practice by using it to support early childhood learning. The KT Core will work with Dr. Brian and her partners help identify these receptors/KUs and broker collaborations so that Social ABC will be implemented and evaluated beyond the research project setting.

If you want the KT Core to help you find partners and receptors/users to help translate your research into early diagnosis, validated interventions and supports throughout the life span please contact the KT Core.

What is “Impact” and how do you measure it?

by Anneliese Poetz, KT Manager, NeuroDevNet

For NeuroDevNet, impacts of research and training are achieved when children with neurodevelopmental disorders:
• Are diagnosed sooner
• Receive validated interventions as soon as possible
• And their families are supported through the life span

Related to these, impact is achieved when we make a difference – changes to existing policies or the implementation of new ones, or changes in the way caregivers and/or health practitioners approach their work with children and families. Impact is also helping improve the quality of life for children and families in unexpected ways. In the field of Knowledge Translation (KT), there is still ambiguity about how to measure and report on ‘impacts’ of KT. NeuroDevNet frames its Knowledge & Technology Exchange and Exploitation (KTEE) activities using Phipps’ Co-Produced Pathway to Impact evaluation framework which encompasses the Network’s KT activities as well. If impact is what we are trying to achieve, then KT is one of the means to help us achieve it.

When thinking about KT in terms of evaluation and reporting on KT activities, several quantitative measures easily come to mind: # of peer-reviewed publications, # of citations of one’s research publications, # of conference presentations, # of KT Products created, etc. However, these measures do not go far enough – notice that these are all indicators in the ‘dissemination’ phase of the CPPI. Typically, this is where KT activities ‘stop’ – it is the point of departure for researchers move onto the next research project. But stopping at the “dissemination” (otherwise referred to as end-of-grant KT) stage doesn’t help you measure the impact of your research.

In order to find out whether your research has been considered useful (in practice, or policy, or otherwise) you have to go and ask the people 1) who you engaged in your research process (integrated Knowledge Translation), and 2) who you imagined would find your research useful once it was completed even if they did not directly participate in informing your research questions or process. Yes, qualitative interviews!

The KT Core conducts qualitative interviews with its researchers, trainees and most importantly its collaborators and partners. There is a lot of good work going on in the Network, and these interviews are for the purpose of discovering stories about how NeuroDevNet’s research and training have made a difference. Some of them might not have otherwise been discovered and/or reported on. The first interview is always with the researcher or trainee. Then, we ask them who their collaborators/partners were, and whether they would be willing to broker an invitation for an interview so we can ask questions about the impact of NeuroDevNet’s work from their perspective. How have we changed things for them in their organization? Their practice? For the children and families they serve?

An example of a story we discovered was through one of our trainees, Angelina Paolozza, in the FASD program of research. Angelina was invited to present at Adopt Ontario after someone from that organization saw her present her research at a local hospital. Angelina was able to adapt her presentation style to be compatible with an audience of prospective parents. After her presentation (the 2 times she has been invited) the audience had the same response – many parents said that now that they understood FASD after hearing her describe her research they would revisit the files they had reviewed on children with FASD. Talk about impact – a child in a stable home has a much improved life trajectory and quality of life. The basic underpinning of any effective KT activity is relationships – and this impact was achieved through the relationship built between NeuroDevNet and Adopt Ontario.

Getting these stories is not just useful for reporting purposes, but it is also valuable for us as a Network to learn what works and what needs more attention/improvement in terms of our collective KT activities. By learning how we can best achieve impact, we can maximize the chances that we can repeat and scale our efforts.

If you are a NeuroDevNet researcher, trainee or collaborator/partner and you have a success story you would like to share, please contact the KT Core and we can help draft it into a formal ‘success story’ to be placed on the NeuroDevNet website as part of a series.

Critical Considerations for doing Stakeholder Engagement for Research

by Anneliese Poetz, KT Manager, NeuroDevNet

When I began my PhD studies in 2003 it was before terms like “Integrated Knowledge Translation” (IKT) and “Co-Produced Research” were common. In my graduate program people would joke about how their predecessors had placed a $10 bill in their dissertations only to return to the library years later to discover it was still there – nobody had read it. Wow, all that time, money and effort wasted on something that would never be used – I didn’t find it funny at all. I really wanted my research to be used, so after I crafted my research questions I went and talked to the people who I imagined would use my research: CEOs, government regulatory agencies, industry. I also attended their conferences and listened for what were the issues that were important to them. As a result of this new understanding about my end-user’s needs, I changed 80% of my research questions because I realized I was asking the wrong ones. The result was my research was found interesting and useful by those end-users that I had consulted. I completed my research in 2010, and emailed the first of 3 papers I published to the President and CEO of the Canadian Nuclear Safety Commission (Dr. Michael Binder). I received this email response on January 22, 2011 “Very interesting study. I have the taken the liberty of circulating to staff at CNSC. Thanks for sharing”. The results of my research which recommended ongoing stakeholder engagement, resulted in Bruce Power’s creation of several social media channels such as twitter and a community blog in 2011, and a facebook page in 2014 (I checked with an employee at Bruce Power who confirmed that these avenues for stakeholder engagement represented uptake of my research results).

It is important to involve a diverse range of stakeholders in any consultation or stakeholder engagement activity

Now, over a decade later, this kind of involvement of end-users including policy- and decision-makers has become a requirement of funding agencies such as CIHR. This is an important overall paradigm shift: from research that is done in isolation with no consultation or thought about who will actually use the research and whether what you are doing will be relevant to them, to a more responsive approach that is intended to meet the needs of decision-makers including practitioners. In the field of Knowledge Translation we call this “stakeholder engagement” (a stakeholder is anyone who will be directly or indirectly affected by decisions made based on your research findings).

There are several researchers who are doing stakeholder engagement. For the past several years NeuroDevNet’s Neuroethics Core has been conducting workshops and other in-person events to engage their diverse stakeholders (such as health care providers, researchers, patient advocacy groups) in their research, so that they can be responsive to their needs.  NeuroDevNet’s FASD project has engaged stakeholders in the development of their “Strongest Families” research project. Customization of the course material was achieved through phone interviews with clinicians (specializing in FASD) and families and the program is based on the needs they articulated.
And that is the key to stakeholder engagement – being responsive to the needs of your stakeholders including end-users. Stakeholder engagement can include regular in-person meetings, dialogue through social media, or other forms of communication such as teleconferences or online meetings.

If you are not prepared to be responsive to stakeholder input, then don’t engage your stakeholders. Be honest with yourself – are you prepared to change your research questions, methodological

If you don’t plan to listen and respond (by adapting your research questions and/or methodology) according to what your stakeholders are telling you, don’t ask for their input

approach, etc. based on input from your stakeholders? If you are not prepared to be flexible and responsive throughout the research process, then don’t ask for input. You will do more harm than good – it will cause irreparable damage to the relationship. People will feel used, and it will create distrust, conflict (which includes avoidance and refusal to use your research in decision-making), and ultimately the usefulness of your research findings will not be maximized.

Be responsive to the system in which your research occurs
The field of KT is now recognizing the importance of the entire system. If stakeholder engagement is considered “Integrated Knowledge Translation”, and IKT occurs within a system, then you as a researcher also need to be responsive to changes to the various levels of the ‘system’ in which you are doing your research. Changes to the system include: new research discoveries, new diseases, new technologies, news in another part of the world, new legislation and regulations…you get the idea.

Keys to effective stakeholder engagement summarized:
1) Make people feel valued by sincerely listening to their input, keeping them informed about the progress of the research, and closing the loop at the end of the project.

2) Use different forms of stakeholder engagement – in-person meetings are best but you can use social media to expand your networks beyond what is possible in-person, as well as to keep in touch with people in between meetings.

3) Involve diverse stakeholders especially end-users in your research design, and be prepared to be responsive to their input even if the research is already in-progress.

The main message is: successful knowledge translation occurs as a result of trust (in the source of the information being used to base policy and practice decisions on) which is a result of good relationships and ongoing relationship building. This requires several levels of flexibility and openness as a researcher. If you sincerely want your research to be useful, good stakeholder engagement is the way.

The KT Core is building networks via social media (LinkedIn, facebook, twitter) and will be leading a cross-jurisdictional cross-Network in-person stakeholder consultation in year 6. If you are a NeuroDevNet researcher, trainee or partner and need help with in-person or online stakeholder engagement for your individual project(s), contact the KT Core for tools and advice.

NeuroDevNet presents at #CKF14 conference in Saskatoon, SK June 9-10, 2014 – Indicators, evaluation framework, risk perceptions in KT

by Anneliese Poetz, KT Manager, NeuroDevNet

NeuroDevNet’s KT Core recently attended the 3rd annual Canadian Knowledge Mobilization Conference in Saskatoon, Saskatchewan on June 9-10, 2014. There were almost 100 Knowledge Translation/Knowledge Mobilization specialists in attendance, the theme was ‘innovation’.

David Phipps (Lead, KT Core) presented a PechaKucha (20 slides, 20 seconds each) on the Network’s Co-Produced Pathway to Impact (CPPI) framework for evaluating the impact of KTEE, and Anneliese Poetz (Manager, KT Core) presented on indicators that have been mapped onto the CPPI.

NeuroDevNet is in the process of completing a suite of indicator definitions for measuring the impact of their KTEE activities.  Defining indicators for data collection is important for 1) reporting impact to funders, 2) for comparison across KT activities/services 3) in future, comparison across NCEs.  This data will help in terms of decision-making (about how to refine services offered by the KT Core).  It is also useful for gathering ‘institutional memory’ for succession planning.  Indicator definitions need to be more comprehensive than just a title such as “number of…” or “percentage of…”.   The KT Core plans to explore the use of MS Access instead of Excel for ease of data entry.  Anneliese’s presentation can be viewed on slideshare:

Indicators forMeasurement at each stage of Knowledge Translation: from research to impact_Poetz_Phipps from NeuroDevNet

An example of a complete definition was handed out to those in attendance at the presentation and can be viewed on slideshare:

HandoutToGoWithIndicatorPresentation_anatomy ofindicator_infographic_poetz_phipps_neurodevnet from NeuroDevNet

 

Anneliese also presented on her previous research about how organizations make decisions using different types of information.  The model that emerged for knowledge translation, introduced the concept of risk perception as an important consideration in any KT strategy.

Relationship-based knowledge mobilization: systems-based KMb and consideration of risk perception for effective uptake_Poetz from NeuroDevNet

 

In addition to the CPPI evaluation framework and activities related to measuring impact of the KT Core’s services and the Network’s activities, the KT Core provides consultation services to NeuroDevNet researchers and trainees.

If you would like to know how to evaluate the impact of your KT activities within NeuroDevNet, are planning to host a KT Event and would like assistance developing an evaluation form, or have any questions related to evaluation of your KT Products, contact Anneliese Poetz, KT Core Manager.