Participatory techniques in measurement from #eval17

by Stephanie Chasteen on November 10, 2017

Another big theme at the conference is how to engage your stakeholders in evaluation. These techniques are also relevant to those in educational reform and institutional change, as these would be great ways to include departments, faculty, students, etc., in data to inform change.

Why engage?  To increase use of your evaluation

Why doesn’t data get used in making decisions?  Often, because the person doing the measurement (e.g., the evaluator) isn’t the same person who needs to make the decisions.  Stakeholders need to be given an opportunity to engage in the evaluation planning, making meaning of the data, and mapping out their systems.  People are more likely to use what they helped create.  Obvious, yes, but I had thought of participatory structures as the solution to helping people learn from the evaluation – but ultimately this learning is really in the service of increasing the use of the evaluation.

Data parties & data galleries

Data parties are a common way to engage stakeholders in messing around with the data – BEFORE you write the report.  I will definitely do this next time – I wrote my report, but had trouble writing specific actionable recommendations based on the data because I was guessing at the best solution.  Testing out solutions with your stakeholders in a data party is a great way to do this.  For some resources on data parties, see Community Solutions website (http://communitysolutions.ca/web/resources-public/) and the great “intentional learning” guide at FSG (https://www.fsg.org/tools-and-resources) — both of which talk about using data placemats, rotating flip charts, data galleries, and other ways to engage people in the data.  My challenge is that many of my activities take place virtually.  One idea that I got at the conference:  I could create Google Docs with the data that I want people to review, put them into breakout groups, and then have the breakout groups rotate through several different Google docs and make comments.  This would be a virtual data gallery.

System mapping

System mapping is making a map of the system at hand.  Who are the actors?  What is the environment?  What are the barriers and enablers to change?  What are the feedback loops?  To the right is an example from ServiceDesignTools.

Why do I list this as a participatory technique?  Because this is something that is most useful to create with the clients or people undertaking change.  The final product is not as important as the process.  This was highlighted by an experienced evaluator (Kyle Hutchinson) who shared the system map she worked hard on for a client, which was utterly confusing to the client.  Anyone who hadn’t created it would have found it hard to use.  I could imagine using this technique with physics departments, professional organizations, many types of people who are trying to work with a system.

A variant of system mapping is a technique pioneered by a great design guy here at the conference (Cameron from Cense) – called attractor mapping.  In attractor mapping, you make a map of the system, but note or overlay information about where most of the action is happening.  This could be geographic, social networks, system models, etc.  Where is energy and action being focused?  Is it in the right place?

Rubrics

Making a rubric can also be a participatory activity.  What would success look like for an institution?  What is unsatisfactory, satisfactory, and exemplary performance for… an executive board?  A site visit?  A rubric can give a useful format for engaging in a collaborative format about what success looks like, being clear about criteria, and working backwards from there.  For more see The Social Profit Handbook which discusses the use of “success rubrics.”

Journey Mapping

I got to do a fun journey mapping activity with Cameron from Cense.  Journey Mapping is a design-thinking activity, to map out the path of a particular person or type of person through your project.  For example, we chose to map the hypothetical path of a person who decides to attend Evaluation 2017 (this conference) to them choosing to take action as a result of what they learn from the conference.  This highlighted several key evaluable questions, such as what types of people are attracted to the conference, how does the word get out, what kinds of communication sets expectations about the conference, how do they choose relevant sessions, how are they supported to take action.  I could imagine using this with a lot of my stakeholders:

  • What is the pathway of a person who engages in the PhysTEC project, taking on new identities around teacher preparation?
  • What is the pathway of a physics major from learning about teaching to licensure at a particular site?
  • How does a department chair choose to enact change in their department and use APS materials to enact that change?

Other ideas

For more great design thinking ideas, see the Design Kit, and Cameron’s blog Censemaking.

Leave a Comment

Previous post: