Participatory techniques in measurement from #eval17

by Stephanie Chasteen on November 10, 2017

Another big theme at the conference is how to engage your stakeholders in evaluation. These techniques are also relevant to those in educational reform and institutional change, as these would be great ways to include departments, faculty, students, etc., in data to inform change.

Why engage?  To increase use of your evaluation

Why doesn’t data get used in making decisions?  Often, because the person doing the measurement (e.g., the evaluator) isn’t the same person who needs to make the decisions.  Stakeholders need to be given an opportunity to engage in the evaluation planning, making meaning of the data, and mapping out their systems.  People are more likely to use what they helped create.  Obvious, yes, but I had thought of participatory structures as the solution to helping people learn from the evaluation – but ultimately this learning is really in the service of increasing the use of the evaluation.

Data parties & data galleries

Data parties are a common way to engage stakeholders in messing around with the data – BEFORE you write the report.  I will definitely do this next time – I wrote my report, but had trouble writing specific actionable recommendations based on the data because I was guessing at the best solution.  Testing out solutions with your stakeholders in a data party is a great way to do this.  For some resources on data parties, see Community Solutions website ( and the great “intentional learning” guide at FSG ( — both of which talk about using data placemats, rotating flip charts, data galleries, and other ways to engage people in the data.  My challenge is that many of my activities take place virtually.  One idea that I got at the conference:  I could create Google Docs with the data that I want people to review, put them into breakout groups, and then have the breakout groups rotate through several different Google docs and make comments.  This would be a virtual data gallery.

System mapping

System mapping is making a map of the system at hand.  Who are the actors?  What is the environment?  What are the barriers and enablers to change?  What are the feedback loops?  To the right is an example from ServiceDesignTools.

Why do I list this as a participatory technique?  Because this is something that is most useful to create with the clients or people undertaking change.  The final product is not as important as the process.  This was highlighted by an experienced evaluator (Kyle Hutchinson) who shared the system map she worked hard on for a client, which was utterly confusing to the client.  Anyone who hadn’t created it would have found it hard to use.  I could imagine using this technique with physics departments, professional organizations, many types of people who are trying to work with a system.

A variant of system mapping is a technique pioneered by a great design guy here at the conference (Cameron from Cense) – called attractor mapping.  In attractor mapping, you make a map of the system, but note or overlay information about where most of the action is happening.  This could be geographic, social networks, system models, etc.  Where is energy and action being focused?  Is it in the right place?


Making a rubric can also be a participatory activity.  What would success look like for an institution?  What is unsatisfactory, satisfactory, and exemplary performance for… an executive board?  A site visit?  A rubric can give a useful format for engaging in a collaborative format about what success looks like, being clear about criteria, and working backwards from there.  For more see The Social Profit Handbook which discusses the use of “success rubrics.”

Journey Mapping

I got to do a fun journey mapping activity with Cameron from Cense.  Journey Mapping is a design-thinking activity, to map out the path of a particular person or type of person through your project.  For example, we chose to map the hypothetical path of a person who decides to attend Evaluation 2017 (this conference) to them choosing to take action as a result of what they learn from the conference.  This highlighted several key evaluable questions, such as what types of people are attracted to the conference, how does the word get out, what kinds of communication sets expectations about the conference, how do they choose relevant sessions, how are they supported to take action.  I could imagine using this with a lot of my stakeholders:

  • What is the pathway of a person who engages in the PhysTEC project, taking on new identities around teacher preparation?
  • What is the pathway of a physics major from learning about teaching to licensure at a particular site?
  • How does a department chair choose to enact change in their department and use APS materials to enact that change?

Other ideas

For more great design thinking ideas, see the Design Kit, and Cameron’s blog Censemaking.


Data viz resources from #eval17 (update)

by Stephanie Chasteen on November 10, 2017

I’m enjoying my first time at the American Evaluation Association (AEA) conference here in DC, and finally getting around to writing about a few things that I’m learning. Today’s post is about some of the great data visualization and representations that I’ve been picking up. This is all really relevant to my education research friends.

Choosing the right chart

The evaluation field’s go-to guru for well-displayed data is Stephanie Evergreen – check out her two books Effective Data Visualization: The Right Chart for the Right Data (on choosing the right chart) and also Presenting Data Effectively: Communicating Your Findings for Maximum Impact (on displaying the data effectively).

To the right is are notes from Stephanie Evergreen’s session here (notes thanks to Kat Haugh; learn more about visual note-taking on her blog pots here).

I enjoyed a webinar by Stephanie earlier this year and would recommend it to anybody. For example, she talked about how good the eye is at comparing lengths. But the standard bar chart confuses the eye, because the bars have width and chunkiness. So she suggests lollipop charts (see example in visual notes below), with a stick and ball at the end – to highlight the important part of the bar: The end point. The eye is more easily able to compare lengths with this visual cue. See her books for more ideas of using cognitive science to choose and design graphs. However, she has acknowledged that the culture of many organizations (including academia) has resisted some of the changes in design she recommends. Too bad. Change is slow.


I also took a great workshop on making infographics by Stephanie Wilkerson and Anne Crosby of Magnolia Consulting.  (Yes, lots of Stephanies around!) Infographics can be great to accompany a large evaluation report, to help facilitate meetings, and to help people remember the key points a bit better. Infographic sheets are also more portable than the big report – you can grab it and bring it with you to the meetings.

These are surprisingly easy to make – using tools like Canva and Powerpoint and a little knowledge of design, you can put together some nice one-pagers. You can get great icons (only some for free) from the NounProject to illustrate the graphics. For example, they put together a nice little one-pager in front of us with headers (the evaluation questions), some big numbers (e.g., # of schools served), and a few key bar charts (e.g., rating of workshops) and quotes.

They have some great resource sheets at, though my only beef is that I would like to see some good examples of one-page infographics there.  Here is a similar style one that I found on Pinterest (screenshot at right).

Update:  One-Pagers

Since I originally wrote this post, I attended a really nice session about one-pagers.  The handouts are at, and have lots of great tips and suggestions and AMAZING other resources on their website (EvaluATE).  One of the best suggestions was to use a grid to lay out your one-pager, to create a “rule of thirds” and make sure it’s intentionally laid out.  And look for the “pearls” in your data …. don’t report all the oysters.  Hunt through your oysters for the few pearls.

Data placemats


A similar idea to an infographic, but with more early-stage data, is a data placemat. This is really a participatory evaluation technique rather than a visualization tool, but I put it here as another display option. When you’re working with stakeholders to try tothink about the implications of data, use a data placemat (print a copy for everyone at the table) and facilitate a conversation about What? So what? Now what? (What is the data? What does it tell us? What should we do about it?). The image at left is from a great session by by FSG which has great resources on their site about systems thinking, leading working groups, and facilitating learning.  See my separate post about participatory techniques in data analysis.


Another session that gave me something to think about is timelines (Anna Newton-Levinson, Megan Higdon). Using Excel (the top part of the image below) and Powerpoint (below) you can create some rich timelines to visualize data that is taking on a recurring basis, and where the context (e.g., seasonal changes or policy changes) matter. You might include funding levels, program activities, etc., and use color groups for different types of activities.


Some helpful tips in project management

November 1, 2017

I’m a member of the American Evaluation Association (, which is honestly one of the most productive professional society memberships I’ve ever encountered.  They offer many webinars and amazing resources, plus a daily blog, which are exactly what I’ve needed as an evaluator.   I recently blogged about the wonderful (paid) webinar that I took with […]

Read the full article →

Fidelity of Implementation: Measuring how instructional innovations are used

October 25, 2017

I recently came across an illuminating article by the (ever-diverse) Marilyne Stains and her colleague Trisha Vickrey discussing a particularly sticky issue in education research – how do we know if research-tested techniques and curricula are as effective in practice as promised by the original study? Of course, we don’t – If Professor A at […]

Read the full article →

My friend Paul

September 27, 2017

I have both a sad and joyous post today — one that I have been meaning to write for some time, but understandably struggled to do so.  On August 18th, I lost one of my dearest friends and most loving mentor, Paul Doherty. I have thought of Paul every day since finding out  he was […]

Read the full article →

Condensing the visual display of comparisons: Data Dashboards

September 13, 2017

I’ve been learning more about effective data visualization lately, and recently was in a wonderful webinar on Data Dashboards (with Ann Emery — whose blog has great posts about data viz, such as using color, and telling stories with data).  It was a wonderfully information-packed session, and I’d recommend it to anybody!  I have a […]

Read the full article →

Want to consult? Here are some resources for education consultants.

August 9, 2017

I’m pleased to announce the launch of our new Physics Consultants Directory on   Here you can list yourself as a consultant, or find consultants to help with a variety of projects.  We are trying to populate the directory intensively by August 17th, so please try to list yourself by then (though the site […]

Read the full article →

My learning goal and clicker workshops all online

August 3, 2017

Giving workshops on the use of clickers / peer instruction, or learning goals?  I wanted to let you all know that my workshop materials for both topics are all compiled and archived online on our SEI Workshop Page.  There are also videos of several of my workshops (though a few years old, they still show […]

Read the full article →

Defining excellence in physics teacher preparation programs: The PTEPA (#AAPTSm17)

July 26, 2017

A big challenge in physics is preparing adequate numbers of well-prepared future physics teachers.  There is a huge dearth of qualified physics teachers at the high school level, and some physics departments have taken it upon themselves to try to address this gap.  Some are very successful.  How do they do it? I’ve been working […]

Read the full article →

Phys21: Preparing students for diverse careers (#AAPTSM17)

July 26, 2017

I just gave an invited talk at AAPT about my work for the Phys21 Report:  Preparing Students for 21st Century Careers.  I was commissioned by the JTUPP committee to create case studies of how institutions achieved success for diverse students.  This was my favorite project last year, it was completely inspiring to talk about what […]

Read the full article →

Improving the bottom quartile with a metacognitive exercise (#AAPTSM17)

July 25, 2017

I’m in an inspiring session by Charles Atwood (University of Utah) about how they improved the performance of at-risk students in introductory chemistry at the University of Utah. Abstract: To improve success rates in large general chemistry sections at the University of Utah, we realized we must improve the bottom two student quartiles performance. We […]

Read the full article →

Mutual mentoring (liveblogging from #AAPTsm17)

July 25, 2017

I’m now attending a session on Mutual Mentoring for physics faculty, presented by Anne Cox (Eckerd College). Abstract: We were part of an NSF ADVANCE grant mutual mentoring project for senior women faculty in chemistry and physics that began in 2007. We have continued our bi-monthly mentoring meetings for the past 10 years (well beyond the […]

Read the full article →

Promoting innovation and entrepreneurship in physics (liveblogging from #AAPTSm17)

July 24, 2017

I’m at the American Association of Physics Teachers meeting this week, and will blog about a few sessions while I’m here. In a talk by Crystal Bailey (American Physical Society), she argued that we need to more explicitly teach Physics Innovation and Entrepreneurship (PIE) to our students.  I find this a really valuable message; having […]

Read the full article →

Educational change: How systemic thinking helps to push social progress

July 5, 2017

In today’s post I want to share some ponderous thoughts about how educational reforms happen, and how systemic thinking helps to support those reforms.  I am fortunate to be a working group leader in the Accelerating Systemic Change Network (ASCN;, and one of the working groups focuses on how theories and models of change can […]

Read the full article →

Changing how universities teach science: The SEI Model

June 21, 2017

We know a lot about how to improve STEM teaching and learning at the college level, and yet these improvements have yet to take hold in a widespread manner.  This is the perennial problem which many of us in STEM education are wrestling with.  The study of institutional change is expanding ever more, including lessons […]

Read the full article →

Data visualization tips

June 14, 2017

Are you trying to tell a story with your data?  This is a big part of my job (as an external evaluator), and I recently attended an excellent webinar on data visualization.  Now, I hate webinars that are trying to sell me a book, but this one was so packed full of great ideas that […]

Read the full article →

You can now embed PhET into Powerpoint!

June 2, 2017

If you’re a PhET user, you’ll be interested in this one.  PhET has a new application that allows you to directly embed the simulations into your Powerpoint.  No more switching back and forth between Powerpoint and the simulation, or awkward pauses while you drag the simulation to your projection screen. Just install their free PhET […]

Read the full article →

A great new book! Teaching and Learning STEM by Felder and Brent

June 2, 2017

I was pleased to be invited to write a review for Physics Today on a new book, Teaching and Learning STEM by Felder and Brent.  I loved this book!  I found it utterly charming, useful, kind, and knowledgable.  I highly recommend it, and am going to be purchasing several copies to be able to give to […]

Read the full article →

Outdoor activities for kids: Big Book of Nature Activities

January 15, 2017

If you’re feeling a little stuck indoors with your kids, here is a resource to get your kids (or students) learning from the outdoors even through the colder months.  Last year I picked up a copy of The Big Book of Nature Activities: A Year-Round Guide to Outdoor Learning.  I found it to be a great […]

Read the full article →

Writing a proposal on institutional change? Here are some great resources!

December 21, 2016

This is a repost of my blog post at the ASCN (Accelerating Systemic Change Network) blog.  Check out and subscribe to their blog for more posts like this! For many of us, it’s proposal writing season. If you are submitting an NSF-IUSE proposal, there are increasing expectations that the proposal will include a theory of […]

Read the full article →