This post was originally published on the American Evaluation Association’s AEA 365 Blog.

I’m an external evaluator on NSF-funded projects aiming to improve STEM education at the university level. This is part 2 of a 2-part series on Theory of Change. In the last post, I shared the process I use to help clients develop Theories of Change (ToC). Today, I’ll share how I’ve been able to work with a client who serves as an evaluation champion, to develop organizational capacity in Theory of Change development.

As an independent consultant, sometimes I have limited capacity to affect internal processes at a client organization, but this is a success story. One of the first things that I did with this new client was to help them develop a ToC for their NSF proposal. This was a learning experience for both of us. Over four months, we worked through the Project Superwoman example from TheoryofChange.org together, consulted relevant literature, passed pictures of post-it notes back and forth, eventually moving to a PowerPoint diagram. We ended up with a diagram that we felt really represented the project, and helped to develop clear project strategy and evaluation measures. (The image here is our final version; see the last post for the in-progress version.)

Because we collaborated so closely on the original ToC, that project leader understood exactly what should go into a ToC, and on the next grant proposal she developed her own ToC without my help. I have since worked directly with her program manager to develop ToC diagrams for smaller initiatives within the original project, spreading that expertise.

Later, this project leader organized a monthly seminar for other directors at the organization on project evaluation.  One of her staff presented about SMART goals, and the following month I facilitated a workshop about ToC. Two of those directors later spoke to me at length about their projects and I was able to guide them through the beginning of a Theory of Change process.

Ultimately, I now feel that I have had a deep impact on a national organization in terms of evaluative thinking, program strategy, and the ability to develop clear logic chains.

Lesson Learned:

Collaborate closely on a Theory of Change. It can be useful to create some drafts, but then spend significant time revising them with the project team. Just as important as the diagram itself is the learning that occurs through creating it – both in terms of the meaning for the project itself, and for the ability to create ToCs in the future.

Hot Tip:

Take opportunities to run workshops on Theories of Change, for clients or others. It’s an excellent way to learn about their projects, and spread evaluative thinking.

Lesson Learned:

When you find an evaluation champion at an organization, treasure this relationship. Such a champion can help your work as an external evaluator go further for their project, as well as let you have deeper impacts on the organization than otherwise possible.

{ Comments on this entry are closed }

This is a post which was originally published on the American Evaluation Association blog AEA 365.

I’m San external evaluator on NSF-funded projects aiming to improve STEM education at the university level. This is Part 1 of a 2-part series on Theory of Change. Today, I’ll share the process I use to help clients develop Theories of Change (ToC). Tomorrow, I’ll share how I’ve been able to work with an evaluation champion within the organization to develop organizational capacity in Theory of Change development.

The National Science Foundation (NSF) has required Theories of Change as part of many educational proposals for some time, but there has been quite a bit of confusion among my clients as to what exactly is meant by a ToC or why it would be useful.

Lesson Learned:

I’ve learned not to back down when a client doesn’t want to develop a ToC (or take the time to do it well). When I have not done so, target audiences and outcomes were usually muddy. One client fought me the whole way, feeling it was confusing exercise without practical value. Later, however, she related to me, “Now I know why you had me do that, and it was totally useful. Now I’m developing one on my own for this new project and it was so much easier.”  Most clients who develop Theories of Change go on to create (very good!) ToCs to guide other projects, and this gives me pride.

Hot Tip:

Developing a Theory of Change is confusing the first time around. Emphasize with clients that boxes represent outcomes, not strategies. Get your hands dirty with post-it notes; write down outcomes, move them around, and ask questions about how they connect. (The image is our in-progress diagram; see tomorrow’s post for the final version.) Give clients permission to do it their way: The test of a good TOC is that it is useful, not that it is perfect or follows a prescribed model. My clients have been very creative in using color coding or organizational elements to help their diagrams.

Questions I like to ask are:

  1. How do you want the world to be different as a result of your project? (Project vision)
  2. Name a few changes that would need to happen to achieve your vision? (Long-term outcomes)
  3. What changes would need to happen to achieve those outcomes? (Short-term outcomes)

Thus, we essentially “work backwards” from the project vision. Don’t skimp on the project vision! It is such a valuable clarification and inspiration moment for most leaders.

During this process I keep my eye open for “miracles” – steps which seem to “magically” lead to the next step.  A common example is “High quality materials are developed” and “People use those materials effectively.” Wow, magic! The missing steps might be “People find the materials and see that they are valuable” and “People learn how to use the materials.” Such insights lead to valuable additions to the project strategy.

Rad Resources:

{ Comments on this entry are closed }

New free resource for leading change: The Science Education Initiative Handbook

January 31, 2019

For the past few years, I have been working (with my collaborator Warren Code) on a guide to help others use a productive model we’ve used at the University of Colorado Boulder to support change – embedding educational experts within departments to catalyze change. The Science Education Initiative Handbook has practical advice and a suite […]

Read the full article →

New paper published in “Science”: Lecture still dominates in STEM classes

March 30, 2018

I am pleased as punch to announce that I’m a co-author on a new paper just published (today!) in the journal Science:  Anatomy of STEM teaching in North American universities.  This was a large multi-author paper; the big kudos to go Marilyne Stains of the University of Nebraska-Lincoln who led the effort.  I contributed data and […]

Read the full article →

Look around you!

December 18, 2017

  Just a little something light…  This always gives me a chuckle. And here is a fun little interview with the creators.

Read the full article →

What gets in the way of useful evaluation?

November 29, 2017

I have been thinking a lot lately about how to make my work as an evaluator more *useful*.  For those of you unfamiliar, external evaluation is a broadly defined role, intended to give some sort of independent review of a project’s progress and merit.  Evaluation can be a super important part of a project, helping it […]

Read the full article →

Participatory techniques in measurement from #eval17

November 10, 2017

Another big theme at the conference is how to engage your stakeholders in evaluation. These techniques are also relevant to those in educational reform and institutional change, as these would be great ways to include departments, faculty, students, etc., in data to inform change. Why engage?  To increase use of your evaluation Why doesn’t data […]

Read the full article →

Data viz resources from #eval17 (update)

November 10, 2017

I’m enjoying my first time at the American Evaluation Association (AEA) conference here in DC, and finally getting around to writing about a few things that I’m learning. Today’s post is about some of the great data visualization and representations that I’ve been picking up. This is all really relevant to my education research friends. […]

Read the full article →

Some helpful tips in project management

November 1, 2017

I’m a member of the American Evaluation Association (eval.org), which is honestly one of the most productive professional society memberships I’ve ever encountered.  They offer many webinars and amazing resources, plus a daily blog, which are exactly what I’ve needed as an evaluator.   I recently blogged about the wonderful (paid) webinar that I took with […]

Read the full article →

Fidelity of Implementation: Measuring how instructional innovations are used

October 25, 2017

I recently came across an illuminating article by the (ever-diverse) Marilyne Stains and her colleague Trisha Vickrey discussing a particularly sticky issue in education research – how do we know if research-tested techniques and curricula are as effective in practice as promised by the original study? Of course, we don’t – If Professor A at […]

Read the full article →

My friend Paul

September 27, 2017

I have both a sad and joyous post today — one that I have been meaning to write for some time, but understandably struggled to do so.  On August 18th, I lost one of my dearest friends and most loving mentor, Paul Doherty. I have thought of Paul every day since finding out  he was […]

Read the full article →

Condensing the visual display of comparisons: Data Dashboards

September 13, 2017

I’ve been learning more about effective data visualization lately, and recently was in a wonderful webinar on Data Dashboards (with Ann Emery — whose blog has great posts about data viz, such as using color, and telling stories with data).  It was a wonderfully information-packed session, and I’d recommend it to anybody!  I have a […]

Read the full article →

Want to consult? Here are some resources for education consultants.

August 9, 2017

I’m pleased to announce the launch of our new Physics Consultants Directory on PhysPort.org.   Here you can list yourself as a consultant, or find consultants to help with a variety of projects.  We are trying to populate the directory intensively by August 17th, so please try to list yourself by then (though the site […]

Read the full article →

My learning goal and clicker workshops all online

August 3, 2017

Giving workshops on the use of clickers / peer instruction, or learning goals?  I wanted to let you all know that my workshop materials for both topics are all compiled and archived online on our SEI Workshop Page.  There are also videos of several of my workshops (though a few years old, they still show […]

Read the full article →

Defining excellence in physics teacher preparation programs: The PTEPA (#AAPTSm17)

July 26, 2017

A big challenge in physics is preparing adequate numbers of well-prepared future physics teachers.  There is a huge dearth of qualified physics teachers at the high school level, and some physics departments have taken it upon themselves to try to address this gap.  Some are very successful.  How do they do it? I’ve been working […]

Read the full article →

Phys21: Preparing students for diverse careers (#AAPTSM17)

July 26, 2017

I just gave an invited talk at AAPT about my work for the Phys21 Report:  Preparing Students for 21st Century Careers.  I was commissioned by the JTUPP committee to create case studies of how institutions achieved success for diverse students.  This was my favorite project last year, it was completely inspiring to talk about what […]

Read the full article →

Improving the bottom quartile with a metacognitive exercise (#AAPTSM17)

July 25, 2017

I’m in an inspiring session by Charles Atwood (University of Utah) about how they improved the performance of at-risk students in introductory chemistry at the University of Utah. Abstract: To improve success rates in large general chemistry sections at the University of Utah, we realized we must improve the bottom two student quartiles performance. We […]

Read the full article →

Mutual mentoring (liveblogging from #AAPTsm17)

July 25, 2017

I’m now attending a session on Mutual Mentoring for physics faculty, presented by Anne Cox (Eckerd College). Abstract: We were part of an NSF ADVANCE grant mutual mentoring project for senior women faculty in chemistry and physics that began in 2007. We have continued our bi-monthly mentoring meetings for the past 10 years (well beyond the […]

Read the full article →

Promoting innovation and entrepreneurship in physics (liveblogging from #AAPTSm17)

July 24, 2017

I’m at the American Association of Physics Teachers meeting this week, and will blog about a few sessions while I’m here. In a talk by Crystal Bailey (American Physical Society), she argued that we need to more explicitly teach Physics Innovation and Entrepreneurship (PIE) to our students.  I find this a really valuable message; having […]

Read the full article →

Educational change: How systemic thinking helps to push social progress

July 5, 2017

In today’s post I want to share some ponderous thoughts about how educational reforms happen, and how systemic thinking helps to support those reforms.  I am fortunate to be a working group leader in the Accelerating Systemic Change Network (ASCN; ascnhighered.org), and one of the working groups focuses on how theories and models of change can […]

Read the full article →