The Basic Strategies for Program Evaluation & Improvement
Barry Sweeny, © 2003

INDEX:

The Challenges of DOING Program Evaluation

Isn't it true that the time we need learn how to provide quality processes is in direct competition with the very time we need to do those processes!?

In other words, we are so busy doing our mentoring program that we don't have the time to evaluate the program for ways to improve what we are doing. As true as this is, this situation may become a fatal flaw. If we do not collect and use data about what we do to improve what we do, we may not be allowed to continue doing it. After all, what decision maker would allow a program to continue that can not demonstrate that it is accomplishing what it is intended to accomplish? Add to this problem that most of us do not count program assessment and evaluation as one of our strengths. These are just a few reasons why effective program evaluation gets put off and often, never is done.

So... what do we need to do?

The Solution Simply Stated

Sorry, there is no easy solution. Just as we are really struggling to more conduct effective employee assessment, and continual improvement processes, so we must struggle and learn how to conduct and use better program assessment. Simply stated, you must be able to demonstrate the impact of your program if you expect decision makers to continue to support your work. Also, do not wait for them to ask for the data. Prepare the processes, data, and report on results before they ask for it. There are four aspects of this task:

Build the Whole Chain of Causes & Effects

Improving bottom line results is almost always the real reason for change initiatives. Yet, such results are the last link in a long chain of events that are each necessary to attain improved results.

Earlier links in the chain include work plans that are aligned to the standards, assessment that is aligned to the work flow, employees who have the skills to effectively do that work, employees who actually do the needed work, sufficient materials and technologies to support effective aligned work, etc.

Before we can expect the last link in the chain to be in place, we understand that every other link in the sequence must first be there.

Collect Data on Every Link in the Chain

Program assessment must provide the data we need, not just about the last link in the chain (desired results), but about every link in the chain. We need to know the current situation with each link and then we need to monitor the progress in getting each link in the chain in place and functioning effectively. For example, what data could tell you the amount of time needed for mentoring to be effective for most proteges? How could you collect that data? OR, What data could tell you the optimal number of coaching cycles needed for a mentoring pair to begin to positively impact their performance? How could you collect that data?

Assess Early and Middle Indicators For Progress

We need to identify and collect data on early indicators and midpoints of progress, those earlier links in the chain. Early and middle indicators are those which will be among the first things to change as improvements are put in place.

ï For example, when a middle school goes to team-based structures, one of the first things to indicate that the change is starting is that student and teacher attendance improves. Conversely, one of the last things to change will be student learning, the real purpose for the original change to an effective middle school structure. It may take several years for all the necessary links to be put in place to where teachers are providing improved instruction and, finally, student learning begins to visibly improve. If we watch for improved student learning early on in the process, we may wrongly conclude that "nothing is happening".

ï Another example is more obvious, as it is a mentoring example. Even if improved employee performance or implementation of training are the desired result from mentoring, the first things that have to change are building a strong, trusting, confidential relationship, setting goals for their work, and meeting sufficient times to gain progress on implementing their work plans. Therefore, some of the first EARLY indicators of change will be frequency and duration of mentoring pair meetings, topics discussed, reports of the quality of the relationship, baseline data on competencies collected, and setting of goals. MID-process indicators for the desired change may be number of coaching observations conducted and the number of other professional growth plan activities completed.

The big lesson here is, to watch for and capture the early and mid-process indicators in the growth process and only expect the later indicators to change when the early indicators are all strong.

Anticipate the Timing of the Whole Sequence

When all the early indicators are strong, we can assume that all the things necessary for later results to improve are in place. Only then is it reasonable and time to expect that productivity and desired results should begin to improve. Improving final results is a developmental process, just like gardening. We cannot expect the harvest until all the prior steps in the process have had time to happen. In development, you can not plant the seeds and then harvest the crop. This is why time is so critical in development.

In mentoring the earlier indicators are vast. To understand what should be assessed, make a list of the sequence that you think must occur to get all the links in the chain of effective mentoring programs and practices in place. In other words, what are the necessary steps for beginning managers to become effective leaders? (if that is the ultimate goal of mentoring) After you have created that list, decide what data could be collected about each step in that process. What data will tell you the strength of each link in the chain?

Then you need a schedule that spells out when that data should be collected. That is, what are the indicators (links in the chain). Data for every indicator should be initially collected to give you a baseline against which to measure for tracking progress. Once you have a baseline, predict the points at which (specifically when) you should begin to see some of the earlier and later indicators change. Again, the experience of mentoring program leaders in other organizations is very helpful with such predictions. Collect the data before the change is expected, when it is expected, and after it is expected. You "bracket" the expected time because you can not assume that your prediction will be correct, and you want the data to tell you when the changes started to happen.

The Ultimate Goal of Program Evacuation is NOT Program Improvement!

Keep in mind that you are not collecting data just to show that the desired early and later changes have occurred. If that is all that you do, you may not learn enough to be able to sustain the changes. You also what to understand when and (if possible) why the changes occurred so that you know what caused the changes. That places you in a much more proactive position. In other words, your goal should not just be to improve results. Your goal should be to learn how to conduct a mentoring program that causes improved results.


Return to TOP

© 2003 by the International Mentoring Association

All materials on this web site are copyright protected and all rights are reserved to the IMA.

http://www.mentoring-association.org

However, if you do so for free, you may duplicate and distribute this information or link to it, as long as the full page and all attributions and contact information are preserved as they appear on this page. You may not sell this information in any form.