[Ailist] How to makes sense of already-collected data?

[Jane Magruder Watkins]

Dear Gary, 

Many years ago in the late 1970's, I had the privilege of working as a
consultant with the first international development grant that USAID gave to
the Adventist. As a mainline Episcopalian, needless to say I knew very
little about Adventists, but over the following decade I worked with them in
over 25 countries around the world and came to be a huge fan of the work
that the church does globally and of the people themselves. This was before
AI. Since that time thanks to my friend (and perhaps yours) Ray Tetz, I have
had the pleasure of introducing AI to several Adventists groups,
particularly in the health arena. All of which made me an equally big fan of
Loma Linda. So, nice to hear from you on the list! 

I can't resist telling you a Ray Tetz story! I was talking to him about the
fact that I recommend to anyone working in the developing world to find the
nearest Adventist hospital just in case. I noted, however, that they were
often located on "the back side of the beyond" as my grandmother used to
say, and could be very hard to get to. Ray told me this story: "We use the
"donkey-died" method for locating hospitals and schools. We start in the
capital city in a truck. When the truck "dies," we find any type of vehicle
that will take us further. When that vehicle "dies" we find a horse. When
the horse dies, we get a donkey. When the donkey "dies" that's where we put
the school and hospital!" And I believe it! They work in the most remote
places with the most deprived people and thereby change the world for many a
struggling community!

About evaluation. It was in those early days working in the "outback" of
Africa that we began trying to figure out how to do evaluation in ways that
helped people see it as a helpful and generative process rather than as
judgment. We talked a lot about formative (help us learn) processes and
summative (what we have to have for the donor's) processes. I came to
believe that what we most often get from those processes are dilemmas such
as the one you find yourself in - data out the wazoo! 

Then as I got into this current reality of continuous change and the
realizations that human communities actually have the capacity to create
their own reality through dialogue and agreements about what is real and
important, I began to understand that "evaluation" data from human systems
is good for about a "nano" second! All of which is to say, the most useful
evaluation data is that which is created in the process of implementation
and used to decide on next steps. So my question to your "client" is, "What
do you expect to get out of all this 'dead' data?" 

I don't mean to be glib about this. My approach is to get the system itself
involved in making sense of what they have found in the past so that they
can find out for themselves what use to make of it. Often they get great
"ah-ha" moments from revisiting past successes; less so from the study of
past 'failures.' 

On the practical side, if the data is anecdotal, there are now computer
programs that work with narrative information using a word search process.
For Lickert scale data, it's all number crunching and I'm not familiar with
methods for that anymore since my approach to "valuation" these days is to
help the group being evaluated to make sense of their own stories and
numbers. 

I hope this helps. And thanks for your patience with my stories! One thing
about getting old is that sharing memories of the past becomes a favorite
pastime! Let me know what you do with all that data! 

Appreciatively, Jane

Jane Magruder Watkins & Ralph Kelly
Appreciative Inquiry Unlimited
An Organization Development Center for Teaching, Consulting and Mentoring
 
Office & Home
233A Woodmere Drive
Williamsburg, VA 23185
(757) 259-9942
 
MarshHaven Retreat Center
P.O. Box 541
1702 Wheat Patch Road
Belhaven, NC 27810
(252) 964-3072 
 
www.appreciativeinquiryunlimited.com
 
Appreciative Inquiry: Change at the Speed of Imagination by Jane Magruder
Watkins & Bernard Mohr can be ordered from Amazon.com or JosseyBassWiley
 
 
-----Original Message-----
From: ailist-bounces at lists.business.utah.edu
[mailto:ailist-bounces at lists.business.utah.edu] On Behalf Of Gary Robbins
Sent: Friday, May 29, 2009 1:03 PM
To: ailist at lists.business.utah.edu
Subject: [Ailist] How to makes sense of already-collected data?

Colleagues:

I'm currently doing some research, and was hoping that someone could 
point me to some literature or ideas that I may be missing.

The evaluation literature seems to be geared largely toward implementing 
evaluation models and methods in 'ideal' situations, where the 
evaluators are able to influence the evaluation procedures from the 
beginning of program implementation, so that the formative and summative 
evaluation components are built in to the program at the onset.

I'm wondering if there's a model of working with a program that has been 
running for some time and that has been collecting data in a haphazard, 
we-think-this-might-be-interesting-to-know-someday sort of way.  In 
particular, I'm thinking of a smaller psychoeducational program that did 
not have any particular evaluation guidance at the beginning, so put 
together a trial pre/post instrument, and has been collecting 
information for a few years now, but the data is just accumulating in a 
storage room at the facility.  The program wants to try to wrangle the 
information that they've collected to see if any of it is informative, 
or can be used to provide a sense of how the program has done in the 
past, but there's so much data, and any or all of it may or may not be 
usable, that they are unsure how to begin wading through it all.

So, does anyone know of any kind of step-by-step process, or list of 
strategies or rules or best practices for wading through evaluation data 
in a less-than-ideal evaluation situation like this, where the evaluator 
is asked to come in and see what they can do with the information that 
the program has already been collecting?

Thanks so much for any insight into this issue.
Gary Robbins
Loma Linda University
_______________________________________________
The Appreciative Inquiry Discussion List is hosted by the David Eccles
School of Business at the University of Utah. Jack Brittain is the list
administrator. For subscription information, go to:
http://mailman.business.utah.edu:8080/mailman/listinfo/ailist