[Ailist] Re: How to makes sense of already-collected data?

[Jeff Miller]

On 30 May 2009 at 16:00, ailist-request at lists.business.utah.edu wrote:

> Colleagues:
> 
> I'm currently doing some research, and was hoping that someone could 
> point me to some literature or ideas that I may be missing.

Gary:

in addition to the things that Jane has shared, you may want to take a look at something 
called portfolio (or dossier) evaluation.  Back in the early 90's I was faced with having to 
coordinate a national evaluation of 90 federally funded community projects in which each 
had already had their own individual evaluation design approved as part of their grants.  
The folks in charge wanted to do something that would "aggregate" the effects of the total 
initiative.

So we embarked on encouraging each of the sites to develop a portfolio of the work they 
were doing.  Much training and explanation of the kinds of things to include in a portfolio 
were done.  And the amazing thing was that many of the project staff actually got excited 
about doing evaluation!  I know, hard to believe.

Once the portfolios were submitted, a review team was convened to read and respond to 
each project.  In their training, we worked with them to provide feedback to the projects 
on their work, not on the quality of the portfolio.  The management of the process was 
done this way......

- portfolios separated into three groups

- review members also assigned to three panels (so each person read/saw about a 1/3 of 
the programs)

- reviewers consisted of peer program directors, administrators of like programs, 
evaluation experts, and the funder

- within each panel, 2-3 portfolios were assigned to each member to be the "shepard" of 
the review process.  They became most familiar with these projects and collected the 
feedback and discussion generated when the panel talked about each one

- feedback was provided on carbonless paper, so the evaluation team could keep a copy 
and the original returned to the local program.  The form asked two major questions....  
"What is exemplarly and exciting about this program", and "What questions, ideas, 
suggestions do you have for the program?"

- After all the reviews were completed the three panels were scrambled and assigned to 
three focus groups.  The focus groups addressed three major questions....
    - what trends, themes, common/similar impacts do you see emerging from these 
    programs?
    - what research questions come to mind as you have reviewed these various 
    programs
    - what advice/suggestions would you have for policy makers as they move forward 
    with this (or similar) initiatives

Helping me in putting all this together was Dr. Ken Peterson from Portland State 
University.  So if you were to do a search of articles (and I believe he has a book about 
these kinds of processes) he's written you would find much detail.  Ken pulled deeply from 
his background as an education professor who worked with teachers on dossier 
development.  So that's where you'll find much of the literature.  

Somewhere in the bowels of my computer I've got a copy of an article we did after our 
review process.  I'd be happy to send it along to you if you want (and I can locate it).

I hope you find this helpful.....    in writing this, I've just realized what a cool "appreciative" 
approach this was long before I'd ever heard of A.I.!!!!

Good luck...  jeff miller

P.S.  My more "glib" response is.......  that's what graduate students are there for (or 
undergrads who what some qualitative data management experience).... actually, I 
wouldn't be opposed to having an advance class help with the sorting, sifting, and 
possibly some of the sense making.


*** Note new Address ***
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jeff Miller, Ph.D.
Innovative Leadership Solutions, Inc.
7863 S. State Road 267
Brownsburg, IN 46112
office: +1 317-733-8635
via Skype: jeffmiller79
http://www.inleadsol.com
------------------------------------------
e-mail:  jeff at inleadsol.com  or                   
miller.3293 at osu.edu 

 - Vist the "ShareAndLearn" blog at:
http://jeffsshareandlearn.blogspot.com
- See the ShareAndLearn e-archive at:
http://lists.topica.com/lists/shareandlearn/read

"Change happens from the outside in but transformation happens from 
the inside out."             - Dan Burrus
.