[Ailist] Capturing impact and raising visibility of 'unseen' work

['Marvin Faure']

Dear Jane and Roger,

Many thanks for adding to this debate. I completely agree with Roger that
“In order to convince leaders that a valuation is sound it needs to be given
to them in a framework that they and shareholders alike will understand”,
and that “any valuation of an organization approach needs to be focused on
something that has a good degree of certainty about it”. 

The question is: how to do this?

Jane has emphasised the socially constructed nature of data gathered in
organisations and hence its fragility. If you ask one set of questions
intended to uncover problems and another set of questions intended to
uncover what’s good you will get entirely different sets of data, from which
you could potentially draw conflicting conclusions. Social constructionism
also suggests that you will have pushed the organisation in the direction of
your questions, implying that either the problems just got worse or the good
stuff got better.

Does this mean that all attempts to measure behaviour and attitudes in human
systems are flawed, in the sense that the measurement is always affected by
the act of measurement? Will we always find what we are looking for? Or is
there some magic design out there that somehow avoids all these flaws? And
if so, can it work in the real world or does it only work in academic
research labs? Perhaps one of the many PhD’s on this mailing list could help
us out with a rigorous answer.

I am inclined to follow a pragmatic approach through this minefield: if the
client is aware of the limitations of what they are asking for, if they
still want it, and if it stands a chance of doing some good, then let’s go
ahead. 

When clients need qualitative data I try to begin by clarifying exactly what
needs to be measured and why. If the measures are linked directly to a
significant strategic goal and there is energy to act upon the feedback
(whether good, bad or indifferent) then it’s probably worth the effort to
analyse the performance of existing business processes and thus identify the
activities, behaviours and attitudes needed to support the strategic agenda.


While these first steps are relatively straightforward, the next ones could
be called “fateful” in the Ai sense. Are we going to measure the good or the
bad? Again speaking pragmatically, managers sometimes need to see “proof” of
problems before acting: for such people, the evaluation needs to be designed
to highlight the problems. In other words one would seek to show to what
extent the desired activities, behaviours and attitudes are currently
ABSENT. This is certainly most - perhaps only - appropriate at the outset of
a change process. 

Once the change process is under way I subscribe strongly to the AI view
that asking for evidence of what you want will get you more of it. The best
way to (e)valuate the success of the change process is therefore to show to
what extent the desired activities, behaviours and attitudes are currently
PRESENT. This not only provides data to satisfy the bean-counters that the
money was well spent, but the very act of (e)valuation also reinforces the
change process.

All this discussion leads to a final question: how to gather the data?
Jane’s paper on the AI Commons (written with Bernard Mohr) describes an
evaluation process based on 1-hour AI interviews of 100 out of 480 program
participants. In cases where it is not feasible to conduct extensive
interviews, I’m not sure what would be the alternative to an on-line
questionnaire. Jane: do I understand you to be strongly against such tools,
no matter what, or do you believe they can work if well designed?

My tentative conclusions are:
- Pragmatically speaking, the need for measurement will never go away.
- We need to be aware of its socially constructed consequences.
- Prefer AI-type questions: ask for evidence of goodness.

Anybody see this differently?

Best regards,

Marvin

Marvin FAURE
MINDSTORE
Chemin du Canal 5
CH-1260 Nyon
Switzerland
Tél. +41 22 363 9286
Mob. +41 786 826 926	
www.mindstore.ch