Unexpected events during an evaluation all too often send evaluators into crisis mode. This insightful book provides a systematic framework for diagnosing, anticipating, accommodating, and reining in costs of evaluation surprises.
1. From Firefighting to Systematic Action Adding "Surprise" to the Mix Historical Roots: Evaluation, Planning, and System Behavior From Explaining Surprise to Dealing with It Development Path of This Book Guiding Principles How to Read This Book In Sum 2. Structure of the Unexpected Where Does Surprise Come From? Beyond Simple Distinctions In Sum 3. Placing Surprise in the Evaluation Landscape When Is the Probability of Surprise High? When Is Surprise Disruptive to Evaluation? In Sum4. Minimizing Foreseeable Surprise Theory: Using Explanatory Power and Simplified Relationships Exploiting Past Experience: Capitalizing on What We Already Know Limiting Time Frames to Minimize the Opportunity for Surprise In Sum5. Shifting from Advance Planning to Early Detection Leading Indicators System-Based Logic Modeling In Sum 6. Agile Evaluation Data Agile Methodology Retooling Program Theory Agility and Stakeholder Needs In Sum 7. How Much Is Too Much?: Appreciating Trade-Offs and Managing the Balance A Framework for Appreciating Design Trade-Offs Maximizing Choice, Minimizing Risk Evaluation Design In Sum 8. Applying the Examples to Categories of Cases: The Life Cycle View "Unintended Consequences": Unity across Programs and Their Evaluations Interpreting Cases through a Life Cycle Perspective In Sum 9. Applying the Examples to Categories of Cases: The Social/Organizational View Navigating through the Cases Placement of Cases on the Social/Organizational Map Categorizations Derived from the Data In Sum 10. Lessons from Individual Cases: Tactics for Anticipating Surprise In Sum 11. Lessons from Individual Cases: Responding to Surprise The Middle Leading Indicators and Agile Evaluation In Sum 12. Unanticipated Program Outcomes Case Descriptions Applying the Cases to Unintended Program Outcomes Comparing the Cases Predicting the Need for Agile Evaluation In Sum 13. Concluding ThoughtsCasesCase 1. Grasping at Straws and Discovering a Different Program Theory: An Exercise in Reengineering Analysis Logic in a Child Care Evaluation Setting, Dennis P. AffholterCase 2. Shifting Sands in a Training Evaluation Context, James W. Altschuld and Phyllis M. ThomasCase 3. Evaluating Programs Aimed at Promoting Child Well-Being: The Case of Local Social Welfare Agencies in Jerusalem, Anat ZeiraCase 4. Assessing the Impact of Providing Laptop Computers to Students, J. Dan Strahl, Deborah L. Lowther, and Steven M. RossCase 5. Quasi-Experimental Strategies When Randomization Fails: Propensity Score Matching and Sensitivity Analysis in Whole-School Reform, Gary L. Bowen, Roderick A. Rose, and Shenyang GuoCase 6. Unexpected Changes in Program Delivery: The Perils of Overlooking Process Data When Evaluating HIV Prevention, Bryce D. SmithCase 7. Evaluating Costs and Benefits of Consumer-Operated Services: Unexpected Resistance, Unanticipated Insights, and Deja Vu All Over Again, Brian T. YatesCase 8. Keep Up with the Program!: Adapting the Evaluation Focus to Align with a College Transition Program's Changing Goals, Kristine L. Chadwick and Jennifer Conner BlatzCase 9. Assumptions about School Staff's Competencies and Likely Program Impacts, Laura Hassler Lang, Christine E. Johnson, and Shana GoldwynCase 10. Mixed Method Evaluation of a Support Project for Nonprofit Organizations, Riki Savaya and Mark WaysmanCase 11. Evaluating the Health Impacts of Central Heating, Jeremy Walker, Richard Mitchell, Stephen Platt, and Mark PetticrewCase 12. Recruiting Target Audience: When All Else Fails, Use the Indirect Approach for Evaluating Substance Abuse Prevention, Molly EngleCase 13. Unintended Consequences of Changing Funder Requirements Midproject on Outcome Evaluation Design and Results in HIV Outreach Services, Lena Lundgren, Therese Fitzgerald, and Deborah ChasslerCase 14. Generating and Using Evaluation Feedback for Providing Countywide Family Support Services, Deborah L. WassermanCase 15. Trauma and Posttraumatic Stress Disorder among Female Clients in Methadone Maintenance Treatment in Israel: From Simple Assessment to Complex InterventionMiriam Schiff and Shabtay LevitCase 16. From Unintended to Undesirable Effects of Health Intervention: The Case of User Fees Abolition in Niger, West Africa, Valery Ridde and Aissa DiarraCase 17. Unintended Consequences and Adapting Evaluation: Katrina Aid Today National Case Management Consortium, Amanda Janis and Kelly M. Stiefel Case 18. Evaluation of the Integrated Services Pilot Program from Western Australia, Peter Hancock, Trudi Cooper, and Susanne Therese Bahn
