problem            need         goals         intervention        outcome

evaluation

• Are the outcomes as they should be?
• Is it the right intervention?
• Are we dealing with the right problem?
• Is there a need for the service?
• Do we have the right goals?

• evaluation research => use of science to plan, monitor, and evaluate effect

 
• two types of evaluation:

PROGRAM EVALUATION Formative Evaluation (Also called Process Evaluation) Gather data on population What are the intervention options? Is the intervention is being employed as planned?  Are the goals and objectives are appropriate for the program? Does a need for the program exist? (Needs Assessment) Is evaluation feasible? (Evaluability study) Do staff have necessary skills?    Summative Evaluation Was the intervention employed as planned?  Are program goals met? (Outcome Evaluation) How much did the program contribute to the outcome? (Internal Validity) How much does the outcome support the theory upon which the program is built?

• Program Goals & Objectives

 
• goal (dependent variable, outcome, final condition) = who will do what to what extent and under what circumstances

 
• objective = concrete steps to meet in order to acheive goal

 
• objectives may also be instrumental, partial, or process goals

 
• instrumental goal = theory-based goal which, if met, will advance final goal

 
• process (facilitative) goal = goal which facilitates final goal, but is not an end itself

 
• partial goal = in program evaluation, a partial goal is a smaller, intermediate step towards the larger goal, usually in terms of quantity

 
 

Goals & Objectives: Example 1 problem = staff dissatisfaction in a large social services agency   Goal 1 (final goal): 90% of direct service and support staff will report they are satisfied with their job (final goal) Goal 2 (partial goal): 75% of support staff report they are satisfied with their job within 12 months  Objective 2.1 (process goal): All support staff will meet with their supervisor in the next 30 days for a job review  Objective 2.2 (process goal): Quality circles have been are implemented in all program units, and meet monthly at least 6 times in next 12 months


 

• Logic Model

 
• useful in evaluating existing programs

 
• Step 1: identify interventions or proogram elements

 
• Step 2: for each program element, identify at least one logical (sucessful) outcome

 
• Step 3: for each logical outcome, identify at least one emirical indicator

 

Risk factor 4: Safety & Resource Utilization
 

 

Program Components (Week Number)	Outcomes	Measures
Safety Visualization (1) Creating a Safety Plan (18) Community Resources (18) Creating Allies (20) Warning Signs (5)	4 Increase knowledge of and intent to utilize community resources	4.1 Resource & Help-seeking Questionnaire (6 items) 4.2 Violence Supportive Attitudes Questionnaire (items 14, 15)

• cost effect & cost benefit

 
• cost

 
• direct cost: how much program money must be spent?
• how much $ will be spent in job satisfaction program?
• how much $ will be spent in dating violence program?

 
• opportunity cost: what opportunities are missed as a result of focusing on this program?
• time spent away from other things

 
• benefit
• to participant: in units of "feeling better"
• to funding source: in units of "productivity"
• to society: in units of "better service" or "reduced crime"

 
• effect
• change units (per goal--depends on how dependent variable is measured)

 
• JOBSAT=objective measure, comp time, sick days, productivity indicators,

 
• unplanned effect: spinoffs, outcomes that may not be planned but can be linked to process (e.g. increased productivity)

• calculation of cost/benefit and cost/effectiveness

 
• e.g.: cost/effect of rape victim advocacy

 
• goal=reduced level of PTSD (measure=Trauma Severity Checklist, 33 items, average scores have been 45, and average % of women above the cutting score of 30 is 50%)

 
• goal=increased rate of successful prosecution (measure = % conviction, baseline=31%

 
• X=2 victim advocates, each handing 200 cases/year, salary=$28,000 ea. + 20% fringe=$67,200

 
• outcome: 350 cases,

 
• PTSD scores reduced to 35 (-10) and 250 of 350 are below the cutting score

 
• old model would have expected 175 above cutting score, so net is 250-175=75 cases not clinical PTSD which would have been w/o program

 
• prosecution rate increases to 50% (+19%, or 175 convictions)

 
• previous rate would have been 109, so netted 175-109=66 convictions)

 
• cost/effect 1 = 67,200/75=$896 per person w/o PTSD

 
• what is not having PTSD worth in terms of opportunity, production, functioning? => cost benefit (economists & taxpayers/funders must answer)

 
• cost effect 2 = 67,200/66 = $1,018 per conviction
• what is a conviction worth in terms of opportunity, risk to society (economists and taxpayer must answer)