GROUP DESIGNS: Looking for reasons for change; causality; group and individual designs; internal and external validity; research design notation; experimental, quasi-experimental, and non-experimental designs; threats to internal validity.

• Two fundamental different research strategies, depending on whether we study the individual organism (and ignore the individual organism's relationship to the group) or study the group (ignoring individual variations)

 
• study the individual, singular organism=>
• individual (i.e person)
• individual family
• individual group
• individual community
• individual policy . . . OR

 
• study the group=>
• a group of individuals
• a group of families
• a bunch of groups
• several communities
• a collection of social policies

 
• What is the difference between INDIVIDUAL and GROUP research?

• Illustration of Group v. Individual Designs
• O=Score on a measure of functioning
• X= case management
• RQ = is case management an effective way to increase level of functionning?
• sample = 3 people who get X and 3 people who dont get X

• From a group perspective, the treatment was a failure. Both groups, alike at the beginning (they weren't!) are alike at the end (they arent)! CONCLUDE: NO EFFECT

 
• CLASS: Is this conclusion Right or Wrong?
• ANS: Its right from the group perspective (there is no difference between groups) and wrong from the individual perspective (there is a big difference beween individuals)

 
• group designs:
• better at establishing causality (internal validity)
• weaker at specifying HOW change occurs
• individual designs
• stronger at specifying HOW change occurs
• weaker at establishing causality
• individual designs are preferred for practice evaluation, where we are more interested in IF/How people are changing rather than the exact mechanism
• group designs are preferred for program evaluation and basic research where we are concerned about the mechanism of change (i.e. did the progrrm or X cause the change or was it something else?)

 
• CAUSALITY

 
• why is cause important?

 
• ANSWER: if cause is known, more able to predict (and change) outcome

 
• if cause is known, can develop interventions

 
• eg., some people think the cause of AIDS is unprotected sex, using dirty needles to inject drugs, or being from west central Africa, three seemingly diverse and unrelated situations
• identifying the cause of AIDS at the cellular level permits researchers to work on developing treatments
• identifying AIDs at interpersonal level permits social intervention

 
• "causality" has some strict requirements: (REVIEW)

 
• causality (1) => independent variable (treatment, intervention, policy, factor or condition) is linked to dependent variable (criterion, outcome) i.e. there is an ASSOCIATION

• note: this is usually a statistical, not deterministic association
• to argue that more education leads to more income, I need to observe that the education rate varies with income (at the group level, not at the individual level)

 
• cause (2) => independent variable exists BEFORE the dependent variable in time (usually a function of theory or model)
• education must exist before income

 
• cause (3) => no third factor is linked to both (NON-SPURIOUS)
• e.g. gender (female) may effect both education and income
• systems theory challenges the notion of classical causality
• eg: am I violent with my wife because I drink, do I drink because I am violent with my wife, or are there unknown factors that are linked to both drinking and violence, eg: the motivation to be a powerful person?

 
• RESEARCH DESIGN is the way we attempt to disentangle the web of causality. There are 3 levels of research design (and three levels of confidence in causal relationship) :

 
• non-experimental designs (cannot establish causality)
• quasi-experimental designs (can suggest causality)
• experimental (can tentatively establish causality -- at least until the NEXT experiment)

 
• most social work is non-experimental; we aspire to be quasi-experimental

 
• NOTE: the key element which differentiates science from non-science (including faith) is its ability to be disproven; if something cannot be disproven (nothing can actually be proven) then it is a belief, and it lies outside science

 
• Experiment = a controlled manipulation of one or independent variables to assess the effect on a dependent variable

 
• eg: one group at your agency gets a new service, while another group gets the old service and you want to know whether the new service improves LOF
• X=group membership (old v. new)
• O=LOF
• dependent var is measured one or more times
• people exposed to treatment are called experimental or treatment group

 
• note: they dont have to in an actual "group" but are grouped together by vitue of getting the experimental procedure

 
• Ss not exposed to treatment condition are called the control OR comparison group, depending on how they came to be in the group

 
• random assign (R) => control
• non-random => comparison

 
• The confidence we have that there is a true relationship between X and Y is called internal validity

 
• NOTE: internal validity refers to a true relationship between variables Experimental (DESIGN)

 
• internal validity ==> extent to which change in the dependent variable can be attributed to the experimental or treatment condition

 
• measurement validity ==> extent to which measure is a true/accurate reflection of a concept

 
• internal validity the extent to which the change in the dependent variable can be accounted for by the independent variable

 
• external validity the extent to which we can generalize the relationship of the dependent and independent variables

 

 
 
 
 
 

level of design validity depends on the design involved

• X => intervention or independent variable
• O => measurement
• R => random assignment or random selection

 
• non-experimental=virtually no validity except case control can handle regression effect

 
• case study X O

 
• e.g. people completing a job training program are evaluated on their job readiness skills

 
• pre-post single group O X O

 
• e.g. people are evaluated on their job readiness skills, attend a job training program, then are re-evaluated on their job readiness skills using the same measure

 
• case control                  X O
                                                     O

• e.g. the job readiness skills of a group of people who have had a job training program are compared with the job readiness skills of a group of people who have not had a job training program

 
• quasi experimental = either random assignment/selection OR a comparison group

 
• time series (OOOXOOO) aka panel study

 
• e.g. the job readiness skills a group of people are measured every month for 3 months, then they are given a job training program, and their job readiness skills are measured every month for another 3 months

 
• eg: nielson ratings

 
• KEY: same group over time

 
• controlled time series O O O X O O

                                 O O O     O O

e.g. the job readiness skills a group of people are measured every month for 3 months, then they are given a job training program, and their job readiness skills are measured every month for another 3 months. These people are compared a group of people whose job readiness skills are measured every month for 6 months

• same group over time w/ same comparison

 
• trend study     R O
                                       XR O
                                                    R O etc

• different group over time
• random sample
• eg: election polls
• e.g. Effect of Impeachment hearings (X) on Clinton's approval rating (O)

• experimental = random assignment to a control group

 
true experiment R O X O
                        R O     O

• e.g. people are randomly assigned to two groups, and their job readiness skills are measured. One group gets a job training program and the other groups gets nothing. Then the job readiness skills of everyone is measured again

 
• post only control group

 
• reduces testing effect

 

                                                    R     O
 

• e.g. people are randomly assigned to two groups. One group gets a job training program and the other groups gets nothing. After the program, the job readiness skills of everyone is measured.

• solomon 4-group (true experiment + post only) measures the effect of the testing

                                R O     O

                                R     X O

                                R         O
 

• Key element in experimental and quasi-experimental designs is control for the effects of other factors which could affect the relationship between the independent and dependent variables

 
• Control provided by one of three means:

 
• random assignment (note: NOT random selection)
• everyone has equal chance of being in tx grp
• reduces the effect of most extraneous variables--participants in experimental and control groups are just as likely to have an error-producing characteristic

 
• e.g randomly assigning people to job training or no job training is likely to control for the effects of
• age
• race
• gender
• work experience
• social class
• motivation to work

 
• (2) matching of 1-2 critical variables

 
• alternative to random asignment

 
• reduces the effect of known critical variables on between group differences

 
• e.g. believe that job readiness would be effected by gender and age, we can match subjects by gender and age.

 
• for every woman under the age of 18 getting the job readiness program, there would be a woman under the age of 18 not getting the job readiness program

 
• requires a different statistical treatment, e.g. t-test, referred to as related groups or matched t-test

 
• matching is very difficult to do with more than a few variables

 
• BLOCKING: for small samples, use random assignment after matching on a key variable

 
• (3) statistical control: allow dependent variable to change while holding one or more independent variables constant

 
• e.g. if we had a singel sample, we can hold gender and age constant and compare people who got the job training program with people who did not get it.

 
• works best with random selection because we can then specify how much error we can expect

 
• threats to experimental validity (HANDOUT)

 
• example: alchohol tx program, compare usual & customary tx to experimental program using U&C program integrated with adventure therapy & outward bound principals

 
• X = adventure therapy program O = stress, depression, problem solving, alcohol craving, externality @ O₁ and O₂, abstinance @ 10 months (note: abstinance not measured at O₁ and O₂ and the other Y's not measured at O₃

 
• design = O₁ X O₂ O₃

(non-equivalent comparison group; quasi-experimental)

                    HISTORY (yes): effect of different external events

 
• did events occur, other than tx, which affected one group and not the other?

 
• e.g. saw a fiery car crash on the 50-mile journey, had a religious conversion, and didnt tell anyone about it (i.e. not part of tx)

 
• can't say--have to believe it might

 
• INSTRUMENTATION (no): effect of different measures

 
• both groups got same measures--should not be a problem

 
• MATURATION (yes but no): effect of internal events & development

 
• could one group, over 10 month period, have developed or matured differently?

 
• probably not

 
• SELECTIVITY (yes!): effect of differences between groups

 
• are the groups different to begin with--yes--there is every reason to believe that more motivated people volunteered for the experimental program

 
• they do NOT, however, differ statistically in the five Y's @ O_1, (statistical control) suggesting they are alike

 
• groups may have differed on abstinance @t1 and @t2, or on 5 vars @t3

 
• EXPERIMENTAL ATTRITION (yes but no): effects of people leaving group

 
• did a different dropout rate occur? no

 
• dropouts counted as relapsed at O₃

 
• REGRESSION (yes): basement & ceiling effects: extremes tend toward the mean

 
• did any "extremes" get more average? Yes: externality

 
• they were not selected for this characteristics, but alcoholics always tend to be very external, and therefore "get better" no matter what

 
• Statistical control => groups did not differ, but all tended to reduce externality over time, perhaps at a different rate

 
• TESTING EFFECT (yes but no): effects of learning to take a test and "psyching out the test"

 
• note: if test changed, it would be an instrumentation threat
• could the participants have learned to take the test?
• is there any reason believe they did? No