Design of Studies (Pt. 2)

Homer White, Georgetown College

In Part 2:

Load Packages

Always remember to make sure the necessary packages are loaded:


Completely Randomized Designs

Experimental Ideal

Remember that our ideal is:

the treatment groups should be as similar as possible with respect to every variable (except \( X \)) that might affect \( Y \).

Interestingly, we always use chance, at least in part, in order to approximate this ideal.

Completely Randomized Design

In a completely randomized design, subjects are divided into the available treatment groups solely by chance.

If a treatment group is to have size \( m \), then every set of \( m \) subjects has the same chance to be that treatment group.


For an experiment on whether aspirin reduces risk of heart disease, suppose that the first 200 members of imagpop have agreed to participate:

AspHeartSubs <- imagpop[1:200,]

Treatment Groups

Group Treatment Size
Aspirin Group Aspirin Pills 100 subjects
Placebo Group Fake Pills 100 subjects

Complete Randomization:

  • pick 100 at random for Aspirin Group
  • rest will be in Placebo Group

The Code

Assignment <- RandomExp(AspHeartSubs,


   sex math income treat.grp
5 male   no  15500   placebo
6 male   no  49800   placebo

       sex math income treat.grp
200 female   no  88000   placebo
1   female   no  40900   aspirin

       sex math income treat.grp
198   male   no  13700   aspirin
199 female   no  52700   aspirin

Groups Similar ...

… with respect to sex?

sex      placebo aspirin
  female      51      56
  male        49      44
        female male Total
placebo     51   49   100
aspirin     56   44   100

Groups Similar ...

… with respect to income?

   .group  mean    sd
1 placebo 39215 26995
2 aspirin 39486 29352


An experiment is said to involve replication if each the treatment group contains more than one subject.

The more replication, the better!

There are two reasons why replication is good.

First Reason

The more subjects you have, the more likely it is that the randomization produces treatment groups that are similar to each other.

(And the more similar they are likely to be!)

Second Reason

This has to do with our notion of causality.


Roughly we say that \( X \) causes \( Y \) if, there exist two different values \( x_1 \) and \( x_2 \) of \( X \) so that whenever two people A and B are alike in every way except that

  • A's value of \( X \) is \( x_1 \), and
  • B's value of \( X \) is \( x_2 \)


the distribution of \( Y \) for A is different from the distribution of \( Y \) for B.

So, for example ...

We would say that asprin causes reduced risk of heart disease IF we could show that:

if two people are alike in every way except that one takes aspirin and the other does not, then the one who takes asprin has a less chance of getting heart disease.

But ...

  • In practice, we can't find two people who are exactly alike.
  • With just two people, we could not estimate very well the chance of heart disease for each one.

So ..

… we settle for two groups, instead. If the number of subjects is large and we randomize, then

  • we can be pretty sure that the groups are pretty similar, and
  • we can use the outcomes for members of each group to estimate the chance of heart disease for each group.

If these estimated chances differ enough, then we can ascribe the difference to the asprin.

Randomized Block Designs

What do you do if you can't have much replication?

Back to Contents

A Small Experiment


16 subjects for experiment to compare two weight-lifting programs:

  • Program A, and
  • Program B

Problem With Limited Replication

Try complete randomization several times:


Do the groups always look similar?

Partial Solution: Block ...

Step One: Create Blocks (groups of very similar subjects):

  • the four females who are athletes
  • the four females who are not athletes
  • the four males who are athletes
  • the four males who are not athletes

... Then Randomize

Step two: In each block, randomly assign subjects to the two groups:

  • the four females who are athletes:
    • two at random to Group A; other two to Group B
  • the four females who are not athletes:
    • two at random to Group A; other two to Group B
  • the four males who are athletes:
    • two at random to Group A; other two to Group B
  • the four males who are not athletes:
    • two at random to Group A; other two to Group B

Code for Randomized Block Design


Then at least sex and athelete won't be confounding factors!

Blocking Example

   salt biomass  block
1    10    11.8 Field1
2    15    21.3 Field1
3    20     8.8 Field1
4    25    10.4 Field1
5    30     2.2 Field1
6    35     8.4 Field1
7    10    15.1 Field2
8    15    22.3 Field2
9    20     8.1 Field2
10   25     8.5 Field2

Blocking Example

Research Question: Does the level of salt in soil affect how much plant life grows there?

  • Four fields
  • Each field is a block
  • Each field divided into six equal plots
  • In each field, plots randomly assigned to the six levels of salt

Blocking Example

You can tell blocking was done:

salt Field1 Field2 Field3 Field4
  10      1      1      1      1
  15      1      1      1      1
  20      1      1      1      1
  25      1      1      1      1
  30      1      1      1      1
  35      1      1      1      1

Saltmarsh Results

Biomass and Salinity.  The higher the concentration of salt, the lower the biomass in the plot.

Matched Pairs/Repeated Measures

Matched Pairs

Suppose you plan two treatment groups.

A matched pairs design is an extreme form of blocking in which each block contains exactly two subjects.

  • In each pair, randomly assign one member to Group A, other to Group B
  • Usually compute the difference between the responses, for each pair

Matched Pairs Example

Research Questions: Which shoe sole (A or B) wears out more quickly?

  • 20 people agree to participate in experiment
  • Subjects are the forty feet (!)
  • 20 pairs (two feet per person)
  • randomize: for each person,
    • pick one foot at random to wear sole A
    • other foot wears sole B

Repeated Measures Design

In a repeated measures design, each subject is measured two or more times under different conditions.

  • We are interested in the effect of the conditions on the measurements.
  • Often we work with the differences between the measurements, for each subject.

Repeated Measures Example


Labels Results

Compute difference between ratings, for each subject:

diff <- labels$jiffrating - 

Numerical look at the differences:

 min Q1 median Q3 max mean   sd  n
  -5  1    2.5  4   8 2.37 2.81 30

Further Concepts

Control Group

If one group in an experiment is not treated in any special way, or is present for comparative purposes, then this group is often called the control group.

Example: In the aspirin-and-heart-disease experiment, the Control Group was the group that took the fake pill.

(Experiment is to see if the aspirin works better than taking nothing at all.)


If the subjects in an experiment do not know which treatment group they are in, then the experiment is said to be single-blinded.

If the people who measure the response variable do not know which groups the subjects are in, then the experiment is also said to be single-blinded.


  • reduce bias due to subjects behaving differently depending on the group they are in;
  • to keep researchers from skewing the results


A placebo is an inert substance given to subjects in the control group.

The placebo resembles the substances given to subjects in the other treatment groups.

Purpose: to make the experiment single-blinded.

Double-Blind Experiment

If neither the subjects nor the people responsible for measuring the response variable know the group assignments of the subjects, then the experiment is said to be double-blinded.

Double-Dummy Design

A double-dummy design is a procedure to blind an experiment, even when the treatments don't resemble each other at all.

Double-Dummy Example

Research Question: Which type of smoking-cessation treatment—nicotine gum or nicotine patch—is more effective in helping people quit smoking?

Suppose 50 smokers agree to participate.

Problem: subject KNOWS whether he/she is getting gum or patch.

Double Dummy Example


  • Members of the Patch Group
    • wear a real nicotine patch, and
    • chew a fake nicotine-flavored gum (dummy)
  • Members of the Gum Group
    • chew real nicotine gum, and
    • wear a fake patch (dummy)

In neither group would subjects be able to tell which group – Patch or Gum – they are in.

The Role of Chance

Observational Study

From the attitudes data:

Reseach Question: Is length of sentence related to choice of major?

Pattern in the Data

      .group  mean    sd
1 humanities 25.26 15.97
2   math.sci 30.49 15.65
3 25.02 15.05
4 social.sci 25.49 14.67

Statistical Significance?

  • Does the pattern we see in the data provide strong evidence that in the GC population math/science majors are harder on crime, OR
  • could the pattern reasonably be ascribed solely to chance variation in the random sampling from the population?

(The question is about chance in the gathering of the subjects!)


From the attitudes data:

Reseach Question: Is length of sentence affected by suggested race of the defendant?

Pattern in the Data

  .group  mean    sd
1  black 27.77 14.95
2  white 25.85 15.76

Statistical Significance?

  • Does the pattern we see in the data provide strong evidence that in the suggested race of the defendant makes a difference in the length of sentence recommended, OR
  • could the pattern reasonably be ascribed solely to chance variation in the random assignment of forms to the subjects?

(The question is about chance in the assignment of subjects to treatment groups!)

Either Way ...

… patterns in the data depend, at least in part, on chance.

So we have to step back and learn more about how chance processes work. This is the subject of probability.