Understanding and Using Statistical Research Methodologies in

Medical Education Programs:

A Primer for Medical Students and Residents

©

Understanding Statistical Research Methodologies in Medical Education Programs:

A Primer for Medical Students and Residents

Introduction

1. Statement of Purpose:

This Statistical Module is designed for residents and students, who are not epidemiologist, and those who do not wish to use extensive epidemiology statistics in their research. This module will allow the user to understand basic statistics used in medical research, and understand their use and concept when reading medical and scientific journals.

This module establishes a framework for understanding the essential principles and practices of understanding the null hypothesis and using various statistics {Mean, Median & Mode; the Chi Square Test; the Pearson Correlation Coefficient – Pearson’s r; the t Test; One-Way Analysis of Variance; Two-Way Analysis of Variance; and Multiple Regression}, with emphasis placed on understanding the role of "qualitative" and "quantitative" evaluation in medical education research.

2. Goal:

Overall Objectives

Residents and Students will:

• Understand what the null hypothesis is and what it means when you reject the null hypothesis or when you fail to reject the null hypothesis.
• Understand what statistically significant means.

• Understand and discern the difference between qualitative and quantitative research using various statistical methods {Mean, Median & Mode, Chi Square, Pearson r, t test, ANOVA one-way and two-way analysis of variance, Multiple Regression and ANCOVA}.
• Understand the history of qualitative and quantitative research in medicine.
• Identify and understand the strengths and limitations of qualitative and quantitative data.
• Analyze a real qualitative data set.
• Identify qualitative assessments that are being used today in medical research.
• Identify and discuss the seminal bodies of qualitative work.
• Identify political limitations of research results (politically correct).
• Understand the function of focus groups and their applications in medical research.
• Understand and be aware of the available software that can be used in providing data results (SAS, SPSS, etc…).

C. Logistical Considerations:

1. Residency year:

PGY 2 and PGY 3

2. Length of program and number of modules:

There are three statistical main modules (each containing from 2 to 4 sections) with a "seat time" of approximately 45 minutes per main module and 30 minutes for the "Foundation of Program Evaluation". The total time for all modules will be approximately 3 hours and 30 minutes (allowing for a 15 minute question & answer period for each main module):

• Understanding and using the Mean, Median & Mode of a data set; Introduction to the Chi Square Test; Calculating the Pearson Correlation Coefficient (Pearson’s r) and it’s meaning.
• Understanding when and where to use and interpret the t Test; ANOVA – One-Way Analysis of Variance; ANOVA – Two-Way Analysis of Variance in medical education research
• Understanding Multiple Regression in medical research. When to use it, how to interpret the results, and what does it mean in your research paper.

Module Summaries and Objectives

Module 1: Basic Statistic Methods and their use

Module One, Basic Statistical Methods and their use, gives residents a simplistic overview in the use of the hypothesis, the mean, median & mode; the chi square test; and the pearson’s r in research, and it defines common concepts in their use.

At the end of this module, the resident and students will be able to:

• Describe and understand the way data is used in research papers in medical and scientific research.
• Compare and understand the results of one research against similar research. Understand chi square and the pearson’s r.
• Define qualitative vs. quantitative research.
• Define "reliability" and "validity" in the use of data and statistical methods.

Module 2: Advanced Statistical Methods: The t Test, and

ANOVA

Module two, Advanced Statistical Methods: The t Test and ANOVA, provides residents and students with a simplistic overview and understanding in using and interpreting advanced statistical methods which are used in medical and scientific journals. Residents and students will learn tools and techniques in using these statistics, in preparation for writing their own research papers in their residency and clinical program, without having to use heavy epidemiology statistics.

At the end of this module, the resident will be able to:

• Describe the purpose and use of the t Test, and ANOVA (One-Way and Two-Way Analysis of Variance).
• Describe the type of data that must be used with these types of test.
• Understand why the user must set alpha (a ) at a level (preferably .05 or less) to test a hypothesis.
• Understand what "statistically significant" means using these statistical methods (t Test, and ANOVA).
• Discuss strategies in the development of their research paper.
• Read and understand research papers that use the t Test and/or ANOVA in analyzing data.

Module 3: Advanced Statistical Methods: Multiple Regression, and

ANCOVA (Analysis of Covariance)

Module three, Advanced Statistical Methods: Multiple Regression, and ANCOVA (Analysis of Covariance), provides students and residents with a simplistic overview and understanding in using and interpreting statistical methods which are used in medical and scientific journals. Residents and students will learn tools and techniques in using and solving statistics, in preparation for writing their own research papers during their residency year, without having to learn epidemiology statistics.

At the end of this module, the resident and student will be able to:

• Describe the purpose and use of Multiple Regression and ANCOVA (Analysis of Covariance).
• Describe the type of data that must be used with these types of test.
• Understand what "statistically significant" means using these statistical methods (Multiple Regression & ANCOVA).
• Discuss strategies in using these statistics in the development of their research paper.
• Read and understand research papers that use the Multiple Regression and/or ANCOVA in analyzing data.

Suppose we drew a random sample each of medical students and residents, administered a self-report measure of medical knowledge, and computed the mean (the most commonly used average) for each group. Furthermore, suppose the mean for the medical students is 63.00 and the mean for residents is 68.00. Where did the five points difference come from? There are three possible explanations, they are:

1. Perhaps the population of residents is truly more knowledgeable about medicine than the population of medical students, and our samples correctly identified the difference. (In fact, our research hypothesis may have been that residents are more knowledgeable about medicine than medical students – which now appears to be supported by the data.)
2. Perhaps there was a bias in procedures. By using random sampling, we have ruled out sampling bias, but other procedures such as measurement may be biased. For example, maybe the residents were contacted during September, when many clinical events (conferences, lectures, etc…) take place and the medical students were contacted during the gloomy month of December when no clinical events (conferences, lectures, etc…) took place. The only way to rule out bias as an explanation is to take physical steps to prevent it. In this case, we would want to make sure that the medical knowledge for both groups was measured in the same way at the same time.
3. Perhaps the populations of residents and medical students are the same but the samples are unrepresentative of their populations because of random sampling errors. For instance, the random draw may have given us a sample of residents who are more knowledgeable, on the average, than their population.

This third explanation has a name – it is the Null Hypothesis. The general form in which it is stated varies from researcher to researcher. Here are three versions, all of which are consistent with each other:

1. Version "A" of the Null Hypothesis:

• The observed difference was created by sampling errors. Note: The term sampling error refers only to random errors, not errors created by a bias.

• There is no true difference between the two groups. (Note: The term true difference refers to the difference we would find in a census of the population, that is, the difference we would find if there were no sampling errors.)

• The true difference between the two groups is zero.

Significance tests determines the probability that the null hypothesis is true. The researcher sets the probability level. Suppose for our example, we use a significance test and find that the probability that the null hypothesis is true is less than 5 in 100. This would be stated as p < .05, where p obviously stands for probability. The researcher should always state the probability level used in their research findings. It can be set anywhere from .001 to > 0, however, we play it safe by setting it to .05 (which has been accepted as the international standard by statisticians). Of course, if the chances that something is true is less than 5 in 100, it’s a good bet that it’s not true. If it’s probably not true, we reject the null hypothesis, leaving us with only the first two explanations that we started with as viable explanations for the difference.

There is no rule of nature that dictates at what probability level the null hypothesis should be rejected. However, conventional wisdom suggests that .05 or less (such as .01 or .001) are reasonable.

When we fail to reject the null hypothesis because the probability is greater than .05, we do just that: We "fail to reject" the null hypothesis and it stays on our list of possible explanations; we never "accept" the null hypothesis as the only explanation. Remember, there are three possible explanations (see above) and failing to reject one of them does not mean that you are accepting it as the only explanation.

An alternative way to say that we have rejected the null hypothesis is to state that the difference is statistically significant. Thus, if we state that a difference is statistically significant at the .05 level (meaning .05 or less), it is equivalent to stating that the null hypothesis has been rejected at that level.

When you read research reported in academic journals, or research papers, you will find that the null hypothesis is seldom stated by researchers, who assume that you know that the sole purpose of a significance test is to test a null hypothesis. Instead, researchers tell you which differences were tested for significance, which significance test they used, and which differences were found to be statistically significant. It is more common to find null hypotheses stated in theses and dissertations since committee members may wish to make sure that the students they are supervising understand the reason they have conducted a significance test.

Exercise

1. How many explanations are there for the differences in medical knowledge between residents and medical students in the example in this topic?
2. What does the null hypothesis say about sampling errors?
3. Does the term sampling error refer to random errors or to bias?
4. The null hypothesis says that the true difference equals what value?
5. What is used to determine the probabilities that null hypothesis are true?
6. For what does p < .05 stand for?
7. Do we reject the null hypothesis when the probability of truth is high or when it is low?
8. What do we do if the probability is greater than .05?
9. What is an alternative way of saying that we have rejected the null hypothesis?
10. Are you more likely to find a null hypothesis stated in a journal article or in a thesis?

1. We all use probabilities in everyday activities to make decisions. For example, before we cross a busy street, we estimate the odds that we will get across the street safely. Briefly describe one other specific use of probability in everyday decision-making.

The most frequently used average is the Mean, which is the balance point in a distribution. Its computation is simple – just sum (add up) the scores and divide by the number of scores. The most common symbol for the mean in academic journals is M (for the mean of a population) or m (for the mean of a sample). The symbol preferred by statisticians is

Because the mean is very frequently used as the average, let’s consider its formal definition, which is the value around which the deviations sum to zero. You can see what this means by considering the scores in Table 1. When we subtract the mean of the scores (which is 4.0) from each of the other scores, we get the deviations (whose symbol is x). If we sum the deviations, we get zero, as shown in Table 1.

Note: If you take any set of scores, compute their mean, and follow the steps in Table 1, the sum of the deviations will always equal zero (it might be slightly off from zero if you use a rounded mean such as using 20.33 as the mean when its precise value is 20.33333333).

Considering the formal definition, you can see why we also informally define the mean as the balance point in a distribution. The positive and negative deviations balance each other out.

A major drawback of the mean is that it is drawn in the direction of extreme scores. Consider the following two sets of scores and their means:

Notice in both sets there are nine scores and the two distributions are very similar except for the scores of 25 and 32 in Set B, which are much higher than the others and, thus, create a skewed distribution. Notice that the two very high scores have greatly pulled up the mean for Set B; in fact, the mean for Set B is more than twice as high as the mean for Set A because of the two high scores.

When a distribution is highly skewed, we use a different average, the Median, which is defined as the middle score. To get an approximate median, put the scores in order from low to high as they are for Sets A and B (above), and then count to the middle. Since there are nine scores in Set A, the median (middle score) is 3 (which is five scores up from the bottom). For Set B, the median (middle score) is 4 (which is five scores up from the bottom), which is more representative of the center of this skewed distribution than the mean , which we noted was 9.44. Thus, one use of the median is to describe the average of skewed distributions. Another use is to describe the average of ordinal data, which you must look at when using NOIR (Nominal, Ordinal, Interval and Ratio) data.

A third average, the Mode, is simply the most frequently occurring score. For Set B, there are more scores of 2 than any other score; thus, 2 is the mode. The mode is sometimes used in informal reporting but is very seldom used in formal reports of research.

Because there is more than one type of average, it is vague to make a statement such as, "The average is 4.11." Rather, we should indicate the specific type of average being reported with statements such as, "The mean is 4.11."

A synonym for the term averages is measures of central tendency. Although the latter is seldom used in reports of scientific research, you may encounter it in other research and statistics.

1. Which average is defined as the most frequently occurring score?
2. Which average is defined as the balance point in a distribution?
3. Which average is defined as the middle score?
4. What is the formal definition of the mean?
5. How is the mean calculated?
6. Should the mean be used for highly skewed distributions? Why and/or why not?
7. Should the median be used for highly skewed distributions? Why and/or why not?
8. What is a synonym for the term averages?

1. Suppose a fellow student gave a report in class and said, "the average was 25.88." For what additional information should you ask? Why?

Suppose we drew at random a sample of 200 members of the American Medical Association and asked them whether they were in favor of a proposed change to their bylaws. The results are shown in Table 1. But do these observed results reflect the true results that we would have obtained if we had questioned the entire population? ( Note: we are using the term true results here to stand for the results of a census of the entire population. The results of a census are true in the sense that they are free of sampling errors. Of course, there may also be measurement errors, which we are not considering here).

Remember that the null hypothesis says that the observed difference was created by random sampling errors; that is, in the population, the true difference is zero. Put another way, the observed difference (n = 120 vs. n = 80) is an illusion created by chance error.

It turns out that after doing some computations, which are beyond the scope of this paper, for the data in Table 1 (above), the results are:

What does this mean for a user of research who sees this in a report? The values of chi square and degrees of freedom (df) were calculated solely to obtain the probability that the null hypothesis is correct. That is, Chi Square and degrees of freedom are not descriptive statistics that you should attempt to interpret. Rather, think of them as sub-steps in the mathematical procedure for obtaining the value of p. thus, the user of research should concentrate on the fact that p is less than .05. As you probably remember, when the probability (p) that the null hypothesis is correct is .05 or less, we reject the null hypothesis. (Remember, when the probability that something is true is less than 5 in 100 – a low probability – conventional wisdom suggests that we should reject it as being true.) Thus, the difference we observe in Table 1 was probably not created by random sampling errors; thus, we can say that the difference is statistically significant at the .05 level.

Up to this point, we have concluded that the difference we observed in the sample was probably not created by sampling errors. So where did the difference come from? Two possibilities remain:

1. Perhaps there was a bias in procedures such as the person asking the question in the survey leading the respondents by talking enthusiastically about the proposed change in the bylaws. If we are convinced that adequate measures were taken to prevent procedural bias, we are left with only the next possibility as a viable explanation.
2. perhaps the population of physicians is, in fact, in favor of the proposed change, and this fact is correctly identified by studying the random sample.

Now let’s consider some results from a survey in which the null hypothesis was not rejected. Table 2 shows the numbers and percentages of subjects in a random sample from a population of Resident Program Directors who prefer each of three methods for teaching statistics.

In table 2, there are three differences (30 for A versus 27 for B, 30 for A versus 22 for C, and 27 for B versus 22 for C). The null hypothesis says that this set of differences was created by random sampling errors; in other words, it says that there is no true difference in the population; we have observed a difference only because of sampling errors. The results of the Chi Square test for the data in Table 2 are:

X² = 1.214, df = 2, p >.05

Using the decision rule that p must be equal to or less than .05 to reject the null hypothesis, we fail to reject the null hypothesis, which is called a statistically insignificant result. In other words, the null hypothesis must remain on our list as a viable explanation for the set of differences we observed by studying a sample.

In this topic, we have considered the use of chi square in a univariate analysis, in which we classify each subject in only one way (such as which candidate each prefers). Chi square can also be useful in bivariate analysis, in which we classify each subject in two ways (such as which candidate each prefers and the gender of each) in order to examine a relationship between the two.

1. When we study a sample, are the results called the true results or the observed results?
2. According to the null hypothesis, what created the difference in Table 1 in this topic?
3. What is the name of the test of the null hypothesis used when we are analyzing frequencies?
4. As a consumer (user) of research, should you try to interpret the value of df?
5. What is the symbol for probability?
6. If you read that a chi square test of a difference yielded a p of less than 5 in 100, what should you conclude about the null hypothesis on the basis of conventional wisdom?
7. Does p < .05 or p > .05 usually lead a researcher to declare a difference to be statistically significant?
8. If we fail to reject a null hypothesis, is the difference in question statistically significant?
9. If we have a statistically insignificant result, does the null hypotheses remain on our list of viable hypotheses?

1. Briefly describe a hypothetical study in which it would be appropriate to conduct a chi square test for univariate data.

When we wish to examine the relationship between two quantitative sets of scores (at the interval or ratio levels), we compute a correlation coefficient. The most widely used coefficient is the Pearson Product-Moment Correlation Coefficient, who symbol is r. It is usually called simply Pearson’s r.

Consider the scores in Table 1 (below). The resident test scores places subjects in roughly the order as the ratings by supervisors. In other words, those who had high clinical test scores (such as Jan and Joe) tended to have high supervisors’ ratings and those who had low test scores (such as Jake and John) tended to have low supervisors’ ratings. This illustrates what we mean by a direct relationship (also called a positive relationship).

Note the relationships in Table 1. They are not perfect. For example, although Joe has a higher clinical test score than Jane, Jane has a higher supervisor’s rating than Joe. If the relationship were perfect, the value of the Pearson r would be 1.00. Being less than perfect, it’s actual value is .89. As you can see in Figure 1 (below), this value indicates a strong, direct relationship. Note: the value of the Pearson r is always between a negative –1 and a positive +1.

The relationships in Table 1 and 2 are strong but, in each case, there are exceptions, which make the Pearson rs less than 1.00 and –1.00. As the number and size of the exception increases, the values of the Pearson r become closer to 0.00. Therefore, a value of 0.00 indicates the complete absence of a relationship (See Figure 1).

It is important to note that a Pearson r is not a proportion and cannot be multiplied by 100 to get a percentage. For instance. a Pearson r of .50 does not correspond to 50% of anything (See Figure 1). To think about correlation in terms of percentages, we must convert Pearson rs to another statistic, the coefficient of determination, whose symbol is r², which indicates how to compute it –simply square r. Thus, for an r of .50, r² equals .25. If we multiply .25 by 100, we get 25%. What does this mean? Simply this: A Pearson r of .50 is 25% better than a Pearson r of 0.00. Table 3 shows selected values of r, r², and the percentages you should think about when interpreting an r.

1. Name two variables between which you would expect to get strong, positive value of r.
2. Name two variables between which you would expect to get a strong, negative value of r.

Suppose we have a research hypothesis that says medical "research investigators who take a short course on the causes of HIV will be less fearful of the disease than research investigators who have not taken the course," and test it by conducting an experiment in which a random sample of research investigators are assigned to take the course and another random sample are designated as the control group (note: random sampling is preferred, because it precludes any bias in the assignment of subjects to the groups and because we can test for the effect of random errors with significance test; we cannot test for the effects of bias).

Let’s suppose that at the end of the experiment the experimental group gets a mean of 16.61 on a fear of HIV scale and the control group gets a mean of 29.67 (where the higher the score, the greater the fear of HIV). These means support our research hypothesis. But can we be certain that our research hypothesis is correct? If you’ve been reading various topics on statistics, you already know that the answer is "no" because of the Null Hypothesis, which says that there is no true difference between the means; that is, the difference was created merely by the chance errors created by random sampling (these errors are known as sampling errors). Put another more simple way, unrepresentative groups may have been assigned to the two conditions quite at random.

The t test is often used to test the null hypothesis regarding the observed difference between two means (to test the null hypothesis between two medians, the median test is used; it is a specialized form of chi square test). For the example, we are considering, a series of computations (which are beyond the scope of this paper) would be performed to obtain a value of t (which, in this case, is 5.38) and a value of degrees of freedom (which, in this case, is df = 179). These values are not of any special interest to us except that they are used to get the probability (p) that the null hypothesis is true. In this particular case, p is less than .05. Thus, in a research report, you may read a statement such as this:

The term statistically significant indicates that the null hypothesis has been rejected. You will recall that when the probability that the null hypothesis is true is .05 or less (such as .01 or .001), we reject the null hypothesis. When something is unlikely to be true, because it has a low probability of being true, we reject it.

Having rejected the null hypothesis, we are in a position to assert that our research hypothesis probably is true (assuming no procedural bias was allowed to affect the results, such as testing the control group immediately after a major news story on a celebrity person with AIDS, while testing the experimental group at an earlier time).

What leads a t test to give us a low probability? Three things:

1. Sample size. The larger the sample, the less likely that an observed difference is due to sampling errors. Large samples provide more precise information. Thus, when the sample is large, we are more likely to reject the null hypothesis than when the sample is small.
2. The size of the difference between means. The larger the difference, the less likely that the difference is due to sampling errors. Thus, when the difference between the means is large, we are more likely to reject the null hypothesis than when the difference is small.
3. The amount of variation in the population. When a population is very heterogeneous (has much variability) there is more potential for sampling error. Thus, when there is little variation (as indicated by the standard deviations of the sample), we are more likely to reject the null hypothesis than when there is much variation.

A special type of t test is also applied to correlation coefficients. Suppose we drew a random sample of 50 medical students and correlated their hand size with their GPAs and got an r of .19. The null hypothesis says that the true correlation in the population is 0.00 - that we got .19 merely as the result of sampling errors. For this example the t test indicates that p > .05. Since the probability that the null hypothesis is true is greater than 5 in 100, we do not reject the null hypothesis; we have a statistically insignificant correlation coefficient. In other words, for n = 50, an r of .19 is not significantly different from an r of 0.00. When reporting the results of the t test for the significance of a correlation coefficient, it is better not to mention the value of t. Rather, it is better to indicate only whether or not the correlation is significant at a given probability level.

1. What does the null hypothesis say about the difference between two sample means?
2. Is the value of t usually of any special interest to consumers of research?
3. Suppose you read that for the difference between two means, t = 2.00, df = 20, p>.05. Using conventional standards, should you conclude that the null hypothesis should be rejected?
4. Suppose you read that for the difference between two means, t = 2.859, df = 40, p<.05. Using conventional standards, should you conclude that the null hypothesis should be rejected?
5. Based on the information in question 4, should you conclude that the difference between the means is statistically significant?
6. When we use a large sample are we more or less likely to reject the null hypothesis than when we us a small sample?
7. When the size of the difference between means is large are we more or less likely to reject the null hypothesis than when the size of the difference is small?
8. If we read that for a sample of 92 subjects, r = .41, p<.001, should we reject the null hypothesis?
9. Is the value of r in question 8 statistically significant?

1. Of the three things that lead to a low probability, which one is most directly under the control of a researcher?

In using the t test, you learned it was used to test the null hypothesis for the observed difference between two sample means. An alternative test for this problem is the analysis of variance (often called ANOVA), sometimes called the F test.

Instead of t, it yields a statistic called F, as well as degrees of freedom (df), sum of squares, mean square, and a p value, which indicates the probability that the null hypothesis is correct. As with the t test, the only value of interest to the typical user of research is the value of p. By convention, when p equals .05 or less (such as .01 or .001), we reject the null hypothesis and declare the result to be statistically significant.

Because the t test and ANOVA are based on the same theory and assumptions, when we compare two means, both tests yield exactly the same value of p and, hence, lead to the same conclusion regarding significance. So for two means, both tests are equivalent and either test can be used to get the same results. However, that is where the comparison stops! Note, that a single t test can compare only two means, but a single ANOVA can compare a number of means, which is a great advantage.

Suppose, for example, as a medical researcher, you tested three drugs, on a population, to treat depression in an experiment and obtained the following means and standard deviations.

Of depression scores for three drugs

Drug A                      Drug B                   Drug C

M = 6.00                  M = 5.50                M = 2.33

Since the higher the score the greater the depression, inspection of the means shows that there are three observed differences:

1. Drug C is superior to Drug A.
2. Drug C is superior to Drug B.
3. Drug B is superior to Drug A.

Table 2 ANOVA for data in Table 1