We've updated our
Privacy Policy effective December 15. Please read our updated Privacy Policy and tap

Study Guides > Mathematics for the Liberal Arts Corequisite

Sampling Methods and Bias

Introduction

What you’ll learn to do: Examine the methods for sampling and how bias can affect the results

As we mentioned previously, the first thing we should do before conducting a survey is to identify the population that we want to study. In this lesson, we will show you examples of how to identify the population in a study, and determine whether or not the study actually represents the intended population. We will discuss different techniques for random sampling that are intended to ensure a population is well represented in a sample.  

Learning Outcomes

  • Identify methods for obtaining a random sample of the intended population of a study
  • Identify types of sample bias
 

Learning statistical processes

Statistical processes, like other mathematical processes, are technical recipes for accomplishing a goal. Statistical processes, though, involve making decisions that feel subjective at times. It takes practice with many statistical studies to build up a good instinct for the type of process that is best applied in a given situation. Look for clues and action-words in the descriptions below that will help you differentiate between sampling methods and types of sample bias.

Selecting a Population

Suppose we are hired by a politician to determine the amount of support he has among the electorate should he decide to run for another term.  What population should we study? Every person in the district? Not every person is eligible to vote, and regardless of how strongly someone likes or dislikes the candidate, they don't have much to do with him being re-elected if they are not able to vote. What about eligible voters in the district? That might be better, but if someone is eligible to vote but does not register by the deadline, they won't have any say in the election either. What about registered voters? Many people are registered but choose not to vote. What about "likely voters?" This is the criteria used in much political polling, but it is sometimes difficult to define a "likely voter." Is it someone who voted in the last election? In the last general election? In the last presidential election? Should we consider someone who just turned 18 a "likely voter?" They weren't eligible to vote in the past, so how do we judge the likelihood that they will vote in the next election? rows of Lego figurines   In November 1998, former professional wrestler Jesse "The Body" Ventura was elected governor of Minnesota. Up until right before the election, most polls showed he had little chance of winning. There were several contributing factors to the polls not reflecting the actual intent of the electorate:
  • Ventura was running on a third-party ticket and most polling methods are better suited to a two-candidate race.
  • Many respondents to polls may have been embarrassed to tell pollsters that they were planning to vote for a professional wrestler.
  • The mere fact that the polls showed Ventura had little chance of winning might have prompted some people to vote for him in protest to send a message to the major-party candidates.
But one of the major contributing factors was that Ventura recruited a substantial amount of support from young people, particularly college students, who had never voted before and who registered specifically to vote in the gubernatorial election. The polls did not deem these young people likely voters (since in most cases, young people have a lower rate of voter registration and a lower turnout rate for elections) and so the polling samples were subject to sampling bias: they omitted a portion of the electorate that was weighted in favor of the winning candidate.

Sampling bias

A sampling method is biased if not every member of the population has equal likelihood of being in the sample.
So even identifying the population can be a difficult job, but once we have identified the population, how do we choose an appropriate sample? Remember, although we would prefer to survey all members of the population, this is usually impractical unless the population is very small, so we choose a sample. There are many ways to sample a population, but there is one goal we need to keep in mind: we would like the sample to be representative of the population. Returning to our hypothetical job as a political pollster, we would not anticipate very accurate results if we drew all of our samples from among the customers at a Starbucks, nor would we expect that a sample drawn entirely from the membership list of the local Elks club would provide a useful picture of district-wide support for our candidate. One way to ensure that the sample has a reasonable chance of mirroring the population is to employ randomness. Here are several common sampling methods that employ some measure of randomness.

Sampling Methods

  • A simple random sample is one in which every member of the population and any group of members has an equal probability of being chosen.
  • In stratified sampling, a population is divided into a number of subgroups (or strata). Random samples are then taken from each subgroup with sample sizes proportional to the size of the subgroup in the population.
  • In cluster sampling, the population is divided into subgroups (clusters), and a set of subgroups are selected to be in the sample.
  • In systematic sampling, every nth member of the population is selected to be in the sample.

Sampling Method examples

Example 1:  If we could somehow identify all likely voters in the state, put each of their names on a piece of paper, toss the slips into a (very large) hat and draw 1000 slips out of the hat, we would have a simple random sample.
(In practice, computers are better suited for this sort of endeavor than millions of slips of paper and extremely large headgear.)   It is possible, however, that even a simple random sample might end up not being totally representative of the population.  If we repeatedly take samples of 1000 people from among the population of likely voters in the state of Washington, some of these samples might tend to have a slightly higher percentage of Democrats (or Republicans) than does the general population; some samples might include more older people and some samples might include more younger people; etc.  In most cases, this sampling variability is not significant.  To help account for variability, pollsters might instead use a stratified sample as in the next example. Example 2: Suppose that in a particular state, previous data indicated that the electorate was comprised of 39% Democrats, 37% Republicans and 24% independents.  In a sample of 1000 people, they would then expect to get about 390 Democrats, 370 Republicans and 240 independents.  To accomplish this, they could randomly select 390 people from among those voters known to be Democrats, 370 from those known to be Republicans, and 240 from those with no party affiliation.  This would be a stratified sample.   Example 3:  If the college wanted to survey students, since students are already divided into classes, they could randomly select 10 classes and give the survey to all the students in those classes. This would be cluster sampling.   Example 4:  To select a sample using systematic sampling, a pollster might call every 100th name in the phone book.  Systematic sampling is not as random as a simple random sample (if your name is Albert Aardvark and your sister Alexis Aardvark is right after you in the phone book, there is no way you could both end up in the sample) but it can yield acceptable samples.
 
The Worst Way to Sample
Perhaps the worst types of sampling methods are convenience samples and voluntary response samples.  In particular, voluntary response samples are typically not representative of the population because it is often only the individuals with particularly strong opinions who respond.

Convenience sampling and voluntary response sampling

Convenience sampling is the practice of samples chosen by selecting whoever is convenient. Voluntary response sampling is allowing the sample to volunteer.

examples

Example 1: A pollster stands on a street corner and interviews the first 100 people who agree to speak to him. Which sampling method is represented by this scenario?

Answer: This is a convenience sample.

 
Example 2: A website has a survey asking readers to give their opinion on a tax proposal. Which sampling method is represented?

Answer:

This is a self-selected sample, or voluntary response sample, in which respondents volunteer to participate.
Usually voluntary response samples are skewed towards people who have a particularly strong opinion about the subject of the survey or who just have way too much time on their hands and enjoy taking surveys.

  Watch the following video for an overview of all the sampling methods discussed so far.  Note that the video includes a method called "Quota Sampling" that we will omit. https://youtu.be/sk4TDJU7QbY

More Sampling Method Examples

In each case, indicate what sampling method was used.
  1. Every 4th person in the class was selected
  2. A sample was selected to contain 25 men and 35 women
  3. Viewers of a new show are asked to vote on the show’s website
  4. A website randomly selects 50 of their customers to send a satisfaction survey to
  5. To survey voters in a town, a polling company randomly selects 10 city blocks, and interviews everyone who lives on those blocks.

Answer:

  1. Systematic sampling
  2. Stratified sampling
  3. Voluntary response sampling
  4. Simple random sample
  5. Cluster sampling

 

Try It

  [ohm_question]6733[/ohm_question]

Problematic Sampling and Surveying

There are number of ways that a study can be ruined before you even start collecting data. The first we have already explored – sampling or selection bias, which is when the sample is not representative of the population. One example of this is voluntary response bias, which is bias introduced by only collecting data from those who volunteer to participate. This is not the only potential source of bias.

Sources of bias

  • Sampling bias – when the sample is not representative of the population
  • Voluntary response bias – the sampling bias that often occurs when the sample is volunteers
  • Self-interest study – bias that can occur when the researchers have an interest in the outcome
  • Response bias – when the responder gives inaccurate responses for any reason
  • Perceived lack of anonymity – when the responder fears giving an honest answer might negatively affect them
  • Loaded questions – when the question wording influences the responses
  • Non-response bias – when people refusing to participate in the study can influence the validity of the outcome

Sampling Bias examples

Example 1: Consider a recent study which found that chewing gum may raise math grades in teenagers[footnote]Reuters. https://www.reuters.com/article/us-gum-learning/chewing-gum-may-raise-math-grades-in-teens-idUSTRE53L79320090422. Retrieved 4/5/2020[/footnote]. This study was funded by the Wrigley Science Institute, a branch of the Wrigley chewing gum company. Identify the type of sampling bias found in this example.

Answer: This is an example of a self-interest study; one in which the researchers have a vested interest in the outcome of the study. While this does not necessarily ensure that the study was biased, it certainly suggests that we should subject the study to extra scrutiny.

  Example 2: A survey asks people “when was the last time you visited your doctor?” What type of sampling bias might this lead to?

Answer:

This might suffer from response bias, since many people might not remember exactly when they last saw a doctor and give inaccurate responses.
Sources of response bias may be innocent, such as bad memory, or as intentional as pressuring by the pollster. Consider, for example, how many voting initiative petitions people sign without even reading them.

  Example 3: A survey asks participants a question about their interactions with members of other races. Which sampling bias might occur for this survey strategy?

Answer: Here, a perceived lack of anonymity could influence the outcome. The respondent might not want to be perceived as racist even if they are, and give an untruthful answer.

  Example 4: An employer puts out a survey asking their employees if they have a drug abuse problem and need treatment help. Which sampling bias may occur in this scenario?

Answer: Here, answering truthfully might have consequences; responses might not be accurate if the employees do not feel their responses are anonymous or fear retribution from their employer. This survey has the potential for perceived lack of anonymity.

  Example 5: A survey asks “do you support funding research of alternative energy sources to reduce our reliance on high-polluting fossil fuels?” Which sampling bias may result from this survey?

Answer:

This is an example of a loaded or leading question – questions whose wording leads the respondent towards an answer.
Loaded questions can occur intentionally by pollsters with an agenda, or accidentally through poor question wording. Also a concern is question order, where the order of questions changes the results. A psychology researcher provides an example[footnote]Swartz, Norbert. http://www.umich.edu/~newsinfo/MT/01/Fal01/mt6f01.html. Retrieved 3/31/2009[/footnote]:
“My favorite finding is this: we did a study where we asked students, 'How satisfied are you with your life? How often do you have a date?' The two answers were not statistically related - you would conclude that there is no relationship between dating frequency and life satisfaction. But when we reversed the order and asked, 'How often do you have a date? How satisfied are you with your life?' the statistical relationship was a strong one. You would now conclude that there is nothing as important in a student's life as dating frequency.”

  Example 6: A telephone poll asks the question “Do you often have time to relax and read a book?” and 50% of the people called refused to answer the survey. Which sampling bias is represented by this survey?

Answer: It is unlikely that the results will be representative of the entire population. This is an example of non-response bias, introduced by people refusing to participate in a study or dropping out of an experiment. When people refuse to participate, we can no longer be so certain that our sample is representative of the population.

 

More Sampling Bias Examples

In each situation, identify a potential source of bias
  1. A survey asks how many sexual partners a person has had in the last year
  2. A radio station asks readers to phone in their choice in a daily poll.
  3. A substitute teacher wants to know how students in the class did on their last test. The teacher asks the 10 students sitting in the front row to state their latest test score.
  4. High school students are asked if they have consumed alcohol in the last two weeks.
  5. The Beef Council releases a study stating that consuming red meat poses little cardiovascular risk.
  6. A poll asks “Do you support a new transportation tax, or would you prefer to see our public transportation system fall apart?”

Answer:

  1. Response bias
  2. Voluntary response bias
  3. Sampling bias
  4. Perceived lack of anonymity
  5. Self-interest study
  6. Loaded question

Try It

[ohm_question]6727[/ohm_question]
   

Licenses & Attributions

CC licensed content, Original

CC licensed content, Shared previously