Confounding Variables in Quantitative Studies - NN/G

Introducing NN/G Membership | Access all self-paced courses for $249 per year | Join by Jan. 26, 2027 |
Find the training option that fits your learning style.
September 24, 2023 2023-09-24
Quantitative studies (such as surveys, quantitative usability testing, and analytics experiments) can offer valuable insights for product development. However, it is crucial to carefully design quantitative studies to ensure accuracy and reliability.
UX teams often rely on quantitative research to help them make important decisions. Basing those decisions on faulty data can lead to serious design mistakes and misallocated resources.
(Quantitative UX research involves lots of terminology and jargon, so consider referring to this quantitative-research glossary for quick definitions.)
Definition: A confounding variable is an unmeasured variable that may unintentionally affect the outcome of a research study.
Typically, UX researchers designate an independent variable and one or more dependent variables for their quantitative studies.
Independent variables are study conditions that are manipulated by researchers. For example, a team might run a quantitative study comparing two different versions of a design. In this case, the design change is the independent variable.
Dependent variables are outcomes that are measured from a quantitative study. Researchers typically expect the values of dependent variables to change based on their independent variables. For example, researchers might measure user satisfaction to see which version of a design leads to higher satisfaction ratings. Put another way, UX researchers typically expect changes to an independent variable to influence dependent variables.
A confounding variable can affect both the independent variable (the conditions you change) and the dependent variable (the outcomes you measure), causing unexpected results in the study.
For example, a confounding variable could:
A research team ran a within-subjects quantitative usability study to test a design change, which means that each participant would test both designs. (If the study were a between-subjects design, each participant would be asked to interact with only one design.)
Design A was tested in the morning with a group of participants, followed by a lunch break. The participants returned to test design B in the afternoon.
After analyzing the data, the team found that design B had a higher task-completion time than design A — in other words, participants completed tasks more slowly with design B in the afternoon than with design A in the morning.
Lower task-completion times are usually associated with better usability, but is that the case in this situation? Does design A realistically have better usability than design B?
Several confounding variables were present in this study; these variables could muddy the reliability of the results:
Some of these confounding variables may add contradictory and conflicting effects: In the afternoon, participants might do better due to experience but could also perform worse due to a post-lunch slump. Confounding variables can make it difficult to predict the result of a study.
To avoid the time-of-day and learning effects introduced in this study, participants should be randomly assigned to either design A or design B in the morning, with the other design in the afternoon. Randomizing the order in which participants are exposed to the study conditions will reduce the influence of time of day, depleted energy, experience, or hunger.
The age of participants can affect many UX-related variables, including satisfaction, time on task, task success rate, and reading ability.
Recruit a representative sample of participant ages for your study. Randomize your participants across the conditions of your study, so that ages of participants are equally distributed across conditions. Record the ages of your participants for later analysis, if needed.
Participants may behave differently depending on the current season. Consumers’ habits drastically shift around holidays (Chinese New Year, American Winter Holidays, etc.). Comparing a Q4 usability study to a Q1 study may be affected by confounding variables.
If you are conducting a study over long periods of time or comparing results from different studies, consider the time of year at which the studies were completed. Ideally, compare data collected over similar time periods.
The coronavirus pandemic has changed how people interact with products; data collected before and after the pandemic may be affected by the pandemic.
Exclude date ranges during the most acute period of the pandemic by identifying outlier data.
Competitive market shifts
If a competitor has a large sale or introduces an entirely new product right before a design change, user behavior may be affected by the competitor’s actions, preventing you from assessing the effect of your design change.
Structure your studies to avoid major market disruptions.
Existing product experience
When comparing metrics like task-completion time for two different products, participants’ prior experience with the products can skew the results into one direction or another..
Recruit a representative sample of participant experience levels or exclude participants with extensive experience with your products (if applicable).
Existing product opinions
Users may have preexisting viewpoints about the products being tested, influencing usability or survey outcomes. When comparing user satisfaction with two products, prior opinions about the two products could bias the results.
During recruitment, screen for extreme positive or negative product sentiments and either control for or exclude those participants.
A quantitative study can be an investment of significant time and money. You must have confidence that your study will be reliable, and its results will be valid. Studies should be constructed such that they could be repeated, with an expectation of the same result.
When a research st