Glossary

97 terms used across the lessons, in plain English.

A

A/A test: A test that splits traffic between two identical versions to check that the testing setup works. Most A/A results should come back inconclusive.
A/B testing: A method of comparing two versions of a page or element by showing each to a random half of visitors and measuring which one performs better on a defined metric.
alpha spending: A method for controlling the error rate when you plan to check a test at several points before the full sample is collected. Instead of using the full 5% significance budget at the end, you spend a small portion at each interim look so the total stays within bounds.
always-valid inference: A sequential testing approach that keeps the false positive rate controlled however often you check the data, even continuously. It uses confidence sequences in place of traditional confidence intervals, so you can stop at any point without paying a penalty.
Anchoring: The tendency for the first number someone sees to shape how they judge the numbers that follow. On a pricing page, the first plan shown changes how the others look.

B

Bait advertising: Advertising a product at a price you have reason to believe you can't supply in reasonable quantities, without saying so. It is banned outright in the UK under the Digital Markets, Competition and Consumers Act 2024.
Business cycle: The usual length of time it takes a customer to go from first visit to purchase, along with the weekly and monthly patterns in when they buy.

C

Cart abandonment rate: The share of people who add something to a basket but leave without buying. Many of them were only browsing, so the figure overstates what checkout changes can win back.
Chi-square test: A statistical test that checks whether observed counts differ from expected ones by more than chance would explain. It is used to check a test's traffic split.
Choice overload: The idea that offering too many options makes people less likely to choose at all. The evidence is mixed: a large meta-analysis found an average effect close to zero, with a lot of variation between studies.
Client-side testing: Testing where a script in the visitor's browser changes the page after it arrives. It is quick to set up but can slow the page and cause flicker.
Cluster randomisation: Assigning whole groups that mostly interact with each other to control or variant together, to reduce interference between them.
Confidence interval: The range of effects that fit the data from a test, shown around the headline lift. A wide interval that includes zero means the test hasn't ruled out no effect at all.
Confidence level: The complement of the significance level. A 95% confidence level and a 5% significance level describe the same threshold from opposite directions. Raising it to 99% means you need stronger evidence, and usually more traffic, before calling a result, which is worth it when a wrong call would be expensive.
Confirmation bias: The tendency to notice and remember what agrees with what you already believe. In research it shows up as listening out for the quote that supports your theory and skating past the ones that don't.
Control: The unchanged, existing version of a page or experience in a test. Everything else gets measured against it, and holding it steady is how outside variables are controlled for.
Convenience sampling: Recruiting whoever is easy to reach, such as colleagues or friends, in place of real customers. It is quick, and the people you reach this way rarely behave like the people who buy from you.
Conversion rate: The percentage of visitors to a website who complete a desired action, such as making a purchase, signing up for a newsletter, or filling out a contact form.
CRO: Conversion rate optimisation. The practice of improving the percentage of website visitors who complete a desired action, such as making a purchase or signing up, by testing and refining the user experience.
CUPED: A method some testing tools use to reduce noise by adjusting results with how visitors behaved before the test, which lowers the sample size needed.
Customer journey map: A picture of the steps, channels, thoughts and feelings a customer goes through to reach a goal. It is only as reliable as the evidence behind each step.

D

Dark pattern: A user interface designed to trick or manipulate users into doing something they did not intend, such as signing up for a subscription, sharing data, or making a purchase.
Decoy effect: When adding an option that is clearly worse than one of the others shifts choice towards the option it is worse than. On pricing pages it often shows up as a tier designed mainly to make another look better value.
Default effect: The tendency for people to stick with whatever option is already selected. Defaults change outcomes a lot, which is why pre-ticked paid extras are not allowed in the UK or EU.
Double blind: A method, common in medicine and rare in business, where whoever analyses and calls a test result does not know which variant is which. It removes the analyst's own bias from the read, at the cost of extra resourcing most experimentation programmes are not willing to pay for.
Drip pricing: Showing a low headline price and adding unavoidable fees as the customer moves through the purchase. The UK's Competition and Markets Authority treats it as unlawful under current consumer protection law.

E

expected loss: The amount of potential upside you would sacrifice by picking the wrong variant. In a Bayesian A/B test, it combines the probability that each variant could be worse and by how much, giving a single number to weigh against the cost of shipping.

F

Fake review: A review that does not reflect a real customer's experience, or one that hides a connection to the business or a reward for writing it. Fake reviews are now banned in the UK and the US.
false positive rate: The proportion of tests that declare a real effect when none exists. In a well-run frequentist test set at 5% significance, you would expect around 5 false positives for every 100 tests where there is no real difference. Peeking and optional stopping push this number up sharply.
False urgency: A deadline or time pressure that is not real, such as a countdown timer that resets or a sale that never ends. In the UK, falsely stating that a product is only available for a limited time is banned outright.
Falsifiable: Capable of being proven wrong by a real result. A falsifiable hypothesis names an outcome that would count as evidence against it. If no possible result could contradict it, it can't be tested, however neatly it is written.
Feature flag: A switch in a product's code that controls who sees a feature, so a change can be released to a share of users, tested and rolled back without a new release.
Five-second test: A quick research method where someone sees a page for a few seconds and is then asked what they remember and what they think it offers. It is good for catching basic misunderstanding, not for choosing between close alternatives.
Flicker: When visitors briefly see the original page before a testing tool swaps in the variant. It means the variant group sees the change plus a jumpy page.
Friction: Anything that slows, confuses, or stops a user from completing what they came to do on a website, such as unnecessary form fields, slow pages, menus that are hard to follow, or being made to create an account.
Funnel: The steps a visitor passes through toward a goal, measured for where people drop out. It tells you where to look for a problem, and you need other research to find out what the problem is.

G

Geo experiment: A test where whole geographic regions are randomly assigned to see a campaign or change or not, and the regions are compared.
Guardrail metric: A metric set before a test runs to catch harm the primary metric would not show, such as complaints, cancellations, return rate, brand and lifetime value. A result only counts as a win if the guardrails hold too.
Guest checkout: Letting someone buy without creating an account first. An account can still be offered once the order is complete.

H

Heuristic evaluation: A systematic review of a product against established usability principles. It is a quick first pass, and what it produces is expert opinion, which still needs evidence behind it.
Holdout group: A small share of visitors kept on the old experience for a set period while winning changes ship to everyone else. Comparing the two measures the combined effect of a programme directly.
Hypothesis: A testable claim about behaviour, built as: if we make this change, then this metric will move, because of this mechanism. A well-written one could be proven wrong by a real result.

I

Information scent: The cues people use to judge whether a link or page will give them what they are looking for. Weak scent makes people give up on a page or link that might have helped them.
Inline validation: Checking each form field as the user fills it in, instead of only when they submit the form. Done too early, it flags errors before the person has finished typing.
Interaction effect: When the effect of one change depends on another change running at the same time. Two experiments interact if their combined result differs from the sum of their separate effects. Usually small unless both touch the same element or funnel step.
Interference: When the treatment given to one group changes the results of another, for example when a discount for some customers uses up stock the control group would have bought. It can make a change look better or worse than it is.

J

Jobs to be done: A way of understanding customers by the progress they are trying to make in a particular situation, as opposed to who they are. People are said to hire a product to do a job and to fire it when it stops doing that job well.

L

Landing page: The first page a visitor reaches from a campaign, advert, email or search result. It works best when it has one job and keeps the promise that sent the visitor there.
Leading question: A question phrased to push the answer one way, such as "don't you find the checkout slow?". People tend to agree with it, so you learn very little from the answer.
Learning library: A single, searchable record of every test a programme has run, with its hypothesis, design, result, decision and what it taught you about the customer.
Lift: The measured difference in a metric between a variant and its control, in this sample, over this period. On its own it tells you nothing about whether the difference will hold or whether the result can be trusted.

M

Message match: How closely a landing page repeats the promise of the advert, email or link that brought the visitor there. When the first screen doesn't reflect what was promised, people tend to assume they are in the wrong place and leave.
Message testing: Showing several versions of a core proposition to people from your target audience to find out what each one means to them and which reason to buy matters most. Finalists are then usually A/B tested live.
minimum detectable effect: The smallest improvement or change you want a test to be able to detect. It directly affects the required sample size: the smaller the effect you care about, the more visitors you need. Set it unrealistically small and a test can end up needing more traffic than you will ever get.
Multiple comparisons: Running many tests, or checking many metrics, raises the chance that one crosses the significance line by luck alone. The more comparisons you make, the more false positives you should expect unless you correct for it.
Multivariate test: A test that varies several elements together and measures every combination, so you can see both their individual effects and how they interact. It needs more traffic than a simple A/B.

N

Novelty effect: A short-lived change in behaviour because something is new, which fades as people get used to it. It can make an early result look bigger than the lasting effect.
null hypothesis: The default assumption in a frequentist test, usually that there is no real difference between control and variant. The test measures how surprising the observed data would be if this assumption were true.

O

Open funnel: A funnel that counts people at each step however they arrived, including those who skipped the earlier steps. A closed funnel only counts people who started at the first step.
optional stopping: The practice of checking a test repeatedly and stopping the moment the result looks significant. It inflates the false positive rate far beyond the nominal 5%, because random noise gets many chances to cross the line. Also called peeking.
Overall evaluation criterion: The single metric, sometimes a weighted mix of a few, that an organisation agrees all its experiments are judged against. It should predict long-term value rather than measure short-term activity such as clicks.

P

p-value: The probability of seeing data at least as extreme as what you observed, assuming the null hypothesis is true. It is not the probability that the variant is better. A p-value of 0.04 does not mean a 96% chance the variant wins, and that is the most common misunderstanding in A/B testing.
Persona: A description of a type of user, meant to represent a real group of customers. It is useful when built from research and misleading when assembled from the team's assumptions.
Personalisation: Showing different groups of visitors different experiences based on what you know about them, such as returning customers seeing a different homepage from new ones.
posterior: In Bayesian statistics, the updated belief about an effect after combining the prior assumption with observed data. It is a full distribution, so you can read probabilities straight off it, such as "the chance B beats A is 96%".
Pre-ticked box: A checkbox that is already selected when the page loads, so the customer has to notice it and untick it to opt out. Under UK law it does not count as consent to extra charges or to marketing.
Primacy effect: When a change performs badly at first because people are used to the old version, and the effect improves as they adapt.
Primary metric: The single metric chosen before a test launches to decide whether it worked. Other metrics are guardrails or background.
prior: Your assumption about an effect before seeing new data, expressed as a probability distribution. A weakly informative prior nudges the result toward realism, because large effects are rare. A prior tuned too strongly can decide the answer before the data has a say, which is the shadow version.
Prioritisation framework: A consistent scoring model, such as ICE or PIE, used to rank test ideas in the open, so the order reflects evidence and value and the loudest voice in the room doesn't set it.

Q

Qualitative research: Research that explores the why and the how through open observation, interviews and watching people use a product. It is good at finding problems and the reasons behind them, and poor at telling you how many people they affect.
Quantitative research: Research that measures how many and how often, through analytics, surveys and controlled tests. It sizes a problem but rarely explains it.

R

Randomisation unit: The thing you assign to control or variant, such as a cookie, an account, a store or a region. It should match the level at which people experience the change.

S

Sample ratio mismatch: When control and variant receive noticeably different numbers of visitors from the split you set. It usually means something in the test is broken, and the result can't be trusted until someone explains it.
Scarcity claim: A message saying a product is in short supply, such as "only 3 left". It is useful when it reflects real stock and misleading when it does not.
Screener: A short set of questions used to confirm a participant fits a study's criteria before they take part, so the wrong people are turned away before you spend time with them.
Secondary metric: A metric reported alongside the primary metric to explain why a result happened, without being used to declare a winner.
Segment: A group of visitors or customers who share a trait, such as device, traffic source or whether they have visited before. Reading results segment by segment shows problems that an overall average hides.
Selection bias: When the people in your data differ in a systematic way from the people you want to understand, because of how they ended up in it. Customers who complain or write reviews are an example.
Self-report: What someone says they want or would do. It often differs from what they are seen to do, so surveys are useful for generating hypotheses and weak at validating them.
Server-side testing: Testing where the version a visitor gets is decided before the page or app screen is sent, usually through feature flags in the product's own code.
Session recording: A replay of an anonymised visitor's clicks, scrolling and typing on a page. It is used to see where people hesitate or get stuck.
significance level: The threshold below which a p-value is considered statistically significant, conventionally 5% (0.05). It means you accept a 5% chance of declaring a difference when none exists. Where you set it is a business decision about how much risk you can live with.
Simpson's paradox: When a pattern that holds inside every group disappears or reverses once the groups are added together. It happens when the mix of people across the groups changes, so a blended rate can fall while every group improves.
statistical power: The probability that a test will correctly detect a real effect of a given size. Usually set at 80%, meaning there is a 20% chance the test misses the effect entirely. Power depends on sample size, effect size, and the significance threshold.
Switch interview: An interview with someone who recently started or stopped using a product, which reconstructs the timeline of their decision from the first thought to the purchase. It looks for what pushed them, what pulled them, and what nearly stopped them.
Switchback test: A test where a whole region or market alternates between the control and treatment versions over time, used when individual customers can't be separated cleanly.

T

Thematic analysis: Coding qualitative data into labels and grouping them, so themes come out of the evidence instead of being decided in advance.
Think aloud: Asking someone to say out loud what they are doing and thinking as they attempt a task, so you can hear where they get stuck and why.
Triangulation: Checking a finding with more than one research method or data source, so the blind spots of one are covered by another. The more a decision costs, the more cross-checking it deserves.
Twyman's law: The rule of thumb that any figure that looks interesting or different is usually wrong. It is a reason to check surprising results more carefully before acting on them.

U

Usability testing: Watching real people attempt real tasks on a product to find where they stall or go wrong. You learn from what they do more than from what they tell you.
User interview: A one-to-one conversation that explores a person's past behaviour and problems. Its value depends on asking neutral questions about what already happened.

V

Value proposition: The reason a customer should choose you over the alternatives, including doing nothing, stated in terms the customer cares about. It should be specific enough that a competitor could not use it unchanged.
Variant: The version being tested against the control, sometimes called the treatment. It is a distinct, deliberate change made to test a specific hypothesis.

W

Warehouse-native testing: A testing setup that calculates results from data already held in the business's own data warehouse, rather than from events the testing tool collects itself.
Winner's curse: The tendency for tests chosen as winners to overstate their true effect, because part of the reason they came out high is luck. Adding up the measured lifts of launched winners therefore overstates their combined impact.

Shoshin is written by Another Web is Possible. Lessons are licensed CC BY-SA 4.0.Our promise.