IIDigital MarketingGuide
Testing Landing Pages: What You Can Check With Few Visitors
With 600 visitors a month, you often wait a year for a clean A/B test result. But you will find most of a landing page's flaws in one afternoon with five people.
Key takeaways
- A/B tests need many visitors. Raising the inquiry rate from 1.2 to 1.8 percent requires more than 10,000 visitors; for small landing pages that takes months or years.
- With few visitors, something else can be checked well: whether people understand the page, find what matters and recognize the next step. A handful of test persons is enough for that.
- According to the model by Nielsen and Landauer (1993), five test persons find about four out of five problems on average, but only if the problems are easy to discover. Difficult pages need more.
- The test ladder orders the tests: understand, find, trust, act. Only when all four steps hold is it worth comparing inquiry rates over fixed periods.
Why do A/B tests fail on small landing pages?
Many guides recommend the same procedure for landing pages: build two variants, distribute visitors randomly, keep the better one. This is methodologically sound but presupposes something most small businesses lack: visitors in large numbers. A landing page for a search ad with 600 visitors a month and 18 inquiries produces differences after one month that are almost always chance.
This piece asks: What can be checked reliably on a landing page with few visitors, and with which methods? To answer it, we first clarify why classic tests fail because of visitor numbers. Then I introduce a model from usability research that explains why a few test persons find many problems, and weigh up two views. At the end there is a sequence of checks a small business can carry out itself.
How a landing page is structured is described in the article building a landing page with the single-goal framework of seven building blocks. This article starts after that: when the page is up and you want to know whether it works.
A number tells you that a page does not work. A person tells you why.
How many visitors an A/B test needs
In an A/B test, visitors are randomly distributed across two variants of a page, and it is measured which one brings more inquiries. For a difference not to be due to chance, the test needs a minimum number of visitors per variant. This number depends on how high the inquiry rate is and how large a difference you want to detect. The lower the rate and the smaller the difference, the more visitors are needed.
The article measuring and improving the website inquiry rate works through an example: with a rate of 1.2 percent and a hoped-for increase to 1.8 percent, a test needs a good 5,000 visitors per variant, i.e. more than 10,000 in total. With 2,000 visitors a month, that takes five months. A landing page fed only by one ad often has far fewer visitors.
Worked example
In addition, a lot changes over such a long period: season, ad copy, competition. A result after five months then measures more than the page. Anyone who stops earlier as soon as one variant is ahead falls for chance. The article evaluating campaigns describes this error as belief in the law of small numbers.
The thinking model: how a few test persons find many problems
With few visitors, you cannot measure well which variant brings more inquiries. What you can check well is whether a page contains obstacles: a headline nobody understands, a button nobody finds, a form that puts people off. For such checks, usability research has a well-known model.
Nielsen and Landauer (1993) analyzed eleven studies and showed that the discovery of problems can be described well as a random process. Each test person discovers a particular problem with a certain probability. The second person finds partly the same problems as the first and partly new ones. With each additional person, fewer new problems are added. The share of problems found can therefore be estimated with a simple formula: one minus the probability that all test persons miss a problem. In what follows I call this the discovery model.
The model is the source of the widespread rule that five test persons are enough. Davids, Harvey, Halperin and Chikte (2015) put it in context: the rule assumes that each person discovers a problem with a probability of about 30 percent. Then five people find around 80 percent of the problems on average. If an application is more difficult or the obvious errors have already been fixed, the probability falls and five people are no longer enough. In their own study with 15 doctors testing a learning application, ten people would have been needed to find 80 percent of the serious problems. With five people, they would have found just over half on average, but depending on which people were chosen, as little as a quarter.
Worked example based on the discovery model by Nielsen and Landauer (1993): share = 1 − (1 − probability) to the power of the number of persons, rounded. The probabilities of 30 and 15 percent are assumed.
For a landing page, this is good news with one qualification. A landing page is simpler than a learning application. Gross errors such as an incomprehensible headline strike almost everyone. Subtler errors, such as an objection only buyers have, are found only by those who test the right people.
Numbers or people: two views
The first view considers only numbers reliable. What test persons say is not what real visitors do. Anyone who knows they are being observed reads more carefully and clicks differently. Research has also found an evaluator effect: Hertzum, Molich and Jacobsen (2014) had 19 experienced professionals analyze the same recorded tests with five users. For moderated sessions, each reported on average only a third of the problems that all of them found together. What is found therefore also depends on who is looking.
The second view considers observation the better tool when numbers are small. An inquiry rate shows that something is wrong, but not what. Five people thinking aloud while reading show where they hesitate and why. With few visitors, this is the only information that is fast and reliable enough to justify a change.
I consider both views justified and derive a sequence from them. First you observe in order to find obstacles. Then you measure whether removing the obstacles changes inquiries over a fixed period. The evaluator effect is mitigated by having at least two people analyze the tests and compare their notes.
The test ladder: understand, find, trust, act
For practice, I suggest a fixed sequence, the test ladder. It follows the path a visitor takes on a landing page. Each step has its own simple test procedure, and each presupposes the previous one: anyone who does not understand what it is about does not look for a button.
Step 1: Understand
Five-second test: show the page for five seconds, cover it and ask what it was about and whom the offer is for. Anyone who cannot say has not understood the headline.
Step 2: Find
Click task: ask where the person would click to make an inquiry or see the price. Hesitation or wrong clicks show that the next step is not visible.
Step 3: Trust
Thinking aloud: the person reads the page and says what goes through their mind. Listen for sentences like “I don't believe that” or “What does it cost?”. These are missing evidence and open objections.
Step 4: Act
Have the form filled in up to just before submitting: which field causes hesitation, what information is the person missing, what do they expect afterwards?
All four steps can be checked in a session of about 20 minutes per person. Five sessions fit into one afternoon. Suitable test persons are people who belong to the target group but do not know the page: customers you can ask for a favor, contacts from your network, acquaintances from the industry. Colleagues from your own business are not suitable because they already know the offer.
| Step | Finding in the test | What it means | Change |
|---|---|---|---|
| Understand | Three out of five name the wrong offer | Headline describes the business, not the result | Take over the headline from the ad |
| Find | Two out of five look for the button at the bottom | Next step only visible after a long text | Repeat the button after the first section |
| Trust | Four out of five ask about the price | Missing price range is an objection | Add a price range or price example |
| Act | Everyone hesitates at the budget field | Required field looks like a trap | Make the field optional or delete it |
What you can still measure with real visitors
Even with few visitors, real figures provide clues if read correctly. Three methods are particularly suitable.
- The before-and-after comparison over fixed periods. One change, then wait at least three months and compare with the three months before, and for seasonal business also with the previous year. This is less clean than an A/B test, but with small numbers it is the more honest method.
- The path through the page. How many visitors scroll down to the process section, how many start the form, how many submit it? A drop-off at a particular point shows where the test ladder should start.
- Asking those who inquired. Anyone who has inquired can say what convinced them and what almost held them back. Two calls a month replace any statistics when numbers are small.
- Visitors from the ad
- Scrolled to the process section45% of previous stage
- Form started17% of previous stage
- Inquiry submitted40% of previous stage
Worked example with freely chosen figures, taken from the article on building a landing page. The biggest loss lies between scrolling and the form: this is where the trust step comes in.
How a form should be designed so that nobody drops off on the last stretch is described in the article the inquiry path on the website. Whether ad and page promise the same thing is checked by the three-promises test in the article aligning ads and landing pages.
What follows beyond the individual landing page
The finding can be generalized. Small businesses rarely have the problem of measuring too little, but of measuring the wrong thing: they look for winners in small numbers instead of looking for reasons in a few conversations. Observation is not a substitute for measurement but its precondition. Only those who know why visitors hesitate can make a change whose effect is worth measuring later.
The limits of the findings are clear. The discovery model was developed on software and applications, not on landing pages. The study by Davids and colleagues concerned a complex medical learning application, the opposite of a simple offer page. Test persons behave differently from real visitors, and five industry acquaintances are not a representative sample. What the tests achieve is to make obstacles visible. Whether removing them increases inquiries is shown only by the comparison over time.
What remains and how to start
The guiding question was what can be checked reliably on a landing page with few visitors. The answer: not which variant brings more inquiries, because the visitors are lacking. But whether people understand the page, find what matters, trust it and can take the next step. According to the discovery model, a few test persons are enough for this, provided they are the right ones and more than one person analyzes the results. You then measure the effect of the changes over fixed periods and by asking those who inquired.
- Pick the landing page with the most visitors and note its path over one month.
- Find five test persons from the target group who do not know the page.
- Go through the four steps of the test ladder in 20-minute sessions.
- Analyze the notes in pairs and adopt only findings that affect at least two people.
- Implement the two most important changes and record the date.
- After three months, compare the inquiry rate with the three months before.
If you have few visitors, do not count them before you have watched them.
Frequently asked questions
How many visitors does an A/B test need?
That depends on the inquiry rate and the hoped-for difference. With an inquiry rate of 1.2 percent and an increase to 1.8 percent, more than 10,000 visitors are needed for both variants together. At 2,000 visitors a month, such a test takes about five months.
How do I test a landing page with little traffic?
With tests in which you watch people use the page: five-second test, click tasks, thinking aloud and filling in the form. Afterwards you measure the effect of changes over fixed periods of at least three months.
Are five test persons enough for a usability test?
For a simple page, usually enough to find the gross errors. According to the model by Nielsen and Landauer, five people find about 80 percent of the problems if each problem is easy to discover. For more difficult pages or subtle errors, more people are needed.
What is a five-second test?
You show a test person the page for five seconds, cover it and ask what it was about and whom the offer is for. If the person cannot answer, the top part of the page does not convey its message.
Who is suitable as a test person?
People from the target group who do not know the page, such as customers, network contacts or acquaintances from the industry. Colleagues from your own business are not suitable because they already know the offer and unconsciously fill gaps.
When is a real A/B test worthwhile?
When there are enough visitors and inquiries to detect a difference within a manageable time, and when the gross obstacles have already been removed. For many small B2B landing pages, this is not the case.
Sources
- Nielsen, J.; Landauer, T. K. (1993): A mathematical model of the finding of usability problems. Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems, 206-213.
- Davids, M. R.; Harvey, J.; Halperin, M. L.; Chikte, U. M. E. (2015): Determining the number of participants needed for the usability evaluation of e-learning resources: A Monte Carlo simulation. British Journal of Educational Technology, 46(5), 1051-1055.
- Hertzum, M.; Molich, R.; Jacobsen, N. E. (2014): What you get is what you see: revisiting the evaluator effect in usability tests. Behaviour & Information Technology, 33(2), 143-162.
- Tversky, A.; Kahneman, D. (1971): Belief in the law of small numbers. Psychological Bulletin, 76(2), 105-110.
