Reading Newsletter Metrics Correctly: Opens, Clicks, Replies

Why do the figures in the sending tool say so little?

After each issue of a newsletter, the sending tool shows an overview: delivery rate, open rate, click rate, unsubscribes, often compared with an industry average. The figures look precise and are gladly shown in meetings. Yet they rarely answer the question an owner cares about: does the newsletter bring conversations and orders?

The question this article sets out to answer: Which newsletter metrics show something a small business can base decisions on, and which are misleading? To answer it, we first clarify the terms and why opens can barely be measured anymore. Then I explain a model from economics for why metrics lose their value as soon as they become targets, and look at a field experiment that shows this. At the end there is a way of reading that puts the figures in order.

When a newsletter is worthwhile in B2B at all and how an issue is structured is described in the article B2B newsletters. It also contains the principle this article follows: success is measured by replies, conversations and orders.

An open shows that an email arrived. A reply shows that it worked.

What the individual metrics measure

Newsletter metrics are simple ratios. What matters is what they refer to, because tools calculate differently. The following overview relates all rates to delivered emails unless stated otherwise.

A hard bounce is a permanently undeliverable address, for example because the recipient has left the company. A soft bounce is a temporary problem, such as a full mailbox. Hard bounces must be removed from the list immediately. In B2B they often indicate that a contact person has changed, and are therefore a reason to ask the customer who is now responsible.

Why opens can barely be measured anymore

An open is measured by including a tiny, invisible image in the email, a tracking pixel. When the email program loads this image from the sending tool's server, the email is counted as opened. The method always had gaps: anyone who blocks images opens invisibly. Since 2021, a bigger problem has been added. With Mail Privacy Protection, Apple introduced a feature in its mail programs that loads remote content in the background and hides the IP address. The tracking pixel is loaded regardless of whether a person reads the email. Other services also check incoming emails automatically.

With business customers, there is the additional fact that corporate security systems often check links and images in emails in advance. This can even create clicks that are not clicks. If you see in the sending tool that a recipient clicked all the links in an email within the same second, you are usually looking at a machine.

Legally, measurement is not unrestricted either. Reading opens and clicks via tracking pixels and tagged links is, according to the prevailing view in Germany, an access to the terminal device under Section 25 of the TDDDG, the German law on data protection in telecommunications and digital services, and requires consent that goes beyond consent to the newsletter or explicitly covers tracking. What is permitted without consent is described in the article cookie banners and measurement. This is not legal advice.

The thinking model: Goodhart's law

The second problem with metrics is not technical but human. In the 1970s, the British economist Charles Goodhart observed in monetary policy that statistical relationships break down as soon as they are used for control. In 1997, the anthropologist Marilyn Strathern put the idea into its now familiar form: when a measure becomes a target, it ceases to be a good measure. This is Goodhart's law.

Applied to newsletters: the open rate originally measures whether recipients consider a newsletter worth reading. If it becomes the target, subject lines are optimized for curiosity, urgency or riddles. The rate rises, but it now measures something else, namely how well the subject line arouses curiosity, not how useful the content is. The consequences show up elsewhere: recipients feel deceived, click less, unsubscribe or mark the email as spam.

What this looks like in figures is shown by a field experiment by Defau, Zauner and Sycik (2023). In two randomly split mailings by a software provider, they inserted just one word into the sender line: the title CEO. Everything else stayed the same.

Among prospects, opens rose by almost 10 percent and clicks by 75 percent, but the unsubscribe rate almost tripled. All three differences were statistically significant. In a second test with about 8,300 existing customers, the same directions appeared, but much weaker and without a significant difference. The authors conclude that a title has a stronger effect on people who do not yet know the company. For reading metrics, the more important finding is a different one: anyone who had looked only at opens and clicks would have reported a success and overlooked the unsubscribes.

Two readings: optimizing metrics or interpreting them

The first reading sees metrics as adjusting screws. Sending tools make it easy to compare subject line, sender or send time in A/B tests. Defau and colleagues explicitly recommend such tests as a way to understand a target group better step by step. Sahni, Wheeler and Chintagunta (2018) showed in a large field experiment that just adding the first name to the subject line increased opens by 20 percent and sales leads by 31 percent and reduced unsubscribes by 17 percent. Small changes can therefore have real effects.

The second reading sees metrics as clues that need interpretation. A small B2B business with 400 recipients usually has too few cases for clean tests. With 400 recipients and a click rate of around 9 percent, a few clicks more or less are chance. The article evaluating campaigns describes this error as belief in the law of small numbers: people draw too much from small samples and see differences that do not exist.

For small lists, I consider the second reading correct. Tests are worthwhile for large lists, and then with an eye on all figures, as the field experiment shows. With a list of a few hundred business contacts, reading every reply brings more than improving the open rate by one percentage point.

The three-level reading: technology, interest, conversation

From the model and the findings follows an order that I call the three-level reading. It sorts the figures by what they can show and determines on which level decisions are made.

1. Level 1: Technology

   Delivery rate, bounces, spam complaints. These figures are reliable and show whether the list is maintained. They require action if they worsen, but they say nothing about the content.

2. Level 2: Interest

   Clicks per article and unsubscribes. They show roughly which topics attract and whether the newsletter still fits. They are read over several issues, not per issue. Opens belong here, with caution.

3. Level 3: Conversation

   Replies, sign-ups for appointments, conversations and orders sourced from the newsletter. These figures are small, but they are the only ones that show whether the newsletter fulfils its purpose.

Decisions are made on the third level. The first two explain why the third looks good or bad. The following worked example takes the figures from the article B2B newsletters and shows one quarter.

No sending tool counts replies. They land in the sender's mailbox. That is why a simple rule is needed: every reply to the newsletter is entered into the inquiry list or CRM with the source “newsletter”. What this measurement chain looks like from first contact to order is described in the article measuring marketing success.

When a figure requires action

Metrics are only useful if it is decided in advance what happens with which change. The following overview gives guidance. The thresholds are suggestions for small B2B lists, not proven limits.

Recipients who have not reacted at all for months play a special role. Their number grows in every list. Someone who does not react is not necessarily lost, but they should occasionally be asked whether the newsletter is still wanted. How such processes can be automated is described in the article marketing automation for beginners.

What follows beyond the newsletter

Goodhart's law does not apply only to newsletters. Every figure a team is supposed to reach will be reached sooner or later, and then the question arises whether the goal the figure stood for has been reached too. Honest measurement therefore means choosing the figure closest to the goal, even if it is small, slow and tedious to count. For a newsletter, that means replies and conversations.

The limits of the findings are clear. The field experiment by Defau and colleagues concerned a single software provider and two mailings. The experiment by Sahni and colleagues was carried out with large lists that have little in common with those of a small business. How high the share of automatically generated opens is in a particular list cannot be determined from the outside. Only the direction is certain: the further a figure is from the conversation, the more cautiously it should be read.

Conclusion and next steps

We set out to ask
newsletter metrics support decisions and which are misleading. The answer: open rates mislead most often, because they are technically distorted and, once they become targets, measure only how curious a subject line makes people. Clicks and unsubscribes show roughly over several issues which topics work. Only third-level figures support decisions: replies, conversations and orders. They are small and have to be counted by hand, but they are the only ones that measure what the newsletter is for.

• Check in the sending tool whether open and click tracking is covered by consent.
• Remove hard bounces from recent issues and ask customers for new contact persons.
• Delete the open rate from the monthly report or label it explicitly as unreliable.
• Set a rule: every reply to the newsletter goes into the inquiry list with its source.
• Compare clicks per article over the last six issues and note the two strongest topics.
• Include a real question in the next issue that can be answered in one sentence.

A newsletter with a twenty percent open rate and no replies is a well-delivered silence.