Introduction
From time immemorial, email campaigns have been designed, executed, and evaluated based on three core metrics: open rates, click rates, and reply rates, which we have accepted as axioms for a long time.
Seldom do we question if these metrics still make sense given recent advancements in technology that help us both track user activity and block trackers that monitor it.
In addition to these technological advancements, we are witnessing a significant push towards enhanced user privacy, to prevent their data from being used without their consent, albeit on our products.
It is time we question these axioms of open rates, click rates, and reply rates and re-examine the validity of these metrics and our ability to accurately and effectively track them.
Realities of email campaigns today
Let us start with the open rate, which is the subject of this black paper. Open rates have historically been a comfort metric for us because it is the first and easiest metric to move in our favour, or so we think.
Open rate instils a sense of comfort that the email campaign is working, which from a psychological perspective is very important, because it gives us the energy to move forward with wishful thinking.
Click rate helps us narrow the funnel of prospects for subsequent actions on relative more expensive channels like LinkedIn, calls, and ads, based on the assumption that prospects who click links have been warmed up.
Reply rate, which is the most important metric, is also the toughest metric to move which as counter-intuitive as it may sound, it not the actual focus for most email campaigns.
In reality, when we design and execute email campaigns, we are trying to warm up prospects so that when we target them using other channels, the likelihood of getting their attention is higher.
Given the abysmal reply rates recently, most people have adopted the outlook that emails are part of an omnichannel strategy aimed at narrowing the funnel.
Problems with our current approach
Open rate is not an accurate metric anymore. We must question our ability to identify prospects who have opened our emails. The purpose of this black paper is to prove through an experiment that we cannot track open rates accurately.
Open rate is most commonly calculated using email pixels, which we no longer can track accurately, as will be explained in detail in the subsequent sections of this black paper.
Click rate, as a metric cannot be taken at face value either. Most of us know that even SMBs have started adopting IT security systems that screen emails to prevent phishing and trojans.
We either conveniently forget this fact when we run email campaigns or do not connect the dots about the existence of these tools and the click rates that we witness in our email campaigns.
IT security systems tend to scan every email before delivering them to the inbox of recipients. As a part of the scanning process, these systems also open links attached to ensure they are safe.
Unfortunately, tools that we use to run email campaigns may not be able to differentiate clicks from such IT security systems from actual user clicks, thereby inflating the click rate of email campaigns.
Subsequent decisions, including retargeting prospects, basis click rates can be significantly incorrect and ineffective, which we do not take into consideration when we build our outbound lead generation strategies.
Reply rate, the toughest metric to move, is the only unquestionable metric exists today, which may also be inaccurate in the future, when bots start responding to emails instead of humans.
However, this is problem for a later date, as seldom do we find people who have adopted bots to respond on their behalf, in an autonomous way, especially to work emails. We hope that it remains this way.
For now, reply rate is the only accurate metric to evaluate the performance of an email campaign we strongly believe that it must be the only north-star metric or NSM for all email campaigns.
As mentioned above, the focus of this black paper is to prove that we can no longer accurately measure open rate, and to do so using a simple yet deterministic experiment as detailed below.
Separate black papers will be authored about our inability to track click rate and the implications of the same, along with our opinions on effective corrective actions for outbound lead generation strategies.
Technicalities of open rate
Email pixels are the most common method used to track email open rates as mentioned above. A pixel is a small, transparent 1x1 pixel image that is embedded in the HTML code of an email message.
When a recipient opens the email, the pixel is automatically loaded, which sends a signal back to the server, which serves as an indication of the email being opened by the respective recipient.
The more traditional method of tracking open rates is using read receipts, wherein a recipient is explicitly asked for their consent to send an alert to the email sender stating that they have opened the email.
While this is a more reliable method to determine email open rates, there are certain apparent and latent issues with this, as mentioned below:
- Read receipts may not be supported by all email clients
- People may disable the feature even if email clients support it
- Recipients generally tend to avoid sending confirmation to senders
Email pixels overcome these shortcomings of read receipts as they can be loaded on almost every email client and they do not require explicit action from recipients to share information about email open rates.
Problem with email pixels
Gmail tends to preload email pixel which fires the pixel before the email is actually opened, leading to inflated open rates, which is difficult to correct or adjust and therefore misleading.
Microsoft tends to block images, irrespective of their size in emails by default, which prevents email pixels from loading in their email client, thereby blocking information about email opens.
Other challenges include region specific compliance requirements like GDPR, which require user consent before using tracking pixels to gather information about email opens, which pose legal challenges.
As a safeguard, we suspect certain software may not embed email pixels for emails that are being sent to regions like Europe, because user consent is seldom obtained in opt-out emails like lead generation emails.
Impact of these problems
If people who use email pixel truly understand these problems and related shortfalls, they can better manage their expectation regarding reports about email open rates, which is a rare occurrence.
Unfortunately, people who manage email campaigns are so biased and excited by email open rates that their subsequent set of actions are largely based on this metric, which can be significantly incorrect.
We have witnessed these actions based on email open rates by marketing, inside sales, business development and even sales teams:
- LinkedIn messages to prospects who have opened emails
- Cold calls to prospects who have opened multiple emails
- Ads, where possible, to retarget people who have opened emails
- Nurture campaigns for prospects who have opened emails
- Expensive data enrichment for prospects who have opened emails
As a result, significant effort and capital tends to be wasted on prospects who are considered warm, because of incorrect and inaccurate inferences that they opened emails, which is an indication of interest.
An even bigger problem lies in the inference that when a prospect opens an email, it is an expression of interest in the topic that is core to the email, which from a common-sense standpoint, is also incorrect.
Logic dictates that if a prospect is really interested in the email that is sent to them, they will act on it, either by replying to the email or by clicking a link that is in the email or by searching online about the sender.
The only accurate and unquestionable expression of interest is that of a reply from the prospect, which as mentioned earlier in this paper, must be the only north-star metric or NSM that sellers must follow.
Experiment to prove open rate does not work
To prove what we have stated above, about open rates being unreliable, let us formulate an experiment wherein we design and execute an email campaign using a popular email campaign tool.
For the purpose of this experiment, let us consider the following:
The expected outcome from this experiment is that open rate from the Gmail ID may be captured accurately on E-Tool-X but the open rate from the Microsoft ID will not be captured accurately on E-Tool-X.
Step-1: Check pre-campaign status of mailboxes
Three mailboxes are in scope for this campaign, i.e., the seller’s mailbox (growth@lifex.services), recipient’s Gmail inbox (np.lifex@gmail.com) and recipient’s Outlook inbox (marketing@getreplies.ai).
Note that neither of the recipient mailboxes i.e., np.lifex@gmail.com nor marketing@getreplies.ai have an email from the seller’s mailbox address i.e., growth@lifex.services as shown in the screenshots.
Step-2: Configure seller’s mailbox in E-Tool-X
The seller’s mailbox i.e., growth@lifex.services has been added to the list of mailboxes in E-Tool-X and the setup is complete as is shown by the setup progress bar which stands at 100% for this mailbox.
To preserve confidentiality of information that is part of this account, certain elements have been blacked out, including but not limited to email IDs, account name and details of past campaigns.
You can also clearly see that no emails have been sent from using this mailbox today as the daily limit indicators reads 0 out of 50 and deliverability score has no information as no emails were sent.
Step-3: Define a single step campaign on E-Tool-X
The single step campaign will have the subject line as “Test” and the body of the message will read “Experiment to test email open rates”. It will be sent to both the recipient IDs mentioned above.
Step-4: Upload test data and execute campaign
A CSV file named “test_data” was created which had contact details of the recipients as mentioned above. The snapshot of the CSV file is also shown below for your reference.
Step-5: Check delivery of emails to recipients
Check both the Gmail & Microsoft mailboxes to ensure that the emails have been delivered and that they reach the inbox and not spam folder. In case of spam delivery, mark the email as not-spam.
The image above shows that E-Tool-X has delivered the email. Now we must confirm the same using the individual mailboxes and ensure that the email was delivered. The account name of the sender will be blacked out.
The images above show that both, the Gmail and the Microsoft recipient IDs have received the emails. The next step is to open the emails and check if E-Tool-X is able to accurately detect the opens.
Step-6: Open email received in Gmail inbox
We will now open the email from Gmail and then check after five minutes if E-Tool-X is able to detect the email open in Gmail. Our best guess at this point is E-Tool-X will be able to detect this open.
As expected, E-Tool-X is able to detect the open in the Gmail mailbox.
Step-7: Open email received in Microsoft mailbox
We will now repeat step-6 but for the Microsoft mailbox which we know will not render images, even if it is just a pixel, without explicit action from the user, therefore preventing detection of open rate.
Seven minutes later, E-Tool-X is unable to detect the open in the Microsoft mailbox. Rest assured, even a day later, the result will be the same.
Conclusion
The experiment clearly proves that email open rate is not detectable when the recipient has a Microsoft mailbox, but the open rate is detectable when the recipient has a Gmail mailbox.
We are NOT to conclude that if recipients have Microsoft mailboxes, the open rates can never be determined, NOR are we to conclude that that if the recipients have Gmail inboxes, open rates can be determined.
The purpose of this black paper was to prove that open rates are unreliable today given the shortcomings of the pixel approach to track if emails are opened as explained earlier, which we have successfully done.
Depending on firewall and mailbox settings of an enterprise, even Gmail mailboxes may not render pixels and Microsoft mailboxes may render pixels by default. Either way, it is beyond the seller’s control.
Question worth asking
It is not unreasonable to assume an even distribution of work email IDs between Gmail and Microsoft. If the reality is skewed, it most likely will be in favor of Microsoft (i.e., more Microsoft IDs).
Therefore, if we send 100 emails to prospects in small, medium, and large enterprises, we are likely to reach approximately 50 Microsoft and 50 Gmail inboxes, assuming no other email provider.
Most enterprises will either not change the default image rendering settings. Even if they do, they will change it to prevent images from being displayed by default without explicit user action.
Under these reasonable assumptions, if an email campaign gets an open rate which is beyond 50%, it is worth asking if the number is reliable and if decisions can be made using the same.
Disclaimer
Please note that the purpose of this black paper is to highlight technical challenges associated with tracking email open rates in a reliable manner and not to critique any existing email campaign software.
We are building a Gen AI tool for personalized email campaigns in which we have eliminates the metric of open rate. We have included click rate, but with necessary disclaimers about the efficacy of the metric.
