Persado is the advanced language personalization platform that delivers outsized impact through AI content generation and decisioning at scale. This article teaches you about how Persado uses experimental design to obtain the maximum information from testing marketing language with the minimum amount of resources. If your agreement doesn’t include Experiments, and you’ll only be employing Predictive Content, the contents of this article won’t apply to you since Predictive Content doesn’t use experimental design.

After reading, you will understand:

Experimental design is a field of statistics that deals with how to efficiently plan an experiment to obtain as much useful information as possible by performing a small number of trials. It allows us to identify cause-and-effect relationships between inputs and outcomes in a rigorous, replicable way, so we have statistically significant knowledge and can be assured our results were not due to chance.

Experimental design is not unique to Persado - it’s an established methodology of statistics, mostly discussed in academic circles before adoption by the pharmaceutical world in the 1970s. Unlike simply collecting and observing data from a sample of patients, medical academics began designing experiments in order to measure the precise impact of changing different, independent variables.

Persado employs experimental design in a unique way by applying it to language. The Experiments we run identify the precise words and phrases that will drive the most engagement with your audience. In the context of your marketing campaign, experimental design allows us to:

Even more, experimental design allows us to assess which combinations of elements have the highest impact, even if we haven’t yet deployed them yet - something other forms of testing, like A/B testing, can’t deliver.

Generally, experiments are scientific procedures carried out to support or refute a hypothesis, but at Persado they have a specific function. Experiments are our practical application of experimental design. These methods of content generation include Full Experiments, One-step Experiments, and Exploration Only Experiments, all of which allow us to gain granular insights into which parts of our messaging are working with your audiences so we can learn continuously over time and deepen our machine's understanding of your brand. Typically, Experiments contain 8 or 16 Variants plus your control.

Imagine testing all the possible versions of a promotional email subject line. It's hard to even imagine how many there are. Hundreds? Thousands? Infinite? You might begin by breaking the subject line down into 4 elements:

You then might come up with 4 different options for each of those parts. That would give us a steep total of 256 possible combinations. Testing this many combinations with limited human resources and budget simply isn’t feasible using traditional marketing techniques. This is why Persado uses experimental design.

One of the most common methods marketers use to test different messages is the A/B test. As the name implies, A/B tests allow brands to randomly divide an audience into 2 groups and deliver 1 version of a message to one group and deliver a different version to the other group, and assess which performs better. A/B testing can be thought of as testing the champion version against the challenger version.

A/B tests are limited in today’s climate of massive scale and real-time interactivity. They cannot, for example, efficiently test multiple versions of multiple-page elements against one another, or show why a particular message performs better than another. Nor can they show why a failed message produced no response, or a negative response.

On top of that, A/B tests can very easily be biased, particularly when there are multiple things that are different between the 2 versions tested. Imagine version A has a green button that reads "click to know more" while version B has a red button that reads "next step," and version A has a 20% higher click rate than version B. How do you know if that 20% is driven by the button color or by the text? The short answer is, you don't.

Tests that use experimental design methodology are extremely efficient. With experimental design, we can test a number of variations and then use predictive analysis to identify the best observed and best predicted combination of elements for the brand. Going back to our previous example, experimental design allows us to test all 4 parts of the subject line, and 4 options of each of those 4 parts, while still deploying a very small number of combinations.

You may be wondering: how? Experimental designs are robust, but they need to be constructed in a specific way. The principles of experimental design discussed in this section help us determine which of the thousands of possible combinations to deploy that are representative of the broader population, and how to test them to ensure that our results are significant and not random.

This is only possible through the following key elements of experimental design:

At Persado, the various marketing messages we deploy for testing - which we refer to as Variants - need to meet two important criteria: balance and orthogonality. These are standards that most experimental designs, and all Persado experimental designs, must meet.

It’s commonplace to test a control (the standard marketing message) with every Experiment. The major advantage of using a control is that it helps detect and eliminate inadvertent bias. An additional benefit is that the use of a control yields a more precise estimate of the difference in performance between the original copy and the Persado Variants.

For the Experiment to be valid, audience assignment for each Variant needs to be randomized. This is to ensure that we are testing the individual elements of the Variants as opposed to a combination of the Variants + the audience. The systematic, random allocation of message Variants to uniform groups of customers ensures that extraneous factors, such as frequency of buying, age group, region, etc., don’t introduce bias into the Experiment.

Fundamental to our algorithm is the use of binary response data, which refers to only 2 types of outcomes, like: yes or no, 0 or 1, the number of clicks versus the number of non-clicks. While non-binary response data, like net promoter score (NPS) or revenue, can technically inform Persado’s algorithm, these metrics generally require very large sample sizes and user-level data. As a result, brands most often use response actions like clicks, conversions, or application submissions.

When you run an Experiment with Persado, you want to find the best ways to drive results for your brand. But not all results are created equal. There’s a chance that one may be a random happening, rather than a product of an intervention. To ensure we avoid this, we rely on statistical significance.

Statistical significance indicates that the result of an Experiment is not likely to have occurred due to chance or luck. It is quantified through a metric called p-value, which is the probability your results were caused by what you were testing. In other words, if what we observed was very unlikely to happen due to random fluctuation, then we must accept that our intervention was the reason for the observed result.

For example, p=0.05 means that if we repeated the same Experiment 100 times, the same results would be expected in about 95 of them. In the other 5 tests, we would report that our test caused the result, when in fact the result could have been caused by random noise. This is called a "type 1 error," or a "false positive." By reaching statistical significance, we minimize our chances of incurring this error.

Persado’s cutoff probability for rejecting our hypothesis is usually very strict - less than 5%. In marketing, application practitioners often accept levels of significance as high as 15%.

Statistical significance is influenced by a few parameters.

The greater the sample size you use, the more significant your results will be - and the more confident you will be that you improved your ad copy. At Persado, we use advanced mathematical methods to calculate the minimum sample size you need to reach statistically significant results every time ad copy changes. Our methodology is based on formulas used in the experimental design community that we then revise as needed per channel. The minimum sample size varies per channel, but it usually ranges from 1,500 to 8,000 responses.

In addition, there needs to be a large enough difference in performance between the best and worst performing Variants in order to generate statistically reliable conclusions. The greater the difference a change makes in response rate, the easier it is to achieve statistical significance. Persado creates extreme ad copy variations that generate big differences in response rate and make it easier to identify which elements significantly influence the response rate.

The fewer Variants you test, the faster you will reach a conclusion on which ones are significantly different. Our statistical algorithms take into account the available population you have for testing and your previous results, and recommend the optimal number of Variants that need to be tested so that you will reach significant results and gain the most insights.

Persado has developed an algorithm to automate the selection of the very specific set of Variants that meet the above principles of experimental design:

For a balanced and orthogonal (i.e., unbiased) experimental design, the smallest number of Variants required to test is the lowest common multiple of the number of values in every element being tested. So, if your message has at least two 4-value elements (like 4 different variations on your emotion and your offer description), then the smallest possible design is 4 x 4 = 16, and it can also support 2-value elements (since 16 is a multiple of 2x2 and 2x4).

Persado has found over years of working with customers that 16 Variants obtain useful results and can still be efficiently produced and approved. However, this is not restrictive; we can use a different experimental design and test more (e.g., 32) or less (e.g., 8) depending on your objectives.

After those 16 Variants are deployed in market, Persado gathers the results and feeds them into a statistical model that can predict the response rate of any of the 16,384 possible combinations. By doing that, we can tell exactly which combination has the maximum response rate, even if it was not part of the initial set of 16 Variants.

This means Persado is able to deliver both a best observed outcome and best predicted outcome.

The best observed option is the one that is actually observed to perform the best in an Exploration phase, where we release our selected Variants to a subset of your customer population to learn in real time what works and what doesn’t.

The best predicted outcome is one that Persado’s statistical model predicts will perform the best based on experimental design. Based on our findings in the Exploration phase, the messages that perform the best and meet statistical significance are then put into real world tests to test actual performance in what we call a Broadcast phase.

It can take a person multiple contacts with a brand before they decide to become a customer, so every opportunity to communicate and engage with people counts. If a brand can increase the number of people who respond to their marketing messages, they increase conversions, so the ability to know what works to engage quickly and effectively is paramount.

Knowing which messages resonate and which do not gets to the heart of competing in customer experience.

During Covid-19, for example, online sales became increasingly important. One major luxury goods retailer wanted to boost awareness and adoption of their mobile app to support digital shopping. They used Persado to run an Experiment on the language in the text above the image as well as the CTA. The winning Variant we identified drove 31% more app installs.

The most critical parts of this creative were the CTAs. The control’s original CTA button said “Install Now.” The subtle change from “Now” to “Immediately” helped drive significantly more engagement. Furthermore, the CTA in the text was also important. “Download it immediately” was more effective than “Download immediately the Stylable app” in the control. Emotion also played a key role. “We invite you to discover,” which conveys exclusivity, was more effective than other emotions like curiosity, gratification, and urgency.

This granular level of insight into which parts of the message contributed most to engagement is only possible through experimental design.