A/B Testing is used in order to test different versions of your
e-mail in real-time and to easily and quickly find out which version
is the best to send to your profiles. In this way, you can test
content variability within the same mail, ranging from simply having
two different subjects to having two completely different versions
of the message. This will allow you to draw conclusions on what
worked best and for which reason.
After creating your A/B campaign, ACTITO will send each version of
the e-mail to a sample of profiles in your database. You can have up
to 4 different versions.
After a test period of the duration of your choice, ACTITO will
calculate which e-mail worked best according to either the opening
or the click rate. The best performing e-mail will then be sent to
the rest of the target (people who have not been reached yet).
Creating your A/B Campaign
A/B testing is a campaign mode in itself, like Marketing or
Scenarized e-mails. In order to create an A/B campaign, is
therefore necessary to make your selection right at the creation
of your message.
In "Manage e-mail campaigns" (Catalog > E-mail
campaigns > Manage e-mail campaigns), click on "Create a
new campaign".
Next, select the "A/B TESTING" mode.
In practice, after the test period, an A/B campaign will
behave like a Marketing campaign. It is therefore used to contact
a large number of profiles.
Good to know
If you want to make A/B tests on Scenarized e-mails, it is
possible to use the "A/B split" block in the
Scenario editor.
Step 1: General Data
As for every campaign, the first step will consist in giving a
name to your campaign, selecting the target Database, the entity
and the language of the campaign.
Tip
In A/B campaigns, you only have the possibility to select
one definition language. Indeed,
A/B testing is used to determine which version of your
e-mail works best, but the behavior of your profiles might
vary greatly according to their culture and their language.
A more informal message might work best in a language but
not in another, and idioms that cannot be translated might
prevent you from having exactly the same message in two
languages, in which case the selection criteria will be
distorted.
If you want to contact your whole database, it is possible
to select a default language at step 2 in order to contact
all your profiles in the same language.
AB test parameters
Next, you will be able to select:
Message headers
If you decided to test variation on the sender, the subject
and/or the pre-header, you will be invited to provide each
version in the 'Message headers' section.
Step 2: Target Group
As for any campaign, you will have to select the Subscription
and the target population of your e-mail. You should target all
the profiles that you want to contact: the test population will
be selected randomly and the repartition criteria will be set at
a later stage.
If you want to contact profiles no matter what their mother
language is, you can un-check the 'Language' box to contact
everyone in the language chosen at step 1.
Step 3: Message
You can create your message from a template, by importing an
HTML or by coding your own HTML.
If you import HTML files to do an AB test on the content,
the ZIP file must include as many HTML files as versions,
with the version being specified in the suffix of the file
name, under the format "_a", "_b", etc.
If you chose to do A/B testing on the content at step 1, you
will have to define a specific content for each version.
You will be able to jump from a version to another by using the
drop-down list in the top right corner of the message creation
screen.
In the template editor, the version that you are editing
will be displayed in the top left corner.
Tip
What if you only have minimal variation between your
versions? In the template editor, you can copy the content
of another version by clicking in the 'Options' in the top
left corner and choosing 'Import content from another
version'. After that, you are free to apply the variations
on which you want to do A/B testing.
Step 4: Check and Test
After creating your message, you will be able to test it by
selecting a sample of profiles from your testing group. Then you
will have to provide one or more e-mail addresses to which the
test e-mails will be sent.
The "A/B" column enables you to select which version
you want to generate for each test profile. If you leave the
default value "A/B", each version of the e-mail will
be generated for this test profile (including version C and D if
they exist).
Alternatively, you can choose to only select
one version from the drop-down list for each profile.
Step 5: Sending
At step 5, you will get a summary of the general data, the
message headers and the target group of your campaign.
In addition, you will be able to set the AB testing parameters.
AB testing parameters
-
Sample size for versions: a
sliding bar will allow you to choose the repartition
between the test version and the winner version. If you
have more than 2 versions (A/B/C/D testing), each
version will automatically have the same sample size.
-
Optimization criteria: you
can choose the criteria used to determine the winner
version, either the opening rate or the clicks rate.
-
Period: determine the test
period at the end of which the winner version will be
decided. It can be set either in hours or in days.
After selecting the AB testing parameters, you can choose to
send the campaign right now or to schedule it for later.
Tip
It is possible to set up an AB test with 100% of the
target as sample, for example if you want to identify the
best version in order to optimize your next campaigns.
To do so, you will still need to choose an optimization
criteria and a test period. They will have no impact on
sending, as the e-mail will be sent immediately sent to
the whole target, but they will be used to identify the
winner version in the campaign report.
Analyzing your A/B campaign
Once you're a/B campaign has been sent, it will be available in
the "In progress" tab during the duration of the test
period that you defined at step 5 of the campaign creation.
During the test period, it is
not possible to "Correct"
this campaign. However, it is possible to "Finish" it if
you do not want to send the winner version of the e-mail.
If you want to check the repartition of the sample
population or view each version of your e-mail, you can click on
the "View definition" button.
View reporting during the test period
You can view the campaign report at any time in order to check
the evolution of the opening or click rate of each version in
real time.
Do to so, click on the "View
reporting/results" button.
The "Summary" tab, which is found in every
campaign report, will provide the global stats of your campaign.
To view the repartition of you're A/B campaign, go to the
"A/B testing" tab, which is only present for this
specific type of campaigns.
This lets you see the number of e-mails sent for each
version. The target population sample will always be the same
for each version, but slight variations may appear, when
addresses are filtered for instance.
Next, you can compare the delivery, opening and click rate
of the different versions of your e-mail. It can be done at any
time during the test period.
The "Optimized" column, which remains empty
during the test period, will display the stats of the winner
version once it has been determined.
Viewing the optimized reporting
When the test period is finished, the campaign will appear in
the "Sent" tab and you will be able to view the
complete campaign report.
In the "A/B testing" tab, the header of the
"Optimized" column will now display the winner
version.
The campaign stats found in the "Optimized" column
only relate to the optimized version, which means the e-mails
sent after the test period, without
including the related test sample.
At the bottom of the page, the "Winner" tab
specifies the winner version and reminds the criteria used to
choose it.
Once the winner version is sent, your A/B campaign will
appear in your last campaigns on your Dashboard, among the other
mass campaigns.