Who can use this feature?
This feature is available on the Advanced and Expert plans. If you're interested in taking advantage of it, please reach out to your customer success manager.
A/B Testing allows you to test and compare up to 5 different variations on a single merchandising campaign, each of which you can make a change using any Visual Merchandising tools: Boost rules, banners, pinned products, removed products, and custom filter order.
Setting up an experiment
An experiment is an individual test run on a campaign over a specific amount of time.
To begin setting up an experiment, go to edit a campaign, open up the A/B Test toolbox, and select "New Experiment".
The basic options for an experiment are shown. You'll need to select an End Date.
Under "Set up your variations", select the variation you'd like to set up. Then, exit the toolbox and make all of the changes you'd like. The variation you're currently changing is marked in blue to the left.
Optionally, you can also choose which variation to use as a default after the experiment ends, but before you've selected a formal winner.
Near the top of the A/B toolbox, you'll see the text: "Upon completion, Variation A will be used until a winner is declared by you (Change)". Click on "Change" to modify which variation you'd like to default to.
You can add up to 5 different variations in a single experiment by clicking "Add another variation".
The more variations you add, the longer you'll need to run your experiment to get accurate data sample sizes.
You can also set up what percentage of shoppers are split into each variation by dragging the sliders under "How many customers should be placed in each variation?"
Save
To begin your experiment, simply save the campaign as you normally would. A popup dialog will ask to confirm the beginning of the experiment. You will not be able to make any merchandising changes until the experiment is complete.
Running an experiment
While the experiment is running, no merchandising tools will be editable.
However, you may still edit the campaign's title, schedule, experiment end date, experiment default winner, and end the experiment manually. (See below)
When viewing your list of campaigns, a blue icon will indicate an experiment is currently running.
A/B Test toolbox while an experiment is running
While an experiment is running, you can still use the A/B Test toolbox to change the end date, and the default ending variation.
You can also view the set up of the running variations by clicking one of them, and you can end the experiment immediately.
Completing an experiment
Ending the experiment manually
To end an experiment immediately, you can click on "End Experiment Now". Here you will need to commit to a winner, which will be used moving forward.
Pending Winner Selection
Once a campaign has reached its end date, the A/B Test toolbox will prompt you to select the winner.
A red icon on the list of campaigns will also indicate that the experiment's end date has been reached, and a winner needs to be declared.
Winner confirmed
In either case, you'll need to save your campaign once a winner has been declared to complete the experiment.
Trying it out
You can easily simulate different shoppers on your live site by clearing your cookies for the site. We suggest a tool such as Cookie Remover for Chrome. Once you've cleared your cookies, simply refresh the page and you will be re-sorted into a group for the experiment. There is a chance that you'll be sorted into the same variation group again, so you may need to do so multiple times.
How do I view reporting for an experiment?
You can view the results of your A/B Test directly in the SMC, by using the Experiment Report. For more details on that report, please refer to the A/B Experiment Reports article.
Glossary
A/B Testing: The name of this feature.
A/B Test: A common synonym for "Experiment" or "A/B Merchandising"
Experiment: An individual test run on a campaign over a specific amount of time. Contains 2-5 variations in a single experiment.
Variation: An individual set of elements. Shoppers are divided randomly and shown only one variation.
Element: Part of the page's behavior that a merchandiser can change. There are 5 elements: banners, boosts, pinned products, removed products, and custom filter order.
Toolbox: The expandable panels available on the right hand side of the Visual Merchandising Edit Campaign interface. Each toolbox may contain multiple tabs to view/manage other content.
Context: The conditions which must be met in order for the campaign to take affect.
Comments
0 comments
Article is closed for comments.