Are you considering running A/B tests this holiday season?
Testing during the holidays may seem like a good idea — after all, the spike in traffic can help complete a test quickly — but it’s important to understand how the results might be skewed.
Consumer behavior is different during the holidays: their sense of urgency is higher, or they may be more concerned with factors like delivery time than with good deals. In addition, any change you’ve made in advertising spend may bring in an audience that isn’t your usual one.
But testing during the holiday season isn’t necessarily a bad idea, either. Holidays are simply one among a long list of external factors that can change your buyers’ moods and behaviors.
Whether you test or not, you can still gain valuable insights from the unique visitor behavior that comes with the holidays.
Here’s what you need to know.
Some businesses are hesitant to test during the holiday season because of the perceived financial risk of sending costly traffic to variations that might underperform, or the operational risk of introducing tests during a seasonal code freeze where little to no change is allowed by website development teams.
But take a closer look. On the financial side, if you’re afraid to run an underperforming variation during this prime time for quick tests, you may be losing more money than you would by running tests.
What tells you that your current page is performing well?
If you’re not testing, you’ll never be able to understand what drives your customers during the holiday season. A possible loss in the short run will generate bigger wins in the long run. The high traffic also provides a good opportunity to get statistically significant results on segments.
As far as operational risk goes, many platforms, including Optimizely, have a visual editor that allows users to create their own variations without touching the code, as long as the changes are simple and don’t change the functionality of the page.
For example, you can remove, resize, add and rearrange content without touching the code. A simple test, such as changing the headline, is easy to do and doesn’t require the development team to drop what it’s doing for a short test.
Option 1: Keep testing and gain urgency insights
You can use the holiday season as your once-a-year opportunity to determine statistically significant winners on multiple tests in a limited timeframe. Try testing your shoppers’ sensitivity to Urgency (one of the six LIFT Model factors.)
Consider what’s important to shoppers at different points in the holiday season, for example:
- Early season shoppers may be less time-sensitive than last-minute shoppers. The most effective messaging will probably be about finding the right gift.
- For later season shoppers, the number of products left in stock and when the user can expect the product to arrive may be more relevant. Use stock counts, recent order counts, or offer expedited shipping options and make gift options more prominent.
Once the shopping season is over, repeat your tests to find out if the insights you gained apply to the other holidays (and possibly, although unlikely, all year round). We used this approach to increase donations over the holiday season for the International Rescue Committee (IRC), a non-profit organization.
In IRC’s case, we discovered that when external urgency was added during a disaster donation season, a different page performed better than during a non-urgency period. The organization learned some important principles about the different types of messaging to use during high and low urgency seasons from the landing page tests we conducted. For more details about how the test was run, watch this landing page optimization webinar.
Option 2: Hold off on testing while you conduct qualitative studies
If A/B testing is out of the question for the holiday season, then use this time to learn about your users through different means. Focus on the “Explore” phase of the Infinity Optimization Process, for example.
Try interviewing customers, sales and support people to understand customers’ motives and why they buy from you. Don’t underestimate the value other departments can bring in helping you better know your users.
We recommend you keep testing during the holiday season, but limit your test timeframe to the season itself. Don’t let a test run any longer, or it will skew your results. Remember that the insights you get are specific to holidays and should be treated as such (unless follow-up tests prove otherwise).
If you can’t do any testing during the season, use the time to keep driving for insights, remembering that any results, even negative ones, can give you a clearer picture of what your customers are looking for.
The post A/B testing during the holidays: Risks and rewards appeared first on WiderFunnel.