Flint logo
Back to guides
Tutorials

A/B Testing with Optimizely on Flint Pages

Optimizely Web Experimentation is an enterprise-grade testing platform with AI-powered experiment recommendations. This guide walks you through connecting Optimizely to your Flint pages and running your first experiment.

Overview

Optimizely Web Experimentation is a powerful testing platform built for teams that need enterprise-grade experimentation, AI-assisted test planning, and deep integration with their analytics infrastructure (Snowflake, BigQuery, etc.).

With Optimizely connected to your Flint site, you can:

  • •Run A/B, multivariate, and multi-page experiments across all your Flint landing pages
  • •Use Optimizely's AI agents to get experiment recommendations based on your goals
  • •Integrate experiment results with your existing data warehouse
  • •Target specific audiences by UTM parameters, geography, device, behavior, and more

Optimizely is best suited for marketing teams that already have or are building out an experimentation program and need robust statistical rigor and reporting.

What you need

  • •A Flint account (sign up here)
  • •An Optimizely Web Experimentation account (optimizely.com) - Optimizely requires a sales conversation for pricing
  • •Your Optimizely Project ID, available in your Optimizely dashboard under Settings > Implementation

Step 1: Add the Optimizely snippet to your Flint site

Optimizely uses a JavaScript snippet that must be loaded synchronously in the <head> of your pages to prevent flicker during experiments.

  1. 1.In Optimizely, go to Settings > Implementation and copy your project's JavaScript snippet. It looks like this:
html
<script src="https://cdn.optimizely.com/js/YOUR_PROJECT_ID.js"></script>
  1. 2.In your Flint editor, click the Settings button in the top navigation, then go to Site Settings > Custom Code.
  2. 3.Paste the Optimizely snippet into the Head field.
  3. 4.Click Save. Optimizely is now active on all your Flint pages.

Important: Optimizely recommends loading this snippet as early as possible in the <head> to avoid a flash of the original content before variants are applied. Placing it in the Head field in Flint ensures this.

Step 2: Create an experiment in Optimizely

Once the snippet is installed, create an experiment in Optimizely:

  1. 1.In Optimizely, click Create New > A/B Test.
  2. 2.Enter your Flint page URL as the Target Page (e.g., https://yourdomain.com/landing-page).
  3. 3.Optimizely's visual editor will load your Flint page. Use it to make changes to any element - headlines, images, buttons, or layout - to create your Variation.
  4. 4.Define your Metrics: choose a primary metric (e.g., a click event on your CTA button, or a page visit on your thank-you URL). Add secondary metrics if desired.
  5. 5.Set up your audience if you want to target a specific segment (e.g., only visitors from a specific UTM source).
  6. 6.Configure your traffic allocation (e.g., 50% original, 50% variation).
  7. 7.Click Start Experiment to begin running traffic.

Step 3 (optional): Full page variant test

For testing two completely different page designs, use Optimizely's Redirect Test:

  1. 1.In Flint, duplicate your landing page to create a variant (e.g., /landing-page-v2) and make your design changes there.
  2. 2.In Optimizely, create a new A/B Test and add a redirect variation instead of a visual variation.
  3. 3.Set the redirect URL to your Flint variant page URL.
  4. 4.Optimizely will split traffic between the original and the redirected variant, and attribute conversions to each.

This is the cleanest approach for testing substantially different page structures.

Step 4: Track custom conversion events

For goals beyond clicks and page visits, you can instrument custom events from your Flint pages:

javascript
// Fire a custom Optimizely event when a specific action happens
window.optimizely = window.optimizely || [];
window.optimizely.push({
  type: "event",
  eventName: "cta_click"
});

To add this to a Flint page element, prompt the Flint agent:

text
When the primary CTA button on /landing-page is clicked, fire a custom Optimizely
event called "cta_click". Optimizely is already loaded via script tag.
Use: window.optimizely.push({ type: "event", eventName: "cta_click" })

The agent will add the event listener to the correct element.

Understanding Optimizely's Stats Engine

Optimizely uses a proprietary Stats Accelerator (Stats Engine) that applies an "always-valid" frequentist approach. This means you can check results at any time without the elevated false positive rate that comes from peeking at a traditional fixed-horizon test.

Key things to watch:

  • •Statistical significance: Optimizely reports a confidence level. A result is considered significant when the improvement is detected with high confidence (typically 95%+).
  • •Sample ratio mismatch: Optimizely will flag if traffic isn't splitting as expected, which can indicate a technical issue with the snippet or targeting rules.
  • •Primary vs. secondary metrics: A variation may win on the primary metric while losing on a secondary one. Review all metrics before declaring a winner.

For critical decisions, run experiments long enough to cover full business cycles (at least one week, ideally two) to account for day-of-week effects.

Using Optimizely's AI agents

Optimizely offers AI-powered experiment planning through Optimizely Copilot, which can analyze your site performance and suggest what to test next. To use it:

  1. 1.In Optimizely, look for the Copilot or Ideas section in the navigation (the exact label depends on your plan and Optimizely version).
  2. 2.Connect your analytics data source (Google Analytics, your data warehouse, or Optimizely's built-in analytics).
  3. 3.Optimizely Copilot will surface high-impact experiment ideas ranked by expected uplift.
  4. 4.When you select an idea, Optimizely can pre-populate the experiment configuration - you just need to build the variation in Flint and point Optimizely at it.

This is particularly useful for teams running a high volume of experiments who want AI to help prioritize the roadmap. Note that Copilot features are available on higher-tier plans - check with your Optimizely account team for availability.

Tips for better results

  • •Optimizely works best when integrated with your data warehouse (Snowflake, BigQuery, etc.) so experiment results can be analyzed alongside your full customer data
  • •Use Optimizely's Exclusion Groups to prevent the same visitors from being in multiple experiments simultaneously, which can skew results
  • •Document your experiment hypotheses before launching - Optimizely's built-in hypothesis field helps teams maintain an organized experiment backlog
  • •For ABM landing pages, use Optimizely's audience targeting to run separate experiments for different industries or company sizes