Draft a CRO QA protocol: instrumentation, bucketing, randomization checks, sample ratio…
- User
Draft a CRO QA protocol: instrumentation, bucketing, randomization checks, sample ratio mismatch detection, and variance audits.
Assistant
1. Understand the Goal: The user wants a "CRO QA protocol" (Conversion Rate Optimization Quality Assurance protocol). Specifically, it needs to cover five key areas: * Instrumentation (tracking setup) * Bucketing (assigning users to groups) * Randomization checks (ensuring groups are equivalent) * Sample Ratio Mismatch (SRM) detection (checking if split sizes are statistically valid) * Variance audits (checking data stability and fluctuation).
-
Target Audience: This is for CRO specialists, data analysts, or developers working on A/B testing (experimentation). The tone should be professional, technical yet actionable, and structured like a formal protocol or standard operating procedure (SOP).
-
Structure the Protocol: A logical flow is essential for a protocol. I'll structure it as follows:
- Title & Objective: Clear statement of purpose.
- Phase 1: Pre-Launch (Instrumentation): Things to check before the test goes live.
- Phase 2: Launch & Assignment (Bucketing & Randomization): How users are split.
- Phase 3: Post-Launch / Ongoing (SRM & Data Integrity): Monitoring the test while it runs.
- Phase 4: Pre-Analysis (Variance Audits): Final checks before interpreting results.
- Tools & Pass/Fail Criteria: Quick reference.
-
Fleshing out the Sections (Drafting Content):
-
Phase 1: Instrumentation (The Foundation):
- Goal: Is data actually being recorded?
- Steps: Visual check (QA mode), Network tab inspection (GA/GTM/Amplitude requests), Cookie persistence (does the user stay in the bucket?), Cross-browser/device testing.
- Key Check: Triggering logic (prevent flicker, ensure triggers fire on DOM ready or specific interactions).
-
Phase 2: Bucketing & Randomization (The Setup):
- Goal: Is the split logic sound?
- Steps: Check user IDs vs. deterministic hashing. Ensure 50/50 (or whatever split) logic holds.
- Randomization Check: We can't do a full SRM check yet, but we can check if the mechanism allows for bias (e.g., bot traffic, geolocation skew).
-
Phase 3: Sample Ratio Mismatch (SRM) Detection (The Alarm):
- Goal: Is the split actually 50/50 (or 33/33/33)?
- Definition: Statistical test (Chi-square) to see if observed counts match expected counts.
- Threshold: p-value < 0.01 usually indicates a serious problem.
- Action: If SRM fails, STOP the test. Debug immediately.
-
Phase 4: Variance & Data Audits (The Stability):
- Goal: Is the data noisy?
- Steps: Check for outlier revenue (Whale problem). Check for unusual spikes in variance (standard deviation).
- Cuped: Mention CUPED (Controlled-experiment Using Pre-Experiment Data) if possible, but keep it simple. Let's stick to "Variance Audits" as requested: checking if metric variance is within expected bounds.
-
-
Refining and Polishing (Self-Correction during drafting):
- Critique: Is "Bucketing" distinct from "Randomization"?
- Correction: Yes. Bucketing is the technical implementation (how we assign the ID), Randomization is the outcome (are the groups statistically similar in composition?). I need to make sure I distinguish technical assignment from statistical equivalence.
- Critique: "Variance Audits" is a bit vague.
- Clarification: I will define this as checking for outliers (revenue) and checking the standard error to ensure the sample size is sufficient for the fluctuation observed.
-
Final Formatting: Use bold headings, bullet points, and "Pass/Fail" criteria to make it look like a checklist. Add a "Tools" section for practical application (Python/R/Excel formulas).
-
Detailed Breakdown per Section (Mental Draft):
- Instrumentation: QA links, pixel helper, console logs.
- Bucketing: User ID consistency (client vs server side), 404 checks on redirect tests.
- Randomization: Checking browser splits, device splits, and OS splits within the control/variant to ensure we didn't accidentally exclude Safari users from Variant A, for example.
- SRM: Provide the formula or logic. .
- Variance: Box plots for revenue, standard deviation checks.
-
Final Review against User Prompt:
- Instrumentation? Yes.
- Bucketing? Yes.
- Randomization checks? Yes.
- SRM detection? Yes.
- Variance audits? Yes.
-
Generate Output. (Proceeding to generate the response based on the structure in step 3 & 4).
(Self-Correction during generation): Make sure to emphasize that SRM is the most critical "stop the line" metric. Also, define "Pre-experiment variance" (CUPED context) vs "In-experiment variance" (Noise).
- Final Polish: Add a disclaimer that this assumes a standard A/B testing platform (Optimizely, VWO, Google Optimize - wait, GO is sunsetting, use generic or Amplitude/Posthog references). Keep it platform agnostic.
(End of thought process. Outputting protocol.) This is a comprehensive Quality Assurance (QA) protocol for Conversion Rate Optimization (CRO) testing. It is designed to be used as a Standard Operating Procedure (SOP) before launching an experiment and during the initial monitoring phase.
CRO Technical QA Protocol
Objective: To ensure the technical integrity of A/B and Multivariate tests, guaranteeing that data collected is valid, unbiased, and statistically sound.
Phase 1: Instrumentation & Flicker QA
Validating that the tracking mechanism captures data accurately and the user experience is seamless.
1.1 Implementation Check
- Snippet Installation: Verify the testing tool’s snippet (e.g., Optimizely, VWO, GA/GTM) is placed correctly in the
<head>of the document (or via a Tag Manager) to ensure early execution. - Trigger Logic: Confirm the experiment triggers only on the intended URLs (including query parameters and trailing slash variations).
- Test: Visit the URL with
?parameter=testandhttpsvshttpto ensure consistent bucketing.
- Test: Visit the URL with
1.2 Visual QA (The "Flicker" Test)
- Flicker Check: Reload the page (Hard Refresh: Ctrl+Shift+R) 10 times.
- Pass: The variation loads instantly with no flash of the original content (FOOC).
- Fail: The original content appears briefly before switching to the variation.
- Responsive Design: Check the variation on Mobile (iOS/Android), Tablet, and Desktop resolutions. Ensure elements do not overlap or hide critical CTAs.
1.3 Event Tracking Validation
- Goals Verification: Open the Network tab in Developer Tools or use a testing tool (like Omnibug or GA Debug).
- Trigger the conversion event (e.g., click a CTA, scroll to depth).
- Pass: A network request is sent to the analytics platform with the correct Event ID/Category/Action.
- Fail: No request fires, or the request fires multiple times on a single click.
Phase 2: Bucketing Logic & User Assignment
Validating that users are placed into the correct groups and stick to them.
2.1 Audience Exclusion
- Exclusion Logic: If specific users are excluded (e.g., internal IP addresses, logged-in admins), verify that these users cannot enter the experiment.
- Action: Test with a VPN or Incognito window (if targeting general public) to ensure the experiment activates.
2.2 Bucketing Consistency (Sticky Sessions)
- Cross-Session Persistence: Enter the experiment as a new user. Note your bucket (Control or Variation). Close the browser completely. Reopen and visit the site.
- Pass: You return to the same bucket.
- Fail: You are assigned to a different bucket (Randomization drift).
- Cross-Device Logic: If applicable. Check if a user logging in on mobile matches the bucket they received on desktop (requires User ID tracking).
2.3 Redirect vs. Visual Editor
- Redirect Tests: If running a split URL test, ensure the redirect happens server-side (preferred) or client-side with minimal latency. Check that
document.referreris preserved on the redirected page.
Phase 3: Randomization Checks (Chi-Square Goodness of Fit)
Ensuring the traffic split matches the configured allocation (e.g., 50/50).
3.1 Initial Traffic Distribution
- The Setup: Configure the test to run at a 50/50 split.
- The Sample: Drive a small "pilot" sample of ~1,000 visitors to the test.
- The Check: Perform a Chi-Square () test or simply eyeball the raw numbers for gross errors.
- Threshold: If the split is 50/50, observed counts should be within +/- 5% (e.g., 525 vs 475 is okay; 600 vs 400 is a fail).
3.2 Segment-Level Randomization
- Stratification Check: Check if the randomization holds true across different device types.
- Audit: Does the Variation receive 80% Mobile traffic while Control receives 50%?
- Action: If device mix varies significantly between buckets, the test is biased (see SRM below).
Phase 4: Sample Ratio Mismatch (SRM) Detection
The "Guardrail" metric. If this fails, the experiment results are invalid.
Concept: SRM occurs when the observed ratio of users in control/variation differs statistically significantly from the expected ratio.
4.1 The Calculation (Automated or Spreadsheet)
Use a Chi-Square Goodness of Fit test.
- Null Hypothesis (): The observed ratio equals the expected ratio.
- Expected: 10,000 visitors (5,000 Control / 5,000 Variation).
- Observed: 10,000 visitors (4,800 Control / 5,200 Variation).
Formula:
4.2 The Decision Rule
- P-Value Threshold: (standard for SRM).
- Action:
- If : No SRM detected. Proceed with analysis.
- If : STOP THE TEST. There is a technical bug in bucketing.
4.3 Common SRM Causes
- Bot filtering affecting one variation more than the other.
- Page load speed differences (Variation loads slower, causing users to bounce before being bucketed).
- Redirect loops blocking the Control page.
Phase 5: Variance Audits & Data Integrity
Ensuring the data isn't skewed by outliers or unexplained spikes.
5.1 Outlier Detection (Revenue/Duration)
- The "Whale" Problem: A single user purchasing $50,000 in the Control group can skew the Average Order Value (AOV), making the Variation look worse than it is.
- Audit Protocol:
- Export raw data.
- Calculate the 95th and 99th percentiles for Revenue.
- Winsorization: Cap values at the 99th percentile (e.g., any purchase over $500 is counted as $500) and re-run the analysis.
- Compare significance with and without outliers.
5.2 Pre-Post Variance (CUPED Check)
- Concept: Compare the variance of the Goal Metric (e.g., Conversion Rate) against historical baseline.
- Audit: Plot the daily conversion rate of the Control group.
- Pass: The daily variance looks similar to the 30 days prior to the test.
- Fail: High variance (spikes and valleys). This suggests the sample is too noisy for a valid conclusion, requiring a longer test duration.
5.3 Cross-Platform Validation
- The "Source of Truth" Check: Compare the sample count (Visitors) in your Testing Tool (e.g., Optimizely) against your Analytics Tool (e.g., Google Analytics/Amplitude).
- Threshold: Discrepancy should be < 5%.
- Reason for Failure: Ad-blockers, differing cookie definitions, or timing issues.
Summary Checklist for Launch
Protocol Item Status (Pass/Fail) Notes Flicker Test [ ] No FOOC observed on hard refresh. Cross-Browser [ ] Verified on Chrome, Safari, Firefox, Mobile. Goal Firing [ ] Network requests verified in Console. Sticky Sessions [ ] User keeps bucket ID after browser restart. SRM Check [ ] Chi-Square p-value > 0.01. GA vs Tool [ ] Visitor count discrepancy < 5%. Outlier Audit [ ] Winsorization check performed (if Revenue). -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.