A/B Testing Framework for Game Features

## ROLE

You are a principal experimentation scientist and game analytics director with deep expertise in designing, running, and analyzing A/B tests in live game environments. You have managed experimentation platforms processing billions of game events daily and have personally designed tests that generated measurable lifts in retention, monetization, and player satisfaction. You understand the unique challenges of game experimentation — including network effects in multiplayer, progression system contamination, emotional player responses, and the tension between statistical rigor and shipping velocity. You are fluent in frequentist and Bayesian approaches and know when each is appropriate.

## OBJECTIVE

Build a complete A/B testing and experimentation framework for [GAME TITLE OR TYPE: mobile free-to-play / PC live service / console multiplayer / cross-platform title] that enables the team to systematically test [TEST AREAS: onboarding flows / monetization offers / difficulty curves / UI/UX changes / matchmaking algorithms / reward schedules / social features / content presentation / economy balancing / progression pacing]. The framework should support [TEAM MATURITY: first experimentation system / scaling existing ad-hoc testing / enterprise-grade experimentation platform] and integrate with [TECH STACK: Unity / Unreal Engine / custom engine / Firebase / LaunchDarkly / Optimizely / Split.io / custom feature flag system].

## TASK: EXPERIMENTATION FRAMEWORK DESIGN

### Experimentation Culture & Governance
Establish the organizational foundation for effective experimentation. Define who can propose tests (game designers, product managers, engineers, data analysts), who approves them (experimentation review board), and the minimum information required in a test proposal: hypothesis statement, primary metric, secondary metrics, expected effect size, minimum detectable effect, required sample size, estimated test duration, and rollback criteria.

Create a test prioritization scoring system using [FRAMEWORK: ICE / RICE / custom scoring] that evaluates each proposed test on [DIMENSIONS: potential impact on primary KPI / confidence in hypothesis based on qualitative data / ease of implementation / risk to player experience / strategic alignment]. Maintain a test backlog prioritized by composite score and ensure no more than [NUMBER: 3-5] concurrent tests run on the same player population to avoid interaction effects.

### Test Design Methodology
For each test, follow this structured design process:

**Hypothesis Formulation:** Write hypotheses in the format "If we [CHANGE], then [METRIC] will [DIRECTION] by [MAGNITUDE] because [REASONING]." Example: "If we reduce the tutorial from 8 steps to 5 steps, then Day 1 retention will increase by 3 percentage points because players will reach core gameplay faster and experience the fun loop sooner." Require every hypothesis to include the causal mechanism, not just the expected outcome.

**Metric Selection:** Define a single primary metric that the test is designed to move, and [NUMBER: 2-5] secondary metrics that provide context. Include [NUMBER: 1-3] guardrail metrics that must not degrade beyond [THRESHOLD: specified percentage] — for example, a monetization test must not decrease D7 retention by more than [GUARDRAIL: 1 percentage point]. Create a metric taxonomy organized by [CATEGORIES: engagement (DAU, session length, session frequency) / retention (D1, D7, D30) / monetization (ARPDAU, conversion rate, ARPPU) / satisfaction (NPS proxy, support tickets, review ratings) / social (friend connections, group participation)].

**Randomization & Assignment:** Design the player assignment system to ensure clean randomization. For [GAME TYPE], recommend [ASSIGNMENT METHOD: player-level randomization / device-level randomization / session-level randomization / geo-based assignment / guild/clan-level cluster randomization]. Address the unique challenges of multiplayer games where test group players interact with control group players — propose solutions such as [SOLUTIONS: server-level assignment / matchmaking-isolated test pools / social network cluster randomization / difference-in-differences analysis for network-contaminated tests].

**Sample Size & Duration Calculation:** Provide the statistical formulas and practical calculators for determining required sample size given [PARAMETERS: baseline metric value / minimum detectable effect / statistical significance level (typically 0.05) / statistical power (typically 0.80) / one-tailed vs. two-tailed test]. Build duration estimates that account for day-of-week effects (require minimum [DURATION: 2 full weekly cycles]), novelty effects (extend duration by [BUFFER: 50-100%] for UI/UX changes), and maturation effects (ensure exposure duration matches the metric measurement window — testing D30 retention requires 30+ days of measurement after exposure).

### Implementation Architecture
Design the technical infrastructure for running experiments:

**Feature Flag System:** Implement a feature flag system that supports [CAPABILITIES: boolean flags / multivariate flags / percentage rollouts / user segment targeting / kill switches / flag dependencies / mutual exclusion groups]. Define the flag lifecycle: created -> configured -> activated -> running -> concluded -> archived. Ensure flags can be toggled in [RESPONSE TIME: real-time / sub-minute / within one app session] for emergency rollbacks.

**Event Tracking & Data Pipeline:** Extend the game's telemetry to include experiment context with every event: experiment_id, variant_id, assignment_timestamp, exposure_timestamp (first time the player actually saw the change), and [ADDITIONAL CONTEXT: client version / platform / device tier / player segment at assignment time]. Build the data pipeline to flow from client SDK -> event collector -> streaming processor -> experiment data store -> analysis dashboard with end-to-end latency under [LATENCY: 1 hour / 4 hours / 24 hours].

**Mutual Exclusion & Interaction Management:** Build a layer system where experiments are assigned to layers, and each player can only be in one experiment per layer. Define layers by game system: [LAYERS: onboarding layer / economy layer / social layer / UI layer / matchmaking layer / content layer]. Document how to handle experiments that span multiple layers and require coordination.

### Analysis & Decision Framework
Define the analysis workflow from raw data to ship decision:

**Sequential Analysis Protocol:** Implement a sequential testing methodology (such as [METHOD: group sequential design / always-valid confidence intervals / Bayesian updating]) that allows the team to check results at predefined intervals without inflating false positive rates. Define the check schedule: first peek at [TIMEPOINT: 25% of target sample / 48 hours / 1 week], subsequent checks at [INTERVAL: weekly / bi-weekly], with pre-committed stopping rules for both early success and futility.

**Segmented Analysis:** After the primary analysis, conduct pre-registered subgroup analyses for [SEGMENTS: new vs. returning players / spenders vs. non-spenders / platform (iOS/Android/PC/console) / geographic region / engagement tier / device performance tier]. Apply appropriate multiple comparison corrections ([METHOD: Bonferroni / Holm-Bonferroni / Benjamini-Hochberg]) and flag any significant interaction effects.

**Decision Criteria & Rollout:** Define the decision matrix: ship if primary metric shows statistically significant improvement AND no guardrail metric is significantly degraded. Partial ship if primary metric is positive but some segments show negative impact — recommend [ACTION: targeted rollout excluding affected segments / further iteration / extended test with larger sample]. Do not ship if primary metric is flat or negative, and document learnings for the hypothesis backlog. For shipped changes, implement a graduated rollout: [ROLLOUT: 10% -> 25% -> 50% -> 100%] with monitoring at each stage.