Help me build a cross-team experiment branching toolkit: governance policies, rollout…
- User
Help me build a cross-team experiment branching toolkit: governance policies, rollout CLI design, version tracking, quality gates, developer communications, and success metrics.
- Assistant
Cross-Team Experiment Branching Toolkit
This toolkit provides a standardized framework for running experiments (e.g., A/B tests, feature flags, canary releases) across teams using Git branching strategies. It promotes safety, consistency, and scalability in a multi-team environment. The design assumes:
- Git as the source of truth (e.g., GitHub, GitLab).
- Branching model: Trunk-based development with experiment branches (e.g.,
exp/team/feature-v1.0). - Promotion pipeline: Dev → Staging → Prod via protected branches and CI/CD (e.g., GitHub Actions, Jenkins).
- Environments: Isolated per-experiment or shared with namespaces.
The toolkit is modular—implement components incrementally. I'll provide templates, code snippets, and best practices.
1. Governance Policies
Establish clear rules to prevent chaos. Document in a
docs/governance.mdrepo or Confluence page. Enforce via GitHub branch protection rules and pre-commit hooks.Key Policies
Policy Description Enforcement Branch Naming exp/{team}/{feature}-{version}(e.g.,exp/marketing/ab-test-v1.2). Limit length < 50 chars.Git hooks or CI linting. Ownership Experiments owned by a "squad" (team + stakeholders). Max 5 active per squad. Labels: squad:marketing,status:draft/active/archived.TTL Experiments expire after 30 days unless promoted. Scheduled GitHub Action to close stale PRs. Promotion Requires 2 approvals (owner + reviewer from another team). No direct prod pushes. Branch protection rules. Rollback Always merge to mainwith revert branch (e.g.,revert/exp-123).Required PR template. Cross-Team Access Read-only for non-owners; owners get merge rights via CODEOWNERS. GitHub teams/groups. Auditing All changes logged in #experimentsSlack channel.Webhooks. Template: governance.md
# Experiment Governance ## Prohibited - Hotfixes on exp branches. - Merging without quality gates. - >10 concurrent exps per repo. Violations: PR blocked + Slack alert to squad leads.Implementation Script (Bash hook for naming):
#!/bin/bash if [[ ! $BRANCH =~ ^exp/[a-z]+/[a-z0-9-]+\-[0-9]+\.[0-9]+$ ]]; then echo "Invalid branch name!" >&2 exit 1 fi2. Rollout CLI Design
A simple CLI (
expctl) for creating, promoting, and tearing down experiments. Built in Go/Bash/Python for portability. Install viapip install expctlor brew.CLI Commands
expctl init --team marketing --feature ab-test --version 1.0 # Creates branch, feature flag YAML, PR. expctl promote --from dev --to staging --branch exp/marketing/ab-test-v1.0 # Triggers CI promotion. expctl metrics --branch exp/marketing/ab-test-v1.0 # Fetches success metrics. expctl teardown --branch exp/marketing/ab-test-v1.0 --reason "Low uplift" # Archives branch, notifies team.Core Design Principles:
- Idempotent (safe to rerun).
- Dry-run mode (
--dry-run). - Config-driven (
.expctl.yamlwith GitHub token, Slack webhook).
Sample Implementation (Python with Click):
import click import git import requests # For GitHub API/Slack @click.group() def cli(): pass @cli.command() @click.option('--team', required=True) @click.option('--feature', required=True) @click.option('--version', required=True) def init(team, feature, version): branch = f"exp/{team}/{feature}-{version}" repo = git.Repo('.') repo.git.checkout('-b', branch) # Create feature flag YAML, open PR click.echo(f"Created {branch}") if __name__ == '__main__': cli()Config File (
.expctl.yaml):github: token: ${GITHUB_TOKEN} repo: myorg/myrepo slack: webhook: https://hooks.slack.com/... environments: - dev - staging - prod3. Version Tracking
Use semantic versioning (SemVer) on branches/tags for experiments. Track in a central
experiments.jsonfile.Strategy
- Branch:
exp/{team}/{feature}-v{major.minor.patch}. - Tags:
v{major}.{minor}.{patch}-{env}(e.g.,v1.0.0-staging). - Metadata File: Commit
experiments/{slug}.jsonwith history.
experiments/marketing-ab-test-v1.0.json:
{ "slug": "marketing-ab-test", "versions": [ {"version": "1.0.0", "status": "prod", "deployed": "2023-10-01", "owner": "alice@team"}, {"version": "1.0.1", "status": "staging", "deployed": "2023-10-02"} ], "traffic": {"dev": 100, "staging": 10, "prod": 1} }Tracking Script (GitHub Action):
name: Track Version on: push jobs: track: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - run: | jq '.versions += [{"version": "${{ github.ref_name }}", "status": "deployed"}]' experiments/${{ github.event.head_commit.message }}.json > tmp.json git add tmp.json && git commit -m "Track v${{ github.ref_name }}"Query via
git tag --list 'v*prod'or CLI.4. Quality Gates
Automated checks in CI/CD. Use GitHub Actions workflow
.github/workflows/promote.yml.Gates by Stage
Stage Gates Pass Criteria Dev Lint, Unit Tests, License Check 100% pass. Staging Integration Tests, Security Scan (Snyk), Perf Baseline >95% coverage, no vulns. Prod Canary (1% traffic), Synthetic Monitoring, Manual Approval Uptime >99%, no alerts. Sample GitHub Action:
name: Quality Gates on: workflow_dispatch: inputs: from: to: jobs: gates: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Tests run: npm test -- --coverage - name: Security uses: snyk/actions/node@master - if: inputs.to == 'prod' name: Canary run: kubectl rollout canary ...Block promotion on failure; notify via Slack.
5. Developer Communications
Keep teams aligned without meetings.
Channels/Tools
- PR Templates: Auto-generated with checklist (goals, metrics, rollback plan).
- Slack Notifications: Bot posts on init/promote/teardown.
- Dashboard: GitHub Project board or Looker dashboard for active exps.
- Docs:
README.mdper exp with "Why this exp?" and hypotheses.
PR Template (
.github/pull_request_template.md):## Experiment: {{feature}} **Hypothesis**: If X, then Y (+Z% uplift). **Metrics**: [Link to dashboard] **Rollback**: `git checkout main && git revert HEAD` - [ ] Quality gates passed - [ ] Cross-team reviewSlack Bot (via CLI integration):
requests.post(slack_webhook, json={ "text": f":rocket: Experiment *{branch}* promoted to {env}! <@{owner}>" })Weekly "Exp Sync" email digest via cron job.
6. Success Metrics
Measure experiment health and business impact. Track in Amplitude/Mixpanel or custom dashboard.
Core Metrics
Category Metrics Target Tool Adoption # Active exps, Time to prod, Rollback rate <7 days, <5% GitHub API Quality Test coverage, Bug rate post-deploy >90%, <1% CI logs Business Uplift (e.g., conversion), Traffic allocation +5%, 50/50 split Analytics Ops MTTR (mean time to recovery), Cost (CI minutes) <1h, <$10/exp Datadog Dashboard Query (PromQL example for Grafana):
sum(rate(experiments_promoted_total{status="success"}[5m])) / sum(rate(experiments_total[5m]))Success Criteria Template:
- Win: >3% uplift, p<0.05.
- Kill: No uplift after 1 week + 10k users.
- Hold: Monitor for 2 weeks.
Implementation Roadmap
- Week 1: Governance + PR templates.
- Week 2: CLI prototype + quality gates.
- Week 3: Version tracking + comms.
- Week 4: Metrics dashboard + pilot with 1 team.
- Scale: Roll to all teams; iterate based on feedback.
Repo Starter: GitHub Template. Ping me for customizations or code! 🕵️♂️
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.