Treatment Effect Attribution

Overview

This skill validates whether an observed treatment effect (e.g., green channel vs normal channel, premium vs standard service, special program participants vs non-participants) is genuinely caused by the treatment itself, or merely reflects self-selection bias — the tendency for high-intent users to self-select into the treatment group. It uses a 5-phase methodology that works with observational (non-experimental) data, progressively narrowing the causal attribution from correlation to near-causal confirmation.

When to Use This Skill

Trigger this skill when any of the following conditions apply:

1. User compares a "treated" group vs "untreated" group and questions whether the observed difference is real or just self-selection
2. User has conversion rate / performance data split by channel, program, or service tier
3. User asks "is X really effective, or just attracting better users?"
4. User wants to validate ROI or cost-effectiveness of a premium channel/program
5. User has observational data (no random assignment) and needs to approximate causal inference

Typical scenarios: green/premium channel analysis, VIP service evaluation, marketing channel attribution, special program effectiveness validation, pricing tier comparison.

Input Requirements

The skill requires the following data structure in the input file (Excel/CSV):

Minimum Required Columns

Ideal Additional Columns (for deeper analysis)

• Per-user cost data (for ROI calculation)
• User-level intent signals (interaction behavior, trial duration, etc.)
• Multiple outcome metrics (conversion rate, revenue, retention)

Data Granularity

Data should be at the entity-time-treatment level (e.g., one row per teacher-month-channel). User-level data is even better but not required.

Workflow: 5-Phase Analysis

Execute phases in order. Each phase builds evidence; skip a phase only if data is insufficient.

Phase 1: Basic Effect Comparison

Calculate and present the treated vs untreated comparison:

1. Sum treated and untreated attendance and conversions across all entities and time periods
2. Calculate conversion rates: conversion_rate = conversions / attendance
3. Calculate absolute lift: treated_rate - untreated_rate
4. Calculate relative lift: (treated_rate - untreated_rate) / untreated_rate
5. Calculate treated group's contribution share: treated_conversions / total_conversions
6. Calculate treated group's attendance share: treated_attendance / total_attendance
7. If cost data available, calculate break-even cost premium ratio: 1 / relative_lift (the maximum cost premium the treatment can sustain while remaining ROI-positive)

Present results in a summary table. Flag the break-even ratio as the key decision threshold.

Phase 2: Self-Selection Bias Detection (5 Dimensions)

This is the core of the skill. Run all 5 tests; each produces a verdict that feeds into the final synthesis.

Test 1: Allocation Ratio Distribution

• For each entity-time combination, calculate treatment attendance share: treated_attendance / (treated_attendance + untreated_attendance)
• Compute the distribution statistics: mean, standard deviation, min, max
• Interpretation: If std > 20pp, the treatment is selectively allocated (potential self-selection at the entity level). If std < 10pp, allocation is relatively uniform.

Test 2: Baseline Ability vs Treatment Allocation Correlation

• For each entity, calculate untreated conversion rate as "baseline ability" proxy
• Calculate the entity's treatment attendance share
• Compute Pearson correlation between baseline ability and treatment share
• Interpretation: If r > 0.3 (positive), high-ability entities receive more treatment resources → entity-level selection bias confirmed. If r near 0 or negative → no systematic entity-level selection.

Test 3: Dilution Test — Treatment Volume vs Treatment Conversion Rate

• For each entity, calculate treatment attendance volume and treatment conversion rate
• Compute Pearson correlation between treatment volume and treatment conversion rate
• Interpretation: If r < 0 (negative), smaller treatment groups have higher conversion → supports "cherry-picking" hypothesis (self-selection). If r > 0 (positive) → refutes self-selection; larger groups perform better, indicating the treatment process itself drives results, not selective curation.

Test 4: Allocation Stability Across Time

• For each entity appearing in multiple time periods, calculate the standard deviation of treatment attendance share across periods
• Compute the average across entities
• Interpretation: If average std > 10pp, treatment allocation fluctuates significantly → may be demand-driven (random variation). If < 5pp, allocation is systematic and stable.

Test 5: Controlled Comparison — Same Entity, Same Time

• For each entity-time combination that has BOTH treated and untreated data, calculate the conversion rate difference
• Count how many show treated > untreated vs treated <= untreated
• Calculate the average lift across all controlled comparisons
• Interpretation: If > 70% of controlled comparisons show treated > untreated, the treatment effect holds after controlling for entity and time → strong evidence of genuine effect.

Phase 3: Dilution Test — Scale Expansion Analysis

This test uses the time dimension to test whether scaling the treatment dilutes its effect:

1. For each time period, calculate the treatment attendance share (aggregate level)
2. For each time period, calculate the absolute conversion rate lift (treated - untreated)
3. Plot treatment share (x-axis) vs absolute lift (y-axis)
4. Calculate Pearson correlation
5. Interpretation: If r > 0 (positive correlation), scaling the treatment does NOT dilute the effect → refutes self-selection. If r < 0, scaling dilutes the effect → supports self-selection.

Key reasoning: Under self-selection, the treatment selectively curates high-intent users. Expanding the treatment (larger share) must include lower-intent users, diluting the effect. If the effect holds or grows as share expands, the treatment process itself is contributing.

Phase 4: Same-Entity Cross-Period Comparison

This test controls for entity identity while allowing time variation:

1. Identify entities that appear in at least 2 time periods with treatment data
2. For each entity, compare across periods:
   - Treatment attendance share change
   - Untreated baseline conversion rate change
   - Treatment conversion rate change
   - Absolute lift change
3. Look for patterns:
   - Same share, different lift → process/flow effect, not user composition
   - Share expands, lift expands → refutes self-selection
   - Share shrinks, lift expands → ambiguous (could be self-selection or process maturation)
4. Interpretation: If the majority of entities show "share expands → lift expands" or "same share → different lift", self-selection is refuted. The treatment effect varies with process maturity, not user curation.

Phase 5: Evidence Synthesis & Conclusion Framework

Synthesize all evidence into a structured verdict:

Evidence Summary Table

Conclusion Tiers

• Near-Causal Confirmation: 4+ tests refute self-selection, controlled comparison > 70%, dilution test positive
• Strong Evidence: 3 tests refute self-selection, controlled comparison > 60%
• Moderate Evidence: 2 tests refute self-selection, controlled comparison > 50%
• Insufficient Evidence: < 2 tests refute self-selection — recommend A/B testing

Cost Decision Framework

If the treatment effect is validated, provide a cost-benefit decision:

• Calculate the relative lift ratio: treated_conversion_rate / untreated_conversion_rate
• The break-even cost premium = relative_lift_ratio - 1 (as a percentage)
• If actual cost premium < break-even → invest more
• If actual cost premium ≈ break-even → marginal, monitor closely
• If actual cost premium > break-even → do not invest further

Residual Confounders (always disclose)

Even with strong evidence, acknowledge:
1. Time-trend confound: both share and lift may grow with time independently
2. Unobserved user characteristics: user-level intent signals not captured in entity-level data
3. Process maturation vs user composition: cannot fully disentangle without randomization

Recommend A/B testing as the definitive solution when feasible.

Output Format

The skill should produce:

1. Summary table of treated vs untreated metrics with lift calculations
2. 5-dimension self-selection test results with correlation coefficients and verdicts
3. Dilution test scatter plot data and interpretation
4. Same-entity cross-period comparison table with key patterns highlighted
5. Evidence synthesis table with final verdict tier
6. Cost decision framework with break-even calculation (if cost data available)
7. Residual confounders disclosure and recommended next steps

Visual outputs (charts, scatter plots) are encouraged when the data supports them.

Limitations

1. Entity-level data only: This methodology works with aggregated data (entity-time-treatment level). It cannot fully control for user-level self-selection within the same entity-time cell.
2. No randomization: Without A/B testing, residual confounders always exist. This skill narrows the plausible confounders but cannot eliminate them entirely.
3. Time-trend confound: When both treatment share and treatment effect grow over time, a time trend (market improvement, process maturation) may explain part of the correlation. Same-entity cross-period comparison partially addresses this but cannot fully rule it out.
4. Correlation is not causation: All tests use correlation-based logic. Strong refutation of self-selection is not the same as proving causation.
5. Sample size sensitivity: Small samples (few entities, few time periods) reduce statistical power. Recommend at least 5 entities and 3 time periods for meaningful analysis.
6. Binary treatment only: The methodology assumes a binary treated/untreated split. Continuous treatment intensity requires different approaches (regression discontinuity, instrumental variables).

When to Recommend A/B Testing Instead

This skill's methodology is a "best available approximation" for observational data. Recommend definitive A/B testing when:
- The business decision is high-stakes (large budget commitment)
- User-level randomization is feasible (same teacher, same day, random assignment)
- The observational analysis yields "Moderate" or "Insufficient" evidence
- User-level intent signals are available for propensity score matching