01 · Experiment Design
The question, the groups, the hypothesis
Users were split into two groups: the ad group (treatment — sees marketing ad) and the psa group (control — sees public service announcement). The binary outcome is conversion. A one-sided test is used because the business question is directional: does the ad perform better?
H₀ — Null Hypothesis
The conversion rate of the ad group is less than or equal to the PSA group. The ad has no positive effect on conversion.
H₁ — Alternative Hypothesis
The conversion rate of the ad group is strictly greater than the PSA group. The ad drives real, measurable uplift.
⚠️ Limitation flagged: The experiment uses a 96%/4% allocation split (ad/PSA) — highly atypical for a standard A/B test. This may indicate an unequal design, a sample ratio mismatch, or a data collection issue. The analysis proceeds transparently with this constraint noted.
02 · Conversion Rate Comparison
The ad group clearly outperforms
Even before formal testing, the raw conversion rates show a consistent gap. The ad group converts at ~2.55% vs ~1.78% for the PSA group — an absolute difference of +0.77 percentage points.
Ad group: 2.55% conversion. PSA: 1.78%. A consistent 0.77pp lift across the full dataset.
TestTwo-proportion z-test (one-sided)
p-value<< 0.05 (highly significant)
95% CI (difference)+0.60% to +0.94%
Effect size+0.77 percentage points
Statistical power~1.0 (very high)
Odds ratio (PSA vs Ad)0.69 (31% lower odds)
ConclusionReject H₀ — Ad wins
✓
The marketing ad produces a statistically significant uplift in conversion. The p-value is far below 0.05, the 95% confidence interval excludes zero (+0.60% to +0.94%), and statistical power is near-perfect. The effect is real, robust, and not explained by temporal confounds.
03 · Confidence Interval & Regression
The effect holds after controlling for time
A key concern: what if the ad group was shown content at more favourable times? Logistic regression controlled for day of week and hour of day simultaneously. The result: PSA group membership remains a negative, statistically significant predictor of conversion even after adjusting for these factors.
Non-overlapping confidence intervals confirm statistical significance. The ad group's higher rate is a reliable, repeatable finding.
OR = 0.69: PSA users convert at 69% the rate of ad users, all else equal. The dashed line at OR=1 represents no effect.
04 · When to Show Ads
Saturday mornings and weekend evenings win
A secondary analysis within the ad group reveals which day-hour combinations produce the highest conversion rates — directly actionable for campaign scheduling and budget allocation.
Saturday 5–6 AM and weekend evenings (7–9 PM) show peak conversion. Late-night slots (2–4 AM across all days) are consistently the weakest. Reallocating budget from dead zones to peak windows improves campaign ROI without additional spend.
05 · Recommendations
Deploy the ad — and schedule it smartly
| Action | Supporting Insight |
|---|
Deploy Roll out the marketing ad at full scale |
The evidence is robust across three independent analyses: z-test (p << 0.05), CI (+0.60% to +0.94% excluding zero), and logistic regression controlling for temporal factors. PSA users are 31% less likely to convert. |
High Priority Concentrate spend on weekend peak slots |
Saturday 5–6 AM and weekend evenings (7–9 PM) outperform average conversion meaningfully. Shifting budget toward these windows increases returns without adding total spend. |
High Priority Eliminate budget in the 2–4 AM window |
Late-night slots show consistently the lowest conversion rates across all days. Every dollar saved here can be reallocated to high-performing Saturday and weekend evening slots. |
Future Experiments Rebalance group allocation in next test |
The 96/4 split limits statistical learning from the PSA group. A 50/50 or 80/20 split in future experiments would provide a cleaner, more statistically efficient comparison baseline. |
Monitor Track for novelty effects over time |
The experiment covers one campaign window. Monitor conversion lift monthly post-deployment — as the ad becomes familiar, novelty-driven lift may diminish and creative refresh cycles should be planned accordingly. |