Senior Data Scientist — Data scientist interview questions

How do you design an experimentation platform that serves multiple product teams running tests simultaneously?

The core problem is interference: teams running overlapping experiments on the same users can contaminate each other's results, so you need a layered assignment system — orthogonal layers or a hash-based bucketing scheme that lets independent experiments run on the same traffic without collision, plus guardrails that flag when two experiments touch a metric in ways that could interact. Beyond assignment, the platform needs automated sample ratio mismatch detection, standardized power calculations built into the launch flow, and a results pipeline nobody can quietly bypass by peeking at raw numbers before the test concludes — the platform's job is making the statistically correct path also the easiest path.

A stakeholder asks you to prove that a marketing campaign caused a revenue increase, and a randomized experiment was never run. How do you approach it?

You reach for the closest quasi-experimental design the data supports: difference-in-differences if there's a comparable untreated group and a clean pre-period, synthetic control if a single untreated comparison group doesn't exist but a weighted combination of others can approximate one, or regression discontinuity if the campaign targeted users above some threshold. Every one of these rests on an assumption — parallel trends, no anticipation effects, no simultaneous confound — that you can't prove, only argue for with pre-period diagnostics, and the deliverable has to include that assumption explicitly rather than presenting a point estimate as settled fact.

How do you translate an ambiguous business question like "why did retention drop last quarter" into a measurable analysis?

Start by forcing precision on "retention" and "drop" — which cohort, which time window, compared to what baseline — because half of ambiguous requests dissolve once the metric is actually defined, and you often find the stakeholder was reacting to a dashboard artifact rather than a real change. From there you decompose the metric (is the drop concentrated in one segment, one channel, one release?) before reaching for causal tools, because segmentation usually narrows an open-ended question into a specific, testable hypothesis faster than any model would.

How do you build data science standards and review practices across a growing team without becoming a bottleneck?

Codify the failure modes you've seen repeat — leakage, underpowered tests, misread p-values, metrics that don't match business cost — into a lightweight review checklist rather than requiring every analysis to pass through you personally, because personal review doesn't scale past a handful of people and creates a queue that trains everyone to skip it under deadline pressure. Pair that with shared tooling (a standard power-calculation library, a template for experiment write-ups) so the correct practice is also the path of least resistance, and reserve your personal review time for decisions with real financial or strategic weight rather than routine analyses.

A/B test says a change wins; a longer observational analysis says it hurts retention. How do you reconcile that?

Check the mechanics before trusting either number: novelty and primacy effects fade over a test's short window but show up in longer observational data, and a metric shift concentrated in a segment the test wasn't powered to detect gets averaged away in the top-line result. Look for interference — did the change affect the control group indirectly, through a shared resource, a marketplace, or a recommendation model retrained on both arms' data — since that breaks the independence assumption A/B testing relies on. If the mechanics check out and the conflict is real, trust the randomized result for causal effect and the observational one for who it affects and why, and say so explicitly rather than picking whichever number matches the decision you already wanted to make.