Middle Data Scientist — Data scientist interview questions

Walk through how you would choose features for a churn model and avoid leakage.

Start from what is knowable at prediction time — a feature computed from data that only exists after the outcome (like "days since cancellation email was opened," if that email only goes out after someone has already decided to churn) leaks the label into the input and gives you a model that looks great offline and fails in production. After ruling out leakage, you narrow the list with domain knowledge, correlation with the target, and something like permutation importance, then check for multicollinearity between the survivors rather than trusting a single importance score in isolation.

How does k-fold cross-validation work, and why is a single train/test split not enough?

You split the data into k folds, train on k-1 of them and validate on the remaining fold, then rotate through all k combinations and average the results. A single split gives you one noisy estimate that depends heavily on which rows happened to land in the test set — a lucky or unlucky split can make a mediocre model look great or a good model look bad. Cross-validation trades more compute for a much more stable estimate of how the model generalizes, and it's what you use to compare models honestly before picking one.

How would you design an A/B test for a change you suspect has a small effect on a low-traffic page?

A small expected effect on low traffic is the worst combination for statistical power, so the first move is checking whether the test is even feasible in a reasonable timeframe given the math, not designing it and hoping. If it's not feasible, the honest options are widening the eligible population, extending the runtime, switching to a more sensitive metric (a continuous proxy metric instead of a binary conversion), or accepting a larger minimum detectable effect and being explicit that smaller true effects will go undetected.

When would you use a window function versus a subquery to compute a running total in SQL?

A window function like SUM(amount) OVER (PARTITION BY customer_id ORDER BY order_date) computes the running total in a single pass without collapsing rows, and most query planners optimize it far better than the equivalent correlated subquery that re-scans prior rows for every output row. The subquery approach still shows up in older codebases and in interviews as the "naive" answer — recognizing when to replace it is itself a signal of SQL maturity.

You show a model with 92% accuracy to a product manager and they want to ship it. What do you check before agreeing?

First, the class balance — if 92% of the data is the majority class, a model that always predicts majority class also hits 92% accuracy while being useless, so precision, recall, and a confusion matrix per class matter more than accuracy alone. Second, whether the metric matches the actual business cost: for fraud detection a false negative is expensive and a false positive is a minor annoyance, so you'd tune the threshold toward recall even if it costs some raw accuracy, and you'd say so explicitly before anyone ships it.