Data engineering interviews lean hard on SQL and on whether your pipelines survive being run twice, late, or out of order. Expect modelling questions with no single right answer. Below are the questions asked most often, each with a model answer. Senior: architecture, trade-offs, mentoring, and decision-making.
Partitioning splits a table within one database so the engine can skip irrelevant chunks; sharding splits data across separate databases so each holds a subset. Partitioning is a performance and maintenance tool, sharding is a scale tool with real cost — cross-shard queries and transactions become hard or impossible. Reaching for sharding before partitioning and indexing are exhausted is a common and expensive mistake.
By choosing a format that carries schema and supports evolution — Avro or Parquet with a schema registry — and by deciding in advance which changes are compatible: adding an optional field usually is, removing or retyping one is not. The staging zone should accept the new shape without failing the whole load, and the failure mode to design for is a silent type widening that corrupts downstream aggregates.
Against named dimensions — completeness, uniqueness, validity, timeliness, consistency — each expressed as a check with a threshold rather than a feeling. Automation means the checks run as part of the pipeline and can stop it, because a quality report nobody reads is not quality control. The one people skip is timeliness, and stale data that looks correct causes more damage than data that is obviously missing.
Start with what changed: data volume, skew, cluster size, or the code. Then look at the stage view — one stage taking most of the time usually means a shuffle, and a few tasks taking far longer than the rest means skew. Skew is the answer more often than people expect, and the fixes are salting the key, broadcasting the small side, or repartitioning.
Bronze holds raw ingested data, silver holds cleaned and conformed data, gold holds business-level aggregates. The problem it solves is that transformations get tangled when everyone reads from wherever is convenient — with layers, lineage is legible and a bad load can be reprocessed from bronze without re-ingesting from the source. It is a convention rather than a technology, and its value is entirely in being enforced.
Almost always for analytics, because a dashboard that is a minute behind changes nothing, and the cost of strong consistency across a distributed store is availability during a partition. You do not accept it where a downstream decision is financial or regulatory, or where two systems must agree on a number at the same instant. The important part is that the choice is stated, since the failure mode is discovering it during an audit.
In a real interview you speak under pressure. Cam asks these same questions, scores every answer, and shows exactly what to fix.
Practice a Senior Data Engineer interview →