Senior AI/ML Engineer — AI/ML engineer interview questions

Explain KV caching and quantization in LLM inference — why do they matter?

KV caching stores the key and value vectors computed for previous tokens so that generating each new token only requires computing attention for that one new token instead of recomputing the whole sequence — without it, autoregressive generation would be quadratically slower. Quantization reduces the precision of weights (from FP16 to INT8 or INT4, for example) to cut memory footprint and increase throughput, trading a small, usually acceptable amount of accuracy for a large reduction in serving cost. Both are what make serving a 70B-parameter model economically viable at all.

What is RLHF, and what problem does it actually solve?

RLHF (reinforcement learning from human feedback) takes a model already trained to predict the next token and further tunes it to produce outputs humans actually prefer — typically by training a reward model on human comparisons of outputs, then using that reward model to fine-tune the policy with an algorithm like PPO. It solves the gap between "predicts plausible text" and "is helpful, honest and follows instructions," which raw next-token prediction does not optimize for at all. Newer approaches like DPO get similar results without a separate reward model and RL loop, which is why teams increasingly skip the classic RLHF pipeline.

How do data, model and pipeline parallelism differ, and how do you decide between them?

Data parallelism replicates the whole model across devices and splits the batch, which is simple but requires the full model to fit on one device. Model parallelism splits the model itself across devices when it is too large to fit on one — tensor parallelism splits individual layers, pipeline parallelism splits the model into stages processed sequentially. In practice, training a large model uses all three at once, and the real engineering problem is minimizing the communication overhead between devices, which is what actually caps throughput.

How do you decide whether to build on a foundation model API, fine-tune an open model, or train from scratch?

Training from scratch is almost never justified outside a handful of well-funded labs — the data and compute cost is enormous and the base capability you would get is worse than what already exists. The real decision is API versus self-hosted open model: an API is faster to ship and has no infra to own, but costs scale linearly with usage and you depend on someone else's uptime and rate limits; a self-hosted fine-tuned model has upfront engineering cost but gives you data control, predictable unit economics at scale, and no dependency on an external vendor's policy changes. Data sensitivity and volume usually settle the argument faster than pure cost modeling does.

How do you decide between fine-tuning, retrieval-augmented generation (RAG), and prompt engineering alone?

Start with prompting — it is the cheapest to iterate on and often gets you 80% of the way for tasks that just need better instructions or examples. Reach for RAG when the model needs facts it was not trained on or that change often, since retrieval keeps the knowledge current without retraining anything and lets you cite sources. Fine-tuning is for teaching a consistent style, format, or specialized skill the base model gets wrong even with good prompting and context — and it is the most expensive to iterate on, since every change means a new training run and eval pass. Most production systems end up combining RAG for facts with a lightly fine-tuned model for tone and format, not picking just one.