MLE Interview Day Checklist: LLM Deployment Scenarios

MLE Interview Day Checklist: LLM Deployment Scenarios

TL;DR

Candidates who treat LLM deployment questions as trivia rounds get filtered out; the ones who treat them as system design with production constraints get offers. The hiring committee does not care if you memorized LoRA paper details—they care if you can ship a model that doesn’t bankrupt the company or expose it to liability. Your checklist must cover latency budgets, cost per token, fallback architectures, and observability, not just model selection.

Who This Is For

You are a machine learning engineer with 2-6 years of experience who has trained models but never owned the full deployment pipeline, or you are a senior MLE interviewing at a company where “LLM deployment” means serving 10,000+ RPM to paying customers rather than running a Colab notebook. You have likely seen questions about quantization or vLLM on Blind or in your recruiter’s prep materials, but you do not yet have a reproducible framework for walking through a deployment scenario under interview pressure. You need judgment, not more papers to read.

What Do Interviewers Actually Test in LLM Deployment Scenarios?

They test whether you have ever been woken up by a PagerDuty alert at 3 AM for a model you shipped.

In a Q3 debrief at a late-stage AI infrastructure company, the hiring manager pushed back on a candidate with a PhD from a top program because the candidate described deploying a 70B parameter model on a single A100 with naive batching and called it “production-ready.” The hiring manager’s exact words in the feedback doc: “Has not felt production pain.” The candidate had optimized perplexity on benchmarks. They had not optimized dollars per thousand tokens, tail latency at the 99.9th percentile, or the blast radius of a bad rollout.

The first counter-intuitive truth is this: the interviewer is not your collaborator. They are your adversary in the best sense—they want to see where your plan breaks. When they ask “what about cost,” they do not want you to say “we can optimize later.” They want you to have already calculated that serving a 70B model at 2,000 tokens per request on full-precision GPUs burns $0.08 per request, and that your competitor using a distilled 7B model with speculative decoding serves the same user query for $0.003.

Here is the framework that passes debriefs. State your latency budget first: “We need first-token latency under 200ms and inter-token latency under 50ms for this use case.” Then your throughput target: “We need 5,000 RPM with burst to 15,000.” Then your cost ceiling: “We cannot spend more than $50,000 per month on inference at this volume.” Only then do you touch model selection. This ordering signals that you have negotiated with a product manager who cares about user experience and a finance partner who cares about gross margin. Candidates who start with “I would use Llama 3 because it scores well on MMLU” signal they have never had that conversation.

The specific numbers that matter: first-token latency (200-400ms for chat, sub-100ms for autocomplete), target throughput (measure in tokens per second per GPU, typically 3,000-10,000 for optimized vLLM on H100), and cost per million tokens ($0.50-$2.00 for hosted APIs, $0.10-$0.40 for self-hosted optimized inference). If you cannot cite ranges, you have not done enough production work to be credible at the senior level.

How Do I Structure My Answer When Asked to Design an End-to-End LLM Deployment System?

You structure it as a risk-prioritized walkthrough, not a feature list.

In a debrief last year for a Series C company’s staff MLE role, two candidates both mentioned KV cache optimization. The one who got the offer described it like this: “If our context window is 8K tokens and batch size is 8, we need 8 × 8K × hidden_dim × precision_bytes just for KV cache. At batch 32, this OOMs on A100 80GB, so we either reduce max context, quantize cache to FP8, or move to H100 with 96GB and better memory bandwidth.” The other candidate said “we should use KV cache to speed things up.” The difference was not knowledge. It was the ability to quantify tradeoffs under constraint.

Your architecture answer must have five layers in this order:

1. Serving layer: API gateway, rate limiting, request routing. Mention specific tools only if you can explain their failure modes—“NGINX with Lua for rate limiting” beats “some load balancer” only if you can discuss what happens when your rate limiter becomes the bottleneck.

2. Model execution layer: inference engine, batching strategy, scheduling. Contrast static batching with continuous batching. Name vLLM, TensorRT-LLM, or TGI only if you can explain why continuous batching improves throughput at the cost of predictability in latency.

3. Model storage and loading: model sharding, weight loading from S3 or parallel file system, cold start time. The specific failure mode: a 70B model takes 30-60 seconds to load from S3 to GPU memory. Your first request after a pod restart times out. What is your mitigation?

4. Observability: logging, metrics, tracing. Not “we need monitoring.” Specifically: track time-to-first-token, time-between-tokens, queue depth, GPU memory utilization, and model output quality drift. The candidate who mentioned “perplexity spike detection on a held-out validation set refreshed hourly” got marked as “has operated systems” in the feedback.

5. Fallback and degradation: circuit breakers, model cascading, graceful degradation. “If the 70B model is down or slow, we route to a 7B model with a prompt engineering layer that preserves 80% of quality but serves at 10x the speed.” This is not a footnote. This is the difference between a system that survives a weekend and one that does not.

The second counter-intuitive truth: your interviewer has a production incident in mind. They are testing whether you can predict it. In one debrief, the hiring manager described a real outage: a batch size increase to improve throughput caused GPU memory fragmentation under continuous batching, leading to OOM kills during traffic spikes. The candidate who proactively discussed memory fragmentation and pre-allocated buffers got the hire vote. The candidate who said “we can tune batch size” did not.

What Specific LLM Deployment Tradeoffs Do I Need to Verbalize Under Pressure?

The tradeoffs that matter are not model quality versus speed. They are business survival versus technical elegance.

The third counter-intuitive truth is that quantization is not automatically good. In a debrief for a financial services client, a candidate advocated 4-bit quantization for a compliance document analysis use case. The hiring manager asked: “What happens when a quantized model hallucinates a regulatory requirement that does not exist, and your quantization noise contributed to the error?” The candidate had no framework for quantifying that risk. They were rejected not for being wrong, but for treating deployment as purely technical.

Here are the specific tradeoffs to verbalize:

Latency versus throughput: speculative decoding reduces latency for single requests but can hurt throughput under heavy load because the draft model consumes GPU cycles. State this. Do not just name the technique.

Cost versus quality: fine-tuning a smaller model to match a larger one costs $5,000-$20,000 in compute and engineering time but reduces serving cost by 60-80%. When does that payback period make sense? At 10 million requests per month, typically within one quarter.

Customization versus maintainability: a custom CUDA kernel for your attention mechanism buys 15% throughput. It also means only you can debug it when the next CUDA version breaks the build. The candidate who said “I would start with vLLM’s optimized kernels and only customize if profiling showed it was the bottleneck” demonstrated engineering judgment.

Multi-tenancy versus isolation: serving multiple customers from the same model instance improves utilization. It also creates side-channel attacks where one customer’s prompt patterns leak through timing. The candidate who mentioned this in a security-sensitive role got marked “exceptional.”

The script for this section, verbatim: “The first question I would ask is what happens if this model produces the wrong output. If it’s a creative writing assistant, wrong means off-brand. If it’s a medical triage system, wrong means liability. My deployment architecture changes completely based on that answer.” This is not a deflection. It is the correct first question, and interviewers at companies that have faced real liability have told me it is the signal they wait for.

How Do I Handle the “Your Model Is Too Slow” Follow-Up Question?

You demonstrate that you have profiled before optimizing, not guessed at solutions.

In a debrief for a consumer AI company, the hiring partner described the following sequence. Candidate was given: “Your LLM endpoint has 99th percentile latency of 4 seconds. Users are churning.” The candidate who passed asked three questions before proposing anything: “What is the distribution between time-to-first-token and time-between-tokens? What is the current batch size and queue depth? What does the request pattern look like—are users sending 400-token prompts and expecting 2,000-token responses, or the reverse?”

This candidate identified that 70% of latency was time-between-tokens for long outputs, not time-to-first-token. They proposed chunked streaming with progressive disclosure in the UI, reducing perceived latency from 4 seconds to 200 milliseconds for the first meaningful token. Then they proposed speculative decoding for the actual throughput improvement, with a fallback to a smaller model if the draft model’s acceptance rate dropped below 80%. They estimated implementation time at two weeks. They got the offer.

The candidate who failed said: “We should use a faster model or quantize.” They had no diagnostic process. They had a bag of techniques.

Your diagnostic script: “I would start with distributed tracing to identify whether latency is in queuing, pre-processing, model forward pass, or post-processing. For LLMs, I would specifically check if we are GPU-bound or memory-bandwidth-bound using nvidia-smi and nsys profiling. If memory-bandwidth-bound, quantization helps. If GPU-compute-bound, speculative decoding or a more efficient attention implementation helps. If queuing, we need more replicas or better load balancing.” This specificity signals you have done this work, not read about it.

What Do I Need to Know About Deployment-Specific Failure Modes That General MLEs Miss?

The failure modes that kill LLM deployments are rarely about the model failing to predict the next token correctly.

In a post-mortem review that became a standard interview question at one company, an MLE team discovered that their model’s output quality degraded every Tuesday at 2 AM. The root cause: their vector database refresh job ran then, locking the retrieval-augmented generation pipeline and causing the system to fall back to the base model without context. The model was not broken. The deployment plumbing was.

Failure modes to verbalize:

Model weight corruption during rolling updates: two pods running different model versions because the new pod failed to download weights completely, causing inconsistent outputs for the same prompt. Mitigation: checksum validation on model load, not just on download.

Prompt injection through user inputs that exploit the system prompt: a deployment issue because your sanitization layer runs in a different service with race conditions. Not a model issue. A deployment architecture issue.

Context window exhaustion in long conversations: your conversation history management leaks tokens, and the 32K context window is actually 28K because of system prompt overhead, and at 29K tokens your truncation strategy drops the most recent user message instead of the oldest. The user sees the model ignore their last three turns. This is a deployment bug, not a model quality issue.

Token accounting errors: your billing system counts input tokens before your prompt guardrails strip PII, so you charge for tokens the model never processes. Your finance team finds this in a quarterly review. You find it when they escalate.

The script: “I would design the deployment with feature flags for model version, prompt template, and retrieval configuration, so that any change can be rolled back in under 30 seconds. I would run shadow traffic to new versions for 24 hours before any user-facing rollout. And I would validate that observability covers not just latency and error rate, but output quality drift using automated comparison against a golden test set.” This is the language of someone who has been burned.

Preparation Checklist

• Walk through three full deployment scenarios out loud, timing yourself to 45 minutes each, with a timer visible—pressure rehearsal changes performance more than reading does

• Build a reference spreadsheet with specific numbers for at least five model-size/hardware/cost configurations you can cite in seconds, not estimate

• Work through a structured preparation system—the PM Interview Playbook covers machine learning system design with real debrief examples, including the exact latency-throughput-cost tradeoff frameworks that separate senior MLE candidates from staff-level ones

• Write out your diagnostic scripts for “too slow,” “too expensive,” “wrong output,” and “system down,” each under 150 words, and practice them until they are automatic

• Profile a real open-source model deployment locally—quantization, continuous batching, and speculative decoding each with measured before/after numbers, not theoretical claims

• Prepare your “first three questions” for any deployment scenario, focused on business constraints, not technical implementation

• Rehearse explaining one complex failure mode from your own experience, or a public post-mortem, in under 90 seconds with specific technical and business impact

Mistakes to Avoid

Mistake one: Treating deployment as model selection with extra steps.

BAD: “I would evaluate several models on benchmarks and pick the one with highest score, then deploy it on a GPU.”

GOOD: “I would establish latency and cost constraints with stakeholders, then select the smallest model that meets quality requirements under those constraints, with a documented downgrade path.”

Mistake two: Discussing optimization without measurement.

BAD: “We can use quantization and speculative decoding to make it faster.”

GOOD: “I would profile with nsys to determine if we are compute-bound or memory-bandwidth-bound. If memory-bound, I would test INT8 quantization and measure perplexity degradation on our specific task. If compute-bound, I would evaluate speculative decoding with a draft model that achieves >80% token acceptance, measuring end-to-end latency improvement.”

Mistake three: Ignoring the human and organizational layers.

BAD: “The engineering team will handle monitoring.”

GOOD: “I would define SLOs with the product team—200ms first-token latency, 50ms inter-token, 99.9% availability—and set up paging thresholds with runbooks. I would schedule a weekly review of output quality samples with domain experts to catch drift before users report it.”

FAQ

What if I have never deployed an LLM to production—can I still pass?

You can pass if you demonstrate transferable judgment from analogous systems. In one debrief, a candidate with recommendation system deployment experience successfully translated their A/B testing, canary analysis, and rollback procedures to LLM contexts. They explicitly stated: “I have not shipped an LLM, but I have managed 50+ model version rollbacks with feature flags, and the principles are identical—validate, shadow, canary, full, with automated quality gates at each stage.” The hiring manager noted “learns fast, has operational discipline.” The candidate who pretended to LLM experience they did not have got caught in follow-up questions about continuous batching behavior and was rejected for misrepresentation.

How deep do I need to go on specific inference engines like vLLM or TensorRT-LLM?

Deep enough to explain their architectural tradeoffs, not their marketing claims. In a debrief, one candidate described vLLM’s PagedAttention as “reducing memory waste from fragmentation,” which any blog post could tell you. The candidate who passed said: “PagedAttention uses block tables analogous to OS virtual memory, which allows dynamic allocation but adds indirection cost. For our workload with very uniform sequence lengths, this overhead might not pay off, and I would benchmark against a simpler continuous batching approach.” This demonstrated they understood the mechanism, not just the headline. Name tools only to explain when they fit and when they do not.

Should I prepare for coding questions about LLM deployment, or is it all system design?

Expect both, with coding focused on the data structures and algorithms of serving, not model implementation. In one interview loop, candidates were asked to implement a simplified KV cache manager with eviction policies. The passing solution handled memory allocation, not attention math. Another common question: implement rate limiting with token bucket for an API gateway. The system design questions test judgment; the coding questions test whether you can implement the components you described. Prepare by writing a basic inference server with batching and queue management in Python, then optimizing the hot path. If you cannot build what you describe, you will fail the coding round even with perfect system design answers.amazon.com/dp/B0H2CML9XD).