Generative AI Interview Questions With Real Answers

Generative-AI interviews often drift into model trivia: transformer internals, parameter counts, benchmark scores. Very little of that predicts whether someone can build a system that works.

The questions below do, because each one has a wrong answer that sounds right. They are written so they are useful for preparing as well as for asking.

Retrieval

Your RAG system gives a confidently wrong answer. Walk me through diagnosing it.

The best single question in this list.

Weak answer: goes straight to the prompt.

Strong answer: looks at what was retrieved first. Was the passage containing the answer among the chunks passed to the model? If not, it is a retrieval failure and no prompt change will help — investigate chunking, the query, hybrid search and filters. If it was retrieved and the answer is still wrong, then it is a generation problem and prompt work is appropriate.

Follow-up: how would you measure retrieval quality? Answer: a set of questions with known source documents, and the proportion where the right source appears in the top results.

How do you chunk documents, and why does it matter?

Strong answer: split on structure — headings, sections, paragraphs — not on a fixed character count, because a window that cuts mid-table produces chunks nobody can use. Small overlap so an answer straddling a boundary is not lost by both. Prefix each chunk with its document title and heading path so the chunk states what it is about. Handles tables and code deliberately rather than letting a generic splitter destroy them.

What it reveals: whether they have looked at their own chunks. Anyone who has will mention tables unprompted.

Vector search, keyword search, or both?

Strong answer: both. Vector search handles paraphrase and misses exact identifiers — error codes and part numbers mean nothing semantically. Keyword search is the reverse. Real question sets are mixed, so hybrid wins. A strong candidate adds that filters are correctness rather than tuning: security trimming must happen in the query, not after retrieval.

Evaluation

How do you know a prompt change improved things?

Weak answer: tried some examples, looked better.

Strong answer: a fixed dataset of real inputs, run before and after, with a mix of deterministic checks and model-graded ones. Includes cases where the correct behaviour is refusal, because those are what prompt edits silently break. Notes that a change improving one metric while degrading another is a trade to make consciously.

What would you measure in production?

Strong answer: retrieval hit rate, groundedness, correct refusal rate, tool-selection accuracy, tail latency and tokens per request. Explains why each matters. A candidate who lists only accuracy has not run one of these systems in front of users.

How do you use a model as a grader without fooling yourself?

Strong answer: calibrate it — sample its judgements against your own until you know how far to trust it — and put cheap deterministic checks first because they are faster, cheaper and unambiguous. Knows that an uncalibrated model-graded metric is a number of unknown meaning.

Agents

How do you stop an agent looping forever?

Strong answer: a bound in code — an iteration counter and a hard exit path — not an instruction in the prompt. Explains that the model deciding when to stop is exactly what fails on an ambiguous task, and that the cost of an unbounded loop is unbounded.

An agent can send emails and issue refunds. How do you make that safe?

Strong answer: a human approval step before the irreversible action, with the pending action shown in specific terms — amount, recipient, date. Authorisation enforced in the code behind the action, not in the prompt, because instructions are guidance and someone will try to talk the agent out of them. Idempotency so a retry does not refund twice. Full logging of what was proposed and what was approved.

What it reveals: whether they treat prompts as a security boundary. Anyone who does has not thought about an adversary.

When is an agent the wrong architecture?

Strong answer: when the task is a fixed sequence. A chain is cheaper, faster and far easier to debug. Agents earn their complexity when the path genuinely varies with the input.

Engineering

What drives cost, and how do you reduce it?

Strong answer: input tokens usually dominate. The system prompt is paid on every call; retrieved context scales with how many passages you send; model choice is the biggest lever, and routing cheap work to a cheaper model is the standard win. Mentions bounding output length and setting a budget alert.

How do you handle provider failures?

Strong answer: retry with exponential backoff and jitter, treating rate limits as a signal rather than an error; timeouts so a request cannot hold a page open; a degraded path that returns something useful; and a decision about whether a fallback model is acceptable for the use case.

What do you do about prompt injection?

Strong answer: assume any retrieved or user-supplied content may contain instructions. Do not grant the model authority it should not have; enforce authorisation in the tools rather than the prompt; separate instructions from data as clearly as the API allows; and treat model output that will be rendered or executed as untrusted input. A candidate who says "I would tell it to ignore instructions in the documents" has not understood the problem.

Judgement

When would you tell a stakeholder not to use an LLM?

Strong answer: when the answer must be exactly right and verifiable — arithmetic, entitlement calculations, anything regulated; when a deterministic rule or a search box does the job; when the data needed does not exist in usable form; and when nobody will own the evaluation. Willingness to answer is the signal.

Tell me about something that failed in production.

What you want is a specific failure, a specific diagnosis, and a specific change. "The retrieval was returning a superseded policy, we added an effective-date filter and put that case in the evaluation set" is the shape of a good answer.

Common mistakes candidates make

• Answering retrieval questions with prompt techniques. The clearest single discriminator.
• Talking about models rather than systems. The model is rarely the hard part.
• No evaluation story. If they cannot say how they knew something improved, they did not know.
• Treating prompts as security. Instructions are not enforcement.
• Quoting benchmark scores. Benchmarks say little about performance on a specific task with specific prompts.

Frequently asked questions

How much machine-learning theory is needed?

For application work, less than people assume. Understanding embeddings, context, sampling and tokenisation covers most decisions. Deep architectural knowledge matters for research roles, not for building systems.

Is prompt engineering a real skill?

A real but small one, and the least transferable part of the job. Retrieval design, evaluation discipline and system architecture are where the durable skill is.

What should a candidate ask?

Whether there is an evaluation set, how quality is measured, who owns the content being retrieved, and what happened in the last incident. The answers reveal whether the team is engineering or experimenting.

What is the most common gap?

Evaluation. Most candidates can build a demo. Far fewer can describe how they would tell whether it got better — which is the whole difference between a prototype and a product.

Preparing for one

Build something small with retrieval, then build the evaluation set for it and use the set to make a decision you would otherwise have made on instinct. That single exercise answers most of the questions here, and it is the experience interviewers are actually probing for.