Generative AI: The Questions Technical Teams Ask

These are the questions that arrive in the first month of building anything with a language model, usually in this order, and usually from people who have to sign off on it.

Behaviour

Why does it make things up?

Because generating plausible text is what it does. A language model produces a likely continuation of its input; it has no separate step that checks whether the continuation is true. When the answer is well represented in what it learned, the likely continuation is usually correct. When it is not, the model still produces a fluent, confident answer — because fluency and confidence are properties of the output, not of the knowledge behind it.

That framing matters, because it tells you what actually helps. Instructing a model not to make things up helps a little. Giving it the correct passage and asking it to answer only from that passage helps a great deal. The problem is not the model's attitude; it is the absence of the information.

Does a bigger context window mean I should send more?

No, and this is one of the most expensive misconceptions available.

Attention across a long context is uneven — material in the middle of a very long input is used less reliably than material at the beginning or the end. Sending twenty retrieved passages instead of five typically produces a worse answer for four times the input cost, because the relevant one is now diluted by nineteen distractors.

Retrieve widely, re-rank, then send few. The number of passages and their order are bigger levers than most prompt edits.

Why does the same prompt give different answers?

Sampling. Temperature and related parameters introduce randomness deliberately. Lower it for extraction, classification and anything that must be consistent; raise it where variety is the point. Even at the lowest setting, exact reproducibility is not guaranteed across model versions or infrastructure, so do not design as though it were.

Architecture

RAG or fine-tuning?

They solve different problems, and the question is usually asked as though they were alternatives.

• RAG changes what the model knows for a given question. Use it for facts, documents, policies, anything that changes, and anything that must be cited.
• Fine-tuning changes how the model behaves — its format, style, or handling of a narrow task. Use it when you need consistent output shape or a specialised task done cheaply by a smaller model.

Almost every "the model does not know our stuff" problem is a RAG problem. Fine-tuning a model on a document set is an expensive way to get worse retrieval, and the resulting model cannot cite anything or be updated without retraining.

When my RAG system answers wrongly, where do I look?

At retrieval, before the prompt. Print what was actually retrieved. In the large majority of cases the passage containing the answer was not among the chunks passed to the model — and no prompt change recovers information that was never supplied.

Once you have confirmed the right passage was retrieved and the answer is still wrong, then it is a generation problem, and then prompt work is the right response.

Do I need a vector database?

Not necessarily. For a small, well-titled corpus, keyword search with good ranking often performs well. Vectors earn their place when users phrase questions differently from how documents are written, which is most of the time at scale. Hybrid search — keyword and vector combined — beats either alone on realistic mixed workloads.

How should agents be structured?

With explicit state and explicit control flow. A loop that calls the model until it says it is finished is fine on your own machine and dangerous in production: it can cycle indefinitely, cannot be paused for approval, and cannot resume after a failure. Bound the iterations in code, put a human approval step in front of anything irreversible, and let code decide routing while the model decides intent.

Evaluation and cost

How do I know whether a change made it better?

Only by measuring on a fixed set of inputs. Without one, every change feels like an improvement — you check the examples you had in mind, they look better, and you ship a regression in three cases you were not thinking about.

The minimum viable version is fifty real inputs with expected characteristics, run before and after every change. Include cases where the correct behaviour is to refuse; those are what prompt edits break most often and what nobody checks.

What drives cost?

Input tokens, usually. Three things dominate:

• The system prompt, paid for on every single call.
• Retrieved context, which is why sending twenty passages instead of five is roughly four times the input cost.
• Model choice. Classification and routing rarely need the largest model, and sending cheap work to a cheap model is the biggest cost lever most systems have.

Set a budget alert on day one. The failure is not gradual drift; it is a retry loop that ran ten thousand times overnight.

Is our data used to train the model?

It depends entirely on the provider and the specific service tier, and it is exactly the question a security review will ask. Get the answer from the provider's current documentation and terms, and keep the link. Do not rely on a summary in an article — including this one.

Common mistakes

• Fixing retrieval failures by editing the prompt. Check what was retrieved first.
• Filling the context window because it is available. More context is not more accuracy.
• Fine-tuning to teach facts. Wrong tool; use retrieval.
• No evaluation set. Every later problem becomes harder to diagnose.
• No citations. Readers cannot verify, so they either over-trust or abandon it.
• Unbounded agent loops. A cost incident waiting for an ambiguous question.
• Treating prompt instructions as a security control. Authorisation belongs in the code behind the action.

Frequently asked questions

How do I stop it answering when it should not?

Apply a relevance threshold at retrieval and return an explicit "not found" when nothing clears it. A model handed irrelevant context will still produce an answer, and that is the worst output the system can generate.

Can I make it deterministic?

You can reduce variation substantially with sampling settings. You cannot rely on byte-identical output across model versions. Design tests around properties — valid JSON, contains a citation, correct classification — rather than exact strings.

How large should chunks be?

Split on document structure rather than character count, with a little overlap, and prefix each chunk with its title and heading path. That last detail measurably improves matching and costs almost nothing.

Should we build or buy?

Buy where the problem is generic and your data is not the differentiator. Build where the value comes from your own content, your own workflows or your own constraints. The interesting question is rarely the model; it is whether you have the content and the evaluation discipline to make anything work.

The one habit

If a team adopts a single practice from all of this, make it the evaluation set — built from real inputs, grown from real failures, run on every change. It is what turns a series of opinions about prompts into engineering, and it is the difference between a system that improves and one that just changes.