There's a conversation that happens in almost every AI project kickoff. Someone asks which model to use. GPT-4o or Claude? Do we need fine-tuning? Should we self-host something open-source? The team debates it seriously, like the answer will determine whether the project succeeds or fails.
I've been building AI products for a while now — shipping real features to real users, watching them break in unexpected ways, fixing them, and shipping again. And I can tell you with some confidence: the model is the smallest problem you will face.
This is not what anyone wants to hear. The model feels like the core decision. It's the thing with the brand name, the benchmark scores, the pricing page everyone pores over. But in practice, the model is also the part that works. The hard parts are everything you build around it.
The prompt is a product.
The first thing that surprises teams is how much the prompt matters — and how hard it is to maintain. A prompt isn't a config value you set once. It's a contract between your application and the model. It encodes assumptions about input format, output structure, tone, edge case handling, and failure modes. When those assumptions change — when a new input type appears, when users start doing something unexpected, when you update the model version — the prompt breaks.
And when a prompt breaks, it usually breaks quietly. The model still returns something. It just returns the wrong thing. In a way that looks right.
I've seen teams ship a feature and then spend the next two weeks doing something that isn't in any job description — prompt archaeology. Trying to figure out why a prompt that worked last week is producing subtly different outputs now. Was it the model update? The input change? The upstream data pipeline that someone quietly modified?
Good prompt engineering is a discipline, not a one-time task. It needs version control, evaluation frameworks, and someone who owns it. Most teams don't realise this until they're already debugging production.
The data is worse than you think.
Every AI feature makes assumptions about its inputs. The demo data is clean because you chose it to be clean. The production data is a different situation entirely.
We built an AI feature once that worked beautifully on our test set. Then we connected it to real data from real users and watched it produce confident, detailed, completely wrong outputs for a significant percentage of requests. The inputs were malformed in ways we hadn't anticipated — missing fields, encoding issues, legacy formatting from a system that predated the current schema by a decade.
The model wasn't broken. The model was doing exactly what models do — generating plausible-sounding output given whatever it received. The problem was that nobody had validated the inputs before handing them to the model. We had treated the model as a black box that could handle anything. It can't. Nothing can.
Input validation for AI features is harder than for traditional software because the failure modes are fuzzier. With a traditional system, bad input usually causes an error. With an AI system, bad input usually causes bad output — which looks exactly like good output until someone reads it carefully.
The cost model is not an afterthought.
I've watched teams build AI features with no cost model and then get the first production invoice. The numbers are always surprising. Not because the per-call costs are outrageous — they're generally not — but because nobody modelled what happens at scale.
A feature that costs $0.002 per call looks almost free. At 500 calls per day, that's $30/month. Fine. But if the feature gets used more than expected, or if someone discovers that your API is being called in a loop somewhere, or if you move to a more capable model and forget to recalculate — the number changes.
Cost and latency need to be first-class engineering concerns, modelled before launch and monitored in production. Not because AI is uniquely expensive, but because the cost structure is different from what most engineers are used to: it scales directly with usage in ways that traditional software costs often don't.
Latency is a UX problem, not an infrastructure problem.
The average inference time for a modern language model is somewhere between one and ten seconds, depending on the model, the prompt length, and the output length. That range is not acceptable for most synchronous user-facing interactions.
So what do you do? You stream. You show partial results. You move the AI call out of the critical path. You cache aggressively. You design the UX around the latency rather than pretending the latency doesn't exist.
All of that is real engineering work, and none of it is about the model. I've seen beautiful AI features get poor user feedback not because the outputs were bad — they were excellent — but because the wait felt too long and the UI gave no feedback while the user waited. The model was working perfectly. The experience was broken.
The model you pick matters less than you think.
After all this — the prompt maintenance, the input validation, the cost modelling, the latency design — the model choice is usually a relatively minor decision. The top-tier models from the major providers are genuinely good and getting better fast. For most production use cases, the difference between models is smaller than the difference between a well-designed integration and a poorly-designed one.
Pick a model that fits your latency requirements, your cost constraints, and your data privacy requirements. Evaluate it on your actual inputs, not on generic benchmarks. Have a migration path if you need to switch.
And then spend the rest of your time on the parts that actually determine whether the feature succeeds: the prompt design, the input validation, the error handling, the fallback logic, the monitoring, and the UX.
The model is the easy part. Build the rest of it well.