We were building a layer over a commercial ERP so a manager could ask questions in plain sentences: “which product sold most last month”, “what is this customer’s payment status”. The product became READERP.
In a product like that the real work is not the model but the integration: the API is documented behind closed doors, field names are full of abbreviations, and what an endpoint returns can only be learned by trying it.
The bill-of-materials question
I needed to read which raw materials a finished product is made of. I asked the model: which endpoint returns recipe data in this ERP?
The answer was clear, well formatted and confident:
GET /api/BillOfMaterials/{stockCode}
→ { "items": [ { "componentCode": …, "quantity": … } ] }
Entirely plausible. An ERP ought to have exactly that endpoint. Consistent naming, sensible response shape.
No such endpoint exists.
Finding the truth
To find the real one I went not to the model but to the source — the API’s controller files:
IReceteAna ← the actual interface name
GET /BOM ← summary list of finished goods
GET /BOM/{code} ← detail, a separate call
And it is two-stage: the list returns only summaries, and components require a call per product. That demands a completely different integration design from the single-call shape the model proposed.
Had I not verified it, I would have built a week of work on something that does not work.
The danger is not the wrongness
What really stayed with me: the wrong answer looked exactly like a right one.
The model did not say “I am unsure”. It did not change tone. It attached no caveat. Verified information and invented information were indistinguishable on screen.
A hallucination is not a wrong answer. It is an unverified answer that looks right. The only difference is that nobody checked.
The product rule came from this
If a manager is going to look at a screen and make a decision, the system must know where every number on it came from.
The rule: every field carries its source; a field without one never reaches the screen.
{
"value": 1284900,
"source": {
"table": "TBLSTOKURM",
"query_id": "q7f3",
"at": "2026-08-25T14:02:11+03:00"
}
}
The model no longer produces an answer; it finds where the answer comes from. The number comes from the database, and the model’s job is to build the right query and turn the result into a sentence a human can read.
That distinction looks small and determines the entire product: the model is not a source of knowledge but an interface. I had applied the same principle in catalogue search.
A side benefit: the argument ends
An unexpected gain: “this number is wrong” arguments disappeared from management reports.
Previously, when a figure was challenged nobody remembered its origin; there was a formula in a spreadsheet and no record of who wrote it when. Now the number carries which data it came from and when.
The argument moves from “this number is wrong” to “this source is wrong” — and the second is a solvable argument.
There is an interesting exception: the management report does not display the source. A general manager does not want to see a table name. The provenance is kept in the system and revealed on request.
Auditability and readability are different things; they do not have to share a screen.
What I learned
When using AI in a project the question is not “how accurate is it?” It is “how will I know when it is wrong?”
With an answer to that, the model’s error rate becomes an engineering parameter. Without one, however good it is, you cannot build anything on top of it.